[Shared task] LLM as a Judge?: From Statute Prediction to Sycophancy Detection in Law

Christophe Bobineau
Calendar
When:
17/12/2026 – 20/12/2026 all-day
2026-12-17T01:00:00+01:00
2026-12-20T01:00:00+01:00
Conferences SimpleText

Date : 2026-12-17 => 2026-12-20
Lieu : Indian Statistical Institute, Kolkata, India

We are glad to announce the track LLM as a Judge?: From Statute Prediction to Sycophancy Detection in Law with FIRE 2026.

Website: https://sycolex.com/

Problem Statement

AI and Law have rapidly grown as an area of research worldwide, across the judiciary. Researchers across multiple legal systems, tasks, and languages have explored the applicability of LLMs to this field. Nonetheless, the AI prediction/recommendation system is perhaps inconsequential unless the legal experts find it useful, and, more importantly, interpretable and reliable. To provide a testbed for a robust evaluation of LLMs in AI and Law, we propose two tasks:

  1. Explainable Statute Prediction

    In this task, we present an empirical platform for evaluating not only the predictions but also the underlying explanations acceptable to legal experts within the classical legal AI paradigm of Statute Prediction.

    So, in this task, given a set of training triplets — facts, predicted statute, explanation — participants will be given a test set of facts only, from which they will have to predict the statute and its explanation.

  2. Detection of Sycophantic Behavior in LLMs

    Sycophancy in Large Language Models — LLMs — is the tendency to echo user beliefs regardless of truth. This poses serious risks in legal contexts, where AI-endorsed misinformation can mislead decision-making.

    So, in this task, given a legal query, the participants have to detect sycophantic behaviors using LLMs. We will have legal questions that incorporate clear outcome expectations for the training set. For the task, participants will have to predict the outcome based on the query.

Please register here:
https://sycolex.com/pages/registration.html

Organizers

  • Kripabandhu Ghosh, Indian Institute of Science Education and Research, Kolkata — IISER-K, India
  • Liana Ermakova, Université de Bretagne Occidentale — UBO, France
  • Shuvam Banerjee Seal, Indian Institute of Science Education and Research, Kolkata — IISER-K, India
  • Subinay Adhikary, Indian Institute of Science Education and Research, Kolkata — IISER-K, India
  • Jaap Kamps, University of Amsterdam — UvA, The Netherlands

Lien direct

Notre site web : www.madics.fr
Suivez-nous sur Tweeter : @GDR_MADICS
Pour vous désabonner de la liste, suivre ce lien.