SoK: How Should Open-Text Legal Answers Be Graded?

Law students are examined through open text: case solutions, essays and legal opinions that apply the law to a set of facts. Such answers are expensive to grade, human graders disagree widely on them, and language models now offer to grade them at scale, both in teaching and in the benchmarks we use to test legal AI. At BenGER we face exactly this question every time an LLM judge grades a legal analysis. With our new paper we take a step back and ask systematically how open-text legal answers can be evaluated at all, and which procedure deserves our trust.

Eight desiderata, eight paradigms

The paper is a Systematization of Knowledge (SoK). From educational measurement and from the law that governs exams (German examination law, the GDPR and the EU AI Act) we derive eight desiderata for good grading: construct validity, reliability, reasons, credit for defensible alternative solutions, robustness to gaming, cost and scale, formative feedback, and fit with human oversight.

Against these desiderata we compare eight evaluation paradigms: unstructured human grading, human grading with an analytic rubric, closed formats, similarity to a reference answer, LLM judges with a reference answer, with generic criteria or with an instance-specific rubric, and pairwise comparison.

What we found

Three findings stand out:

  • A rubric written for the task is the strongest lever. It brings LLM judges and human graders closest together. But studies measure that agreement against very different references and rarely report how far the human graders agree with each other.
  • The central validation gap lies where examination law is most demanding: defensible alternative solutions, follow-on errors, and the examiner's correction of model judgments.
  • Rubric-centred grading is the most auditable family, because it makes the standard, its application and the final decision checkable.

The trustworthy rubric judge

From this we derive a blueprint, the trustworthy rubric judge: an instance rubric that credits arguments and alternative paths, evidence checked in code, probes, repeated runs, calibration against independent human grades, logging, and an examiner who reads every script. Today the evidence supports such a judge for formative feedback and for benchmarks. Where credit is at stake, it is an assistant to the examiner, not a replacement.

This is the same line we follow on the BenGER platform: task-specific rubrics, validated judges, and human grading as the final authority. The SoK lays the conceptual ground for our next paper, which puts this approach to an empirical test.

Read and cite

  • Preprint: arXiv:2610.19992
  • Status: submitted to the 6th ACM Symposium on Computer Science and Law (CSLAW '27)
  • Sources: the manuscript, the coding of the studies and the table scripts are open on GitHub

If you research or teach grading, rubrics or LLM judges in law, we would love to hear from you. Feedback is very welcome, especially on the open problems the paper names at the end.