SoK: 开放式法律答案评分：从法学院考试到可信的评分标准裁判

arXiv:2610.19992v2 Announce Type: replace Abstract: Law students are examined through open-text answers: case solutions, essays and memos that apply the law to a set of facts. Such answers are expensive to grade, human graders disagree widely on them, and large language models (LLMs) now offer to grade them at scale, both in teaching and in the benchmarks that test legal AI. This paper systematizes how open-text legal answers can be evaluated. We derive eight desiderata from educational measurement and from the law that governs exams, among them reasons, credit for defensible alternative solutions, robustness to gaming and human oversight. We then compare eight evaluation paradigms against them, from unstructured human grading to LLM judges with instance-specific rubrics and pairwise comparison. Three findings stand out. A rubric written for the task is the strongest lever for agreement between an LLM judge and human graders, but studies measure that agreement against different references and rarely report how far the human graders agree with each other. The central validation gap lies where examination law is most demanding: defensible alternative solutions, follow-on errors, and the examiner's correction of model judgments. Rubric-centred grading is the most auditable family, because it makes the standard, its application and the final decision checkable. From this we derive the trustworthy rubric judge: an instance rubric that credits arguments and alternative paths, evidence checked in code, probes, repeated runs, calibration against independent human grades, logging, and an examiner who reads every script. Today the evidence supports such a judge for formative feedback and for benchmarks, and as an assistant to the examiner, not a replacement, where credit is at stake.