cs.CL提交于 2026-10-27已译

超越提示词：基于 LLM 的机器翻译评估器系统研究

Beyond Prompts: A Systematic Study of LLM-Based Machine Translation Evaluators

Dmitry Popov · Roman Bokhyan · Ekaterina Enikeeva · Artem Mekhraliev · Stepan Vysotsky · Nikolay Karpachev

中文摘要

基于大语言模型(LLM)的评估器已迅速成为机器翻译(machine translation, MT)评估的标准工具，但塑造其行为的设计选择仍缺乏系统理解。研究分析了指令设计、少样本示例、骨干模型选择及多步工作流对片段级与排序式 MT 评估的影响。作者为 WMT24 和 WMT25 英俄翻译收集了新的专家 RATE 标注，并在五个额外语言对的 MQM 与 ESA 标注基准上开展迁移验证。实验发现：指令结构会塑造评估器行为；高质量示例比数量更重要；前沿骨干模型呈现不同的评估特征，分别偏向召回率、精确率或排序；融合不同骨干模型的输出，优于让单一评估器进行多轮调用。尽管评估器设计存在庞大的组合空间，若干设计原则可跨标注协议和语言对稳定迁移。作者据此构建了 ORBIT 系列 LLM 机器翻译评估器。最强配置在错误片段检测以及准确性与流畅性排序方面达到专家水平，成本约为专家人工标注的 1/20。

关键要点

01系统研究指令、少样本示例、骨干模型与多步工作流对 MT 评估的影响。
02收集 WMT24、WMT25 英俄翻译的新专家 RATE 标注。
03在五个额外语言对的 MQM 与 ESA 基准上验证迁移能力。
04高质量示例比示例数量更影响评估器表现。
05融合多种骨干模型输出优于单一评估器多轮调用。

解读

尚无解读。

原始英文摘要

arXiv:2610.33741v1 Announce Type: new Abstract: LLM-based evaluators have rapidly become a standard tool for machine translation evaluation, yet the design choices that shape their behavior remain poorly understood. We present a systematic study of how instruction design, few-shot demonstrations, backbone model choice, and multi-step workflows affect span-level and ranking-based MT evaluation. We collect new expert RATE annotations on WMT24 and WMT25 English-Russian translations, with transfer validation on MQM- and ESA-annotated benchmarks across five additional language pairs. Our experiments reveal four robust findings: instruction structure shapes evaluator behavior, high-quality demonstrations matter more than quantity, frontier backbones exhibit distinct evaluator profiles (recall-, precision-, or ranking-oriented), and fusing outputs across diverse backbones outperforms single-evaluator multi-call setups. Although evaluator design exposes a large combinatorial space, several discovered design principles transfer consistently across annotation protocols and language pairs. We synthesize these findings into ORBIT, a family of LLM-based MT evaluators. Our strongest configurations achieve expert-level performance in error span detection and in Accuracy and Fluency ranking at roughly 20x lower cost than expert human annotation.