cs.CL提交于 2026-10-23已译

提示策略的效果取决于评判者：LLM-as-a-Judge 稳健性的多语言研究

Prompting Strategy Effects Depend on the Judge: A Multilingual Study of LLM-as-a-Judge Robustness

Asal Mehrabi · Mohammad Aliannejadi · Clemencia Siro

中文摘要

大语言模型评判（LLM-as-a-Judge）日益用于多语言任务，但思维链（Chain-of-Thought）、角色设定（Role Conditioning）等提示策略通常仅依据英文基准推荐，其效果能否迁移至不同评判模型和语言仍不清楚：提示敏感性研究多以英文为主，现有多语言评判研究则固定提示模板。该研究通过 9 种提示策略、4 个开放权重评判模型、27 种语言，以及覆盖 13 个数据集的 4 类任务，测量二者的交互作用；在标准参考输出上产生超过 1M 个逐点评分，并以人类判断验证。提示策略不可迁移：评判模型的选择比提示策略更关键，同一策略在不同评判模型上的效果方向可能相反；低资源语言中的稳定性更差，而汇总报告会掩盖这些效应。结果表明，多语言评估中的提示效果取决于评判者、任务和语言。因此，应针对预定评估设置验证提示策略，跨研究比较时也应同时考虑评判模型与提示配置。

关键要点

01现有提示敏感性研究偏向英文，且多语言评判常固定提示模板。
02研究覆盖 9 种策略、4 个开放权重评判模型、27 种语言和 13 个数据集。
03在超过 1M 个标准参考输出逐点评分中，评判模型选择比提示策略更关键。
04策略效果在不同评判模型间可能反转，低资源语言中的稳定性更差。
05提示策略需按目标评估设置验证，跨研究比较须记录评判模型与提示配置。

解读

尚无解读。

原始英文摘要

arXiv:2610.31172v1 Announce Type: cross Abstract: LLM-as-a-Judge evaluation is increasingly applied to multilingual tasks, yet prompting strategies such as Chain-of-Thought and Role Conditioning are recommended based solely on English benchmarks. Whether their effectiveness transfers across judge models and languages remains unknown: prompt-sensitivity studies are largely English-centric, while existing multilingual judge studies fix the prompt template. In this work, we address this gap by measuring this interaction through a benchmark of 9 prompting strategies, 4 open-weight judges, 27 languages, and 4 tasks across 13 datasets, yielding over 1M pointwise scores on gold-reference outputs, validated against human judgments. We find that prompting strategies are not transferable: the choice of the judge model dominates the choice of prompting strategy, strategy effects reverse direction across judges, and stability degrades in low-resource languages, where these effects are concealed by aggregate reporting. These findings show that prompting effects in multilingual evaluation are conditioned on the judge, task, and language. Consequently, prompting strategies should be validated for the intended evaluation setting, and cross-study comparisons should account for both judge and prompt configuration.