单一置信阈值，覆盖不均：类型化决策接口中的语言差距及翻译输入如何缩小差距
One Confidence Threshold, Unequal Coverage: Language Gaps in Typed Decision Interfaces and How Translating Inputs Reduces Them
Junwei Yao · Yongxi Zhou · Zike Yang · Zhijun Wang · Jianan Liu · Yanxian Chen · et al.
中文摘要
已部署的分类器通常仅在预测置信度超过阈值时才自动处理，否则转交人工。研究考察在类型化决策接口（对固定多选题每个选项返回概率的 API）上，单一阈值是否对所有语言一视同仁。在一项有书面分析计划的验证性研究中，三家供应商的接口以 0.9 为阈值分类了 1,000 条 MASSIVE 条目，每条覆盖 51 种语言。即使某请求在英语和另一语言中均被正确回答，英语版本超过阈值的频率平均仍高出 8.6–11.7 个百分点。观测覆盖率（自动处理的请求占比）在其他每种语言中都低于英语，而门控处理的请求很少出错。将每种语言重新加权至相同准确率后差距依然存在，且标准校准指标（ECE）在一个接口上未能检测到该差距。在探索性分析中，当每种语言各自拟合阈值以达到相同的 5% 错误目标时，英语的覆盖率仍更高。将输入翻译成英语在 0.9 阈值下几乎消除差距，但在该错误目标下英语仍领先。以非洲语言原生书写的请求，当英语为其人工翻译时，差距方向相同。建议报告操作阈值下的分语言覆盖率。
关键要点
- 01问题：单一置信阈值在类型化决策接口上对不同语言覆盖不均，英语自动处理率更高。
- 02方法：用三家供应商接口、51 种语言、1,000 条 MASSIVE 条目，阈值 0.9 做验证性研究。
- 03结果：英语正确请求超阈值频率平均高 8.6–11.7 个百分点，其他语言覆盖率均低于英语。
- 04结果：翻译输入为英语在 0.9 阈值下几乎消除差距，但相同错误目标下英语仍领先。
- 05建议：应报告操作阈值下的分语言覆盖率，标准校准指标 ECE 未必能检测该差距。
解读
尚无解读。
原始英文摘要
arXiv:2610.37688v1 Announce Type: cross Abstract: Deployed classifiers often act on a prediction only when its confidence clears a threshold and pass the rest to a person. We ask whether one threshold serves all languages equally on typed decision interfaces, APIs that return a probability for each option of a fixed multiple-choice question. In a confirmatory study with a written analysis plan, three vendors' interfaces classified 1,000 MASSIVE items, each in 51 languages, with a threshold of 0.9. Even when a request is answered correctly in both English and another language, the English version clears the threshold more often, by 8.6-11.7 percentage points on average. Observed coverage, the share of requests handled automatically, is lower than English's in every other language, while the requests the gate handles are rarely wrong. The gap remains when every language is reweighted to the same accuracy, and a standard calibration metric (ECE) does not detect it on one interface. In exploratory analyses, English's coverage stays higher when each language gets its own threshold fitted to the same 5% error target. Translating inputs into English nearly closes the gap at 0.9 but leaves English ahead at that error target. Requests written natively in African languages show a gap in the same direction when English is their human translation. We recommend reporting per-language coverage at the operating threshold.