cs.CL提交于 2026-10-26已译

LLM 标注者能否达到人类一致性？人类一致性基准与基于对比知识的多智能体优化

Can LLM Annotators Match Human Consistency? Benchmarking Human Agreement and Optimizing Multi-Agent with Contrastive Knowledge

Xiaopeng Ke · Zhixin Wang · Xuebo Liu · Xinyu Ma · Xing Zheng · Bowen Song · et al.

中文摘要

现有大语言模型(LLM)标注基准与真实标注实践存在偏差：实际任务通常包含冗长的专用指南，不同标注者在边界案例上可能给出不同但合理的判断，单标签数据集无法体现这种分歧，单智能体及工作流式标注系统也难以适应复杂且持续变化的指南。ANT-A Benchmark 汇集跨领域、跨模态的 7 个数据集，提供任务专用指南，并为每个样本配备 5 个独立人工标签；通过 Fleiss kappa 衡量模型标注与人工标签分布的接近程度。ANT-OR 则从正负反馈中提炼对比决策规则并优化提示词。实验显示，ANT-OR 的 agreement@>=2 准确率较最强基线提高 4.5 个百分点，其 Fleiss kappa 与人工标注分布的对齐度最高，差距不超过 2 个百分点。代码与基准已公开。

关键要点

01现有基准难以反映真实标注指南及边界案例中的人类分歧
02ANT-A 覆盖 7 个跨领域、跨模态数据集，每例含 5 个独立人工标签
03使用 Fleiss kappa 评估模型标注与人工标签分布的一致性
04ANT-OR 从正负反馈提炼对比决策规则，以优化多智能体标注提示词
05agreement@>=2 较最强基线提升 4.5 个百分点，Fleiss kappa 差距不超过 2 个百分点

解读

尚无解读。

原始英文摘要

arXiv:2610.32415v1 Announce Type: new Abstract: LLM-based annotation has advanced rapidly, yet existing benchmarks remain misaligned with real-world practice. Actual annotation often involves lengthy, task-specific guidelines, and different annotators may produce divergent but reasonable judgments, especially on boundary cases, where single-labeled dataset fail to reflect this disagreement. This gap also limits existing single-agent and workflow-based annotation systems, which often lack adaptability to complex, evolving guidelines. To bridge this gap, we introduce ANT-A Benchmark, a real-world multi-annotator benchmark spanning 7 datasets across domains and modalities, with task-specific guidelines and 5 independent human labels per instance. ANT-A enables a distribution-aware evaluation, using Fleiss kappa to assess how closely model annotations align with human labeling distributions. We further propose ANT-OR, a multi-agent annotation framework that refines prompts by distilling contrastive decision rules from positive and negative feedback. Experiments show that ANT-OR improves agreement@>=2 accuracy by 4.5 points over the strongest baseline and achieves the highest alignment with human annotator distributions as measured by Fleiss kappa within 2 points. Code and benchmark are available at https://github.com/Krueske/ANT-OR.