知道你所知:人机认知系统中审阅者价值的特质理论
Knowing What You Know: A Trait Theory of Reviewer Value in Human-AI Cognitive Systems
Priyanka Sinha · Sugam Pandey
中文摘要
当能力强的模型与人类审阅者合作进行同一决策时,哪个人类仍能增加价值?我们认为这是一个关于单一认知特质的问题:元认知校准(metacognitive calibration),即审阅者的行为信心与其判断实际正确性之间的对齐程度。我们发展了一种审阅者价值的特质理论来使其精确化。该理论提出五个主张:价值由校准而非活动、速度、一致性或任期决定;校准在行为遥测中留下可观察的痕迹;这些痕迹本身是非诊断性的,因为校准专家的快速、低修订特征与自信新手的特征无法区分,除非以结果为门控;人类校准与模型的自我知识是同一构念,因此联合系统应在每个项目上服从校准更好的代理;个体校准只有在单个判断层面以上才能因果识别。我们基于一个AI辅助会计系统六个月的产线窗口(541621个审阅者事件,483名审阅者)为每个主张提供依据,发现纯行为专长画像颠倒了真实情况,将过度自信的群体(5.2%纠正存活率)评分高于真正的专家(99.2%);审阅者质量在任期内持平,因此该特质并非在工作中习得;将区分好坏审阅者的相同校准逻辑应用于模型,可将92.5%的人类努力集中在真正的错误上。我们以理论必须尊重的边界作结:在多数投票聚合下,个体因果价值不可识别。
关键要点
- 01问题:人机协作中,人类审阅者的价值由元认知校准而非活动、速度等决定。
- 02方法:提出五主张的特质理论,并用AI辅助会计系统六个月生产数据(541621事件)验证。
- 03结果:纯行为专长画像颠倒真实情况,过度自信群体评分高于真正专家。
- 04结果:审阅者质量在任期内持平,校准特质非工作习得;校准逻辑使92.5%人类努力集中于真正错误。
- 05局限:多数投票聚合下,个体因果价值不可识别。
解读
尚无解读。
原始英文摘要
arXiv:2611.24932v1 Announce Type: new Abstract: When a capable model and a human reviewer cooperate on the same decision, which human still adds value? We argue this is a question about a single cognitive trait, metacognitive calibration: the alignment between a reviewer's behavioral confidence and the actual correctness of their judgments. We develop a trait theory of reviewer value to make it precise. The theory makes five claims: value is governed by calibration rather than by activity, speed, agreement, or tenure; calibration leaves observable traces in behavioral telemetry; those traces are non-diagnostic on their own, because the fast, low-revision signature of a calibrated expert is indistinguishable from that of a confident novice until it is gated by outcomes; human calibration and a model's self-knowledge are the same construct, so the joint system should defer to whichever agent is better calibrated on each item; and individual calibration is causally identifiable only above the level of the single judgment. We ground each claim in a six-month production window of an AI-assisted accounting system (541621 reviewer events, 483 reviewers), and find that a purely behavioral expertise profile inverts the ground truth, scoring an overconfident cohort (5.2 percent correction survival) higher than genuine experts (99.2 percent); that reviewer quality is flat across tenure, so the trait is not learned on the job; and that the same calibration logic that separates good from bad reviewers, applied to the model, concentrates 92.5 percent of human effort on genuine errors. We close with the boundary the theory must respect: under majority-vote aggregation, individual causal value is unidentifiable.