cs.CL提交于 2026-10-07已译

一致性并非有效性:跨模型 LLM 共识在诊断 K-12 数学辅导对话中学生失败模式中的研究

Agreement Is Not Validity: Cross-Model LLM Consensus in Diagnosing Student Failure Modes in K-12 Math Tutoring Dialogue

Clayton Cohn · Joyce Fonteles · Kirk Vanacore · Gianni Mazza · Candida Crawford · Tom Hooper · et al.

中文摘要

在 K-12 数学辅导场景中,师生对话蕴含着丰富的学习者解题过程与困难来源信息。学习分析研究越来越多地依赖大语言模型(LLMs)从对话中抽取此类信息,用于知识追踪、行为建模以及学生推理错误的诊断等下游任务。然而,这些由模型生成的解读的有效性仍缺乏充分理解。本探索性研究使用一套操作型诊断编码手册,考察 LLM 对数学辅导对话中五类学生失败模式的分类有效性,五类失败模式分别为:不确定性(uncertainty)、错误归因(misattribution)、算子选择错误(operator selection)、概念缺口(conceptual gap)和程序性失误(procedural slip)。跨模型来看,人类与 LLM 之间的一致性为中等水平(kappa = .524–.597),而模型之间的相互一致性则显著更高(kappa = .755–.781;alpha = .769)。这些结果表明,跨模型一致性可能营造出正确性的虚假表象,挑战了「LLM 之间的共识即可证明学习者解读有效」这一假设。对学习分析领域的启示十分明确:可扩展的标注工作仅在所推断的构念(concept)具备有效性时才有意义,而模型共识无法替代对这种有效性的独立证据。

关键要点

01问题:LLM 对辅导对话中学生失败模式的解读有效性尚未被充分验证,存在将「模型共识」等同于「标注正确」的隐含假设。
02方法:基于五类失败模式的操作型诊断编码手册(不确定性、错误归因、算子选择、概念缺口、程序性失误),对比人类标注与多个 LLM 的分类结果,并考察 LLM 之间的相互一致性。
03结果:人机一致性中等(kappa = .524–.597),而跨模型一致性显著更高(kappa = .755–.781;alpha = .769),表明共识并不反映正确性。
04局限:结论来自探索性研究,诊断维度仅覆盖五种失败模式,人机一致性上限仍处于中等区间,有效性证据仍需独立来源支撑。

解读

尚无解读。

原始英文摘要

arXiv:2610.08703v1 Announce Type: new Abstract: In K-12 mathematics tutoring, student-tutor dialogue provides rich evidence of learners' problem-solving processes and sources of difficulty. Learning analytics research increasingly relies on large language models (LLMs) to extract such information from dialogue for a variety of downstream tasks, including knowledge tracing, behavioral modeling, and diagnosis of student reasoning errors. However, the validity of these model-generated interpretations remains insufficiently understood. In this exploratory study, we examine the validity of LLM classifications of five student failure modes in mathematics tutoring dialogue using an operational diagnostic codebook: uncertainty, misattribution, operator selection, conceptual gap, and procedural slip. Across models, human-LLM agreement was moderate (kappa = .524-.597), while cross-model agreement was substantially higher (kappa = .755-.781; alpha = .769). These findings show that cross-model agreement can create a misleading appearance of correctness, challenging the assumption that consensus among LLMs constitutes evidence of valid learner interpretation. For learning analytics, the implication is clear: scalable labeling is useful only if the inferred constructs are valid, and model consensus cannot substitute for independent evidence of that validity.