cs.CL提交于 2026-10-28已译

言语还是行动？LLM 如何在人机对话中预测用户决策

Words or Deeds? How LLMs Predict Users' Decisions in Human-Agent Dialogue

Baiqiao Zhang · Shuai Ma · Zhifeng Liao · Chao Zhou · Jianshu Wang · Yuhan Zhao · et al.

中文摘要

Web 系统常依据用户言语预测其决策，但在利益冲突时，用户可能承诺合作却随后违背。研究测试了 LLM 如何结合用户陈述与决策记录进行预测。实验包含 1,260 轮重复囚徒困境游戏，42 名用户与五个具有不同大五人格特质的智能体互动。在八款开源 LLM 中，仅基于决策记录的预测区分合作与背叛效果良好（AUC = .80），加入对话未带来显著增益。虽然加入对话后预测更常符合用户陈述，但错误集中在用户违背陈述的回合：违背陈述时的准确率从 .65 骤降至 .17，而遵守陈述时从 .76 升至 .95。LLM 过度加权用户陈述：当用户陈述尚未做出的决策时，预测保持概率从 .13 升至 .69，而实际保持率仅为 .40。此外，用户与不同特质智能体的合作率差异显著（如宜人性 70.6% vs 外向性 54.5%），但预测值几乎相同（64.5% vs 63.1%）。这表明基于对话的预测应结合用户过往决策及面对的智能体特质来权衡其言语。

关键要点

01仅用决策记录预测用户合作或背叛的 AUC 达 .80，加入对话数据未带来可检测的性能提升。
02LLM 过度信任用户言语，当用户违背承诺时，预测准确率从 .65 骤降至 .17，严重偏离实际行为。
03预测模型未能捕捉用户与不同人格特质智能体互动时的行为差异，预测合作率几乎恒定。
04建议未来预测应结合用户历史决策记录及所面对智能体的特质，以动态调整对用户言语的信任权重。

解读

尚无解读。

原始英文摘要

arXiv:2610.36813v1 Announce Type: new Abstract: Web systems increasingly predict users' decisions from what users say, yet when interests partly conflict, users may promise to cooperate to win the other side's cooperation and then not do so. Sales or service agents that rely on such promises may grant concessions to users who then do not keep their promises, at a cost to the party they represent. We test how LLMs combine users' statements with decision records to predict users' decisions, using 1,260 rounds in which 42 users played a repeated prisoner's dilemma-style game with five agents, each prompted to express a dominant Big Five trait, and could talk with the agent before every decision. Across eight open-weight LLMs, predictions from decision records separated cooperation from defection well (AUC = .80), and adding dialogue brought no detectable gain. With dialogue, predictions agreed with users' statements more often, and errors concentrated on rounds in which users broke their statements: accuracy on rounds with a statement barely changed (.73 to .74), while it rose on kept statements (.76 to .95) and fell sharply on broken ones (.65 to .17). Predictions weighted statements more heavily than users' behavior warranted: when users stated a decision they had not yet made in that game, the predicted probability of keeping it rose from .13 to .69, whereas users kept .40 of such statements. Users' decisions also varied with the agent they faced, but predictions hardly reflected this: users cooperated more with the Agreeableness agent than with the Extraversion agent (70.6% vs. 54.5%), whereas predicted cooperation was nearly identical (64.5% vs. 63.1%). Users also cooperated more with an agent when they themselves scored higher on that agent's trait, whereas predicted cooperation barely changed with these scores. Dialogue-based predictions should therefore weigh users' statements by their earlier decisions and the agent they face.