cs.CL提交于 2026-10-28已译

HEART-Bench: 大语言模型能否做出符合人设的决策？

HEART-Bench: Do LLMs Make Persona-Consistent Decisions?

Weihan Peng · Chenxu Zhang · Qianao Wang · Yuling Shi · Heng Lian · Qihong Mao · et al.

中文摘要

针对互动娱乐及社交应用中基于性格与过往经历进行决策的需求，引入 HEART-Bench 评估基于记忆的人设一致性响应选择。该基准包含 11 个基于大五人格（Big Five）的合成角色，每个角色拥有 1,000 条自传式记忆，结合 64 个由 DIAMONDS 指导的决策场景，构成 673 道选择题。领域专家独立比对候选响应，由资深专家解决分歧并修订；其他角色对同一场景的判定响应作为干扰项。主评估中，模型接收角色 ID、职业、场景及选项，记忆增强设置下加入检索记忆，但隐藏详细人格与价值观档案。在 No-Retrieval、Naive-RAG、Mem0 和 PersonaDB 设置下评估 16 个 LLM。无记忆时准确率接近 25% 随机水平，最佳记忆增强配置达 64.9%。在匹配的 150 项子集上，人类标注者达 80.8%，最佳模型为 64.0%。消融实验表明，检索同时匹配目标角色与决策情境的经历至关重要。

关键要点

01针对 LLM 在互动娱乐中缺乏基于记忆和性格的一致性决策能力，提出 HEART-Bench 基准。
02构建包含 11 个大五人格角色、1,000 条记忆及 64 个决策场景的 673 道多选题数据集。
03评估 16 个 LLM，无记忆时准确率近随机水平，最佳记忆增强配置准确率达 64.9%。
04人类标注者在 150 项子集上准确率为 80.8%，显著高于最佳模型的 64.0%。
05消融实验显示，检索需同时匹配目标角色特征与具体决策情境才能提升性能。

解读

尚无解读。

原始英文摘要

arXiv:2605.30058v2 Announce Type: replace Abstract: LLM agents in interactive entertainment and social applications, including companions, role-play characters, and persona clones, need decisions grounded in personality and past experiences. We introduce HEART-Bench to evaluate memory-grounded persona-consistent response selection. The benchmark combines 11 Big Five-anchored synthetic characters, each with 1,000 autobiographical memories, and 64 DIAMONDS-guided decision scenarios into 673 multiple-choice questions. Domain experts independently compare candidate responses, with a senior expert resolving disagreements and revising responses when needed; other characters' adjudicated responses to the same scenario serve as distractors. In the main evaluation, models receive character IDs and occupations, scenarios, and options, with retrieved memories added in memory-augmented settings. Detailed personality-and-value profiles are withheld. We evaluate 16 LLMs under No-Retrieval, Naive-RAG, Mem0, and PersonaDB. Accuracy without memories stays near the 25% chance level, and the best memory-augmented configuration reaches 64.9%. On a matched 150-item subset, human annotators reach 80.8%, compared with 64.0% for the best model. Memory-content ablations show the importance of retrieving experiences that match both the target character and the decision context.