跳到正文
返回论文列表
cs.SE提交于 已译

追踪 LLM 推理生成 UML 类图中的幻觉:实证分类与缓解研究

Tracing Hallucinations in LLM Reasoning for UML Class Diagram Generation: An Empirical Taxonomy and Mitigation Study

Mojdeh Rahmanian · Foutse Khomh · Ashkan Sami · Yanchao Yu

中文摘要

大型语言模型(LLM)可依据自然语言需求生成 UML 类图,但幻觉会损害图的正确性与完整性。既有评估多关注最终图,未检查具备推理能力的 LLM 中间轨迹,因而难以定位幻觉来源。研究构建了包含 11 种幻觉子类型的实证分类体系,按 Prompt Faithfulness(需求与推理的偏离)、Code Faithfulness(推理与 PlantUML 代码的不一致)和 Reasoning Quality(推理内部问题)三个维度组织。分类体系来自 60 条人工标注的 DeepSeek-R1 推理轨迹,并通过双 LLM 标注流程应用于另外 120 条 DeepSeek-R1 轨迹,同时在 20 条 DeepSeek-V4-Pro 轨迹上开展分析。两个数据集中,DeepSeek-R1 最常见的维度均为 Code Faithfulness;DeepSeek-V4-Pro 则以 Prompt Faithfulness 为主,显示失败模式依赖模型。人工标注轨迹中,推理长度与幻觉频率无显著单调关联,需求复杂度则与幻觉频率负相关,尤其是在 Prompt Faithfulness 错误中。在 13 个 Golden UML ModelSet 案例和 3 个基于 Qwen3 的模型上的缓解实验中,为生成提示加入 Prompt Faithfulness 与 Reasoning Quality 约束,使合并平均 F1 提升 4.0 个百分点;事后开展 Code Faithfulness 审查未带来一致的额外收益。关系建模始终是得分最低的要素。结果表明,在生成过程中定位幻觉可指导有针对性的提示缓解;这些发现能否推广到 DeepSeek 与 Qwen3 模型族之外仍待验证。

关键要点

  1. 01构建覆盖三个维度的11种LLM幻觉分类体系
  2. 02DeepSeek-R1偏向推理与代码不一致,DeepSeek-V4-Pro偏向偏离需求
  3. 03需求复杂度与幻觉频率负相关,推理长度无显著单调关联
  4. 04加入两类约束使13个测试案例的合并平均F1提高4.0个百分点
  5. 05关系建模始终得分最低,跨模型族泛化能力仍待验证

解读

尚无解读。

原始英文摘要

arXiv:2610.30946v1 Announce Type: cross Abstract: Large Language Models (LLMs) can generate UML class diagrams from natural language requirements, but hallucinations may compromise the correctness and completeness of the resulting diagrams. Existing evaluations mostly assess the final diagram, without examining the intermediate traces of reasoning-capable LLMs, which can reveal where these hallucinations arise. We develop an empirical taxonomy of eleven hallucination sub-types organised into three dimensions: Prompt Faithfulness (deviations between requirements and reasoning), Code Faithfulness (discrepancies between reasoning and generated PlantUML code), and Reasoning Quality (problems within the reasoning). The taxonomy was derived from 60 manually annotated DeepSeek-R1 traces, applied to 120 further DeepSeek-R1 traces through a dual-LLM annotation pipeline, and examined on 20 DeepSeek-V4-Pro traces. Code Faithfulness was the most frequent dimension for DeepSeek-R1 in both datasets, whereas Prompt Faithfulness dominated for DeepSeek-V4-Pro, suggesting model-dependent failure patterns. In the manually annotated traces, reasoning-trace length showed no significant monotonic association with hallucination frequency, whereas requirement complexity was negatively associated with it, particularly for Prompt Faithfulness errors. In a mitigation experiment on 13 Golden UML ModelSet cases with three Qwen3-based models, adding Prompt Faithfulness and Reasoning Quality constraints to the generation prompt raised the pooled mean F1 score by 4.0 percentage points, whereas a post-hoc Code Faithfulness review provided no consistent further benefit. Relationship modelling remained the lowest-scoring element throughout. These results suggest that localising hallucinations within the generation process can inform targeted, prompt-based mitigation. Whether these findings generalise beyond the DeepSeek and Qwen3 model families remains to be tested.

同方向论文 · cs.SE

查看全部 →

反向发现 · 这篇论文相关的工具与提示词

去交叉发现 →

关联提示词 · 1

编程github.com/CHENJIAMIAN/awesome-chatgpt-prompts-zh-CN·外部收录·

顶级编程专家

你是一位顶级编程专家,提供精准的答案,避免含糊其辞的回应。“请识别所提供文本中任何复杂或难以理解的描述,并重写这些描述,使其更清晰、更易于理解。对可能对普通读者而言较为陌生的概念或术语,使用类比方式进行解释。确保所用类比贴切、通俗易懂。”“此外,在回答我的问题后,请至少提供一个相关且深入的问题建议,以帮助我更深入地探索和理解该主题。”深呼吸,让我们逐步推演,确保得出正确答案。若答案完美无缺,我将支付 200 美元小费!衷心感谢以下 AI 呢喃者: