cs.HC提交于 2026-10-19已译

面向不完美提示词的设计：基于知识图谱的 LLM 企业数据助手仿真评估

Designing for Imperfect Prompts: A Simulation-Based Evaluation of an LLM-Based Enterprise Data Assistant Using Knowledge Graphs

Niklas Wagner · Alexander M\"adche

中文摘要

基于大语言模型（LLM）的企业数据助手旨在实现组织内数据访问的民主化。在行业合作伙伴 CFO 部门数据湖的实际部署中，发现业务用户常提出包含领域特定词汇且缺乏模式引用或结构信息的不完美提示词。这项设计科学研究考察了使用语义中介层进行自动解析。研究设计并评估了两种变体：基于语义相似度的模式中介，以及利用从数据湖自动构建的知识图谱的知识图谱中介。使用真实数据进行的仿真评估，结合 LLM-as-a-Judge 评估和人工验证，表明知识图谱中介在精确率、召回率、F1 分数及保留用户意图方面优于基于模式的方法。研究提出了知识图谱中介的设计原则，证明结构化知识表示能连接用户意图与数据库结构，并指导不同技能水平下企业数据助手的设计。

关键要点

01解决业务用户提出的包含领域词汇且缺乏结构信息的不完美提示词问题。
02设计并对比了基于语义相似度的模式中介与基于自动构建知识图谱的中介方法。
03仿真评估显示知识图谱中介在精确率、召回率、F1 及意图保留上优于模式中介。
04提出设计原则，证明结构化知识表示能连接用户意图与数据库结构。

解读

尚无解读。

原始英文摘要

arXiv:2610.23304v1 Announce Type: cross Abstract: Enterprise data assistants using large language models (LLMs) promise democratized data access in organizations. Yet in our real-world deployment within the CFO unit's data lake at an industry partner, we observed that business users frequently pose imperfect prompts that include domain-specific vocabulary and lack schema references or structural information. This Design Science Research study examines automatic resolution using a semantic mediation layer. We design and evaluate two design variants: schema-based mediation via semantic similarity, and knowledge graph-based mediation leveraging an automatically constructed knowledge graph from the data lake. Our simulation-based evaluation using real-world data, supported by LLM-as-a-Judge assessments and human validation, shows that knowledge graph-based mediation outperforms the schema-based approach in Precision, Recall, F1, and preserving user intent. We propose a design principle for knowledge graph-based mediation, demonstrating that structured knowledge representation bridges user intent and database structure and guides the design of enterprise data assistants across skill levels.