SAUCE：基于对话蕴含的摘要分析
SAUCE: Summary Analysis Using Conversation Entailment
Man-Ling Sung · Hemanth Kandula · Jeff Ma · William Hartmann · Matthew Snover
中文摘要
随着评估大语言模型（LLM）及其语音应用的需求增长，对缺乏明确最终目标的对话进行摘要与评估仍存在挑战。SAUCE 是一个无参考、基于事实的跨语言对话语音摘要评估流程，通过对话与文本之间的蕴含关系衡量摘要的准确性和事实覆盖度。将 SAUCE 与多种常用摘要指标对比，展示其捕捉转录和翻译错误导致的信息损失以及识别破损摘要的有效性。与黑盒 LLM 评估器或稠密嵌入指标不同，SAUCE 具有内在可解释性：将摘要分数映射到离散、可验证的事实，使用户能定位确切的幻觉或遗漏。这种可解释性帮助开发者系统性地刻画 LLM 行为，并为终端用户提供在嘈杂真实条件下验证摘要准确性的可操作工具。初步研究表明 SAUCE 与人类判断高度一致。
关键要点
- 01针对缺乏明确目标的对话摘要评估难题，提出无参考、基于事实的跨语言评估流程 SAUCE。
- 02通过对话与文本间的蕴含关系衡量摘要准确性与事实覆盖度，可捕捉转录和翻译错误导致的信息损失。
- 03将摘要分数映射到离散可验证事实，能定位幻觉或遗漏，优于黑盒 LLM 评估器与稠密嵌入指标。
- 04初步研究表明 SAUCE 与人类判断高度一致，可帮助开发者刻画 LLM 行为并支持终端用户验证摘要。
解读
尚无解读。
原始英文摘要
arXiv:2611.06812v1 Announce Type: new Abstract: With the growing need for evaluating Large Language Models (LLMs) and their applications to speech, challenges persist in summarizing and evaluating conversations that lack a clear end goal. We introduce SAUCE -- a reference-free, fact-based evaluation pipeline for cross-lingual conversational speech summarization. It measures the accuracy and the fact coverage of a summary through the entailment between conversation and text. We compare SAUCE against several popular summarization metrics and demonstrate the effectiveness of capturing information loss due to transcription and translation error and identifying broken summaries. Crucially, unlike black-box LLM evaluators or dense embedding metrics, SAUCE is inherently explainable: it maps summary scores to discrete, verifiable facts, allowing users to pinpoint exact hallucinations or omissions. We illustrate how this interpretability helps developers systematically profile LLM behaviors and gives end-users an actionable tool to verify summary accuracy in noisy, real-world conditions. Preliminary investigations show SAUCE strongly align with human judgment.