继续之前：评估对话助手中的时间敏感安全感知
Before You Continue: Evaluating Time-Sensitive Safety Awareness in Conversational Assistants
Peidi Dong · Yunze Xiao · Yu-Lin Tsai · Yichen Zheng · Heyuan Huang · Weihao Xuan · et al.
中文摘要
大语言模型（LLM）助手越来越多地帮助用户处理涉及时间依赖约束的日常活动。但用户可能在后续请求中不提及这些约束，要求助手自行识别并调整建议。为评估这一能力，研究提出 TASC（Temporal Action Safety in Conversation）基准，用于评估 LLM 在回应对话请求时是否应用时间依赖约束。TASC 包含由 128 个模板生成的 3,418 条输入，以及一组独立的 33 个场景，用于比较模型计算已过时间与应用该信息的能力。在 13 个模型中，当建议行动违反规则时，助手仅在 30.5% 的情况下考虑了时间依赖约束，同时很少发出不必要的警告。更重要的是，尽管模型正确计算了已过时间，仍有 62.1% 的情况未能应用这些约束。明确提供时间信息仅能解决其中约一半的失败，揭示了时间推理与其在对话建议中应用之间的差距。
关键要点
- 01问题：用户后续请求常省略时间约束，助手需自行识别并调整建议，现有评估缺乏此类基准。
- 02方法：构建 TASC 基准，含 128 个模板生成的 3,418 条输入及 33 个场景，测试 13 个模型。
- 03结果：建议违反规则时仅 30.5% 考虑时间约束；62.1% 情况虽正确计算时间却未应用。
- 04局限：明确提供时间信息仅解决约一半失败，时间推理与应用之间存在明显差距。
解读
尚无解读。
原始英文摘要
arXiv:2611.06839v1 Announce Type: new Abstract: Large language model (LLM) assistants increasingly help users with everyday activities that involve time-dependent constraints. However, users may not mention these constraints in later requests, requiring assistants to recognize and adjust their advice accordingly. To evaluate such capability, we introduce TASC (Temporal Action Safety in Conversation), a benchmark that evaluates whether LLMs apply time-dependent constraints when responding to conversational requests. TASC consists of 3,418 inputs derived from 128 templates, along with a separate set of 33 scenarios designed to compare models' ability to calculate elapsed time with their ability to apply that information. Across 13 models, assistants account for time-dependent constraints in only 30.5% of cases where the proposed action violates a rule, while rarely warning users unnecessarily. More importantly, models fail to apply these constraints in 62.1% of cases despite correctly calculating the elapsed time. Explicitly providing the timing resolves only about half of these failures, revealing a gap between temporal reasoning and its application in conversational advice.