cs.LG提交于 2026-10-28已译

面向多轮智能体评估的意图保持对抗性用户模拟

Intent-Preserving Adversarial User Simulation for Multi-Turn Agent Evaluation

Atri Vivek Sharma · Brian Formento · Alessio Lomuscio

中文摘要

多轮智能体通常使用基于大语言模型（LLM）的模拟用户进行评估，但这类用户往往过于配合，导致高估智能体性能。StressSim 是一种意图保持的对抗性用户模拟框架，旨在通过具有挑战性但有效的交互对智能体进行压力测试。该框架针对每个任务实例优化模拟器的高层对抗计划，利用遗传提示优化循环演化候选规划器提示，结合 rollout 轨迹、细粒度任务完成信号及有效性反馈，优先选择在不改变用户底层目标的前提下降低任务成功率的对话。在 MultiWOZ 数据集及 tau-bench 的零售和航空领域，使用三个目标 LLM 智能体进行评估，并与标准模拟器及手工设计的非协作用户行为进行对比。结果显示，StressSim 有效暴露了前沿 LLM 的鲁棒性缺口，gemini-3.8-flash 和 gpt-5.6-sol 的 tau-bench pass^4 分数分别下降高达 69% 和 62%。在每任务 20 次 rollout 的优化预算下，目标保持性能下降不超过 10%。敏感性分析表明，增加预算会进一步降低任务成功率。StressSim 仅通过轻量级提示优化实现，无需模型微调，在保持对话真实性的同时揭示了标准模拟器遗漏的鲁棒性问题。

关键要点

01现有 LLM 模拟用户过于配合，导致多轮智能体评估中性能被高估。
02StressSim 利用遗传提示优化演化对抗计划，在不改变用户目标的前提下降低任务成功率。
03在 tau-bench 上，gemini-3.8-flash 和 gpt-5.6-sol 的 pass^4 分数分别下降 69% 和 62%。
04每任务 20 次 rollout 的优化预算下，目标保持性能下降不超过 10%，无需模型微调。

解读

尚无解读。

原始英文摘要

arXiv:2610.35426v1 Announce Type: new Abstract: Multi-turn agents are commonly evaluated with LLM-based simulated users, which can be overly cooperative and can therefore overestimate an agent's performance. We here introduce StressSim, an intent-preserving adversarial user simulator framework for stress-testing agents with challenging yet valid user interactions. StressSim optimises the simulator's high-level adversarial plan for each task instance. Candidate planner prompts are evolved with a genetic prompt-optimisation loop that uses rollout traces, granular task-completion signals, and validity feedback to favour conversations that reduce task success without changing the user's underlying goal. We evaluate the approach on the MultiWOZ dataset and the retail and airline domains from tau-bench with three target LLM agents, comparing against standard simulators and hand-designed non-collaborative user behaviours. We demonstrate the effectiveness of StressSim in exposing robustness gaps across frontier LLMs, with tau-bench pass^4 scores decreasing by up to 69% on gemini-3.8-flash and 62% on gpt-5.6-sol. StressSim maintains goal preservation performance (which decreases by at most 10%), using an optimisation budget of 20 rollouts per task. Sensitivity analysis suggests that increasing the budget further decreases task success. Overall our results demonstrate that StressSim exposes robustness gaps missed by standard and hand-designed user simulators while maintaining conversation realism. This is achieved through lightweight prompt optimisation alone, without model fine-tuning.