面向 LLM 智能体的轮级 Best-of-N 采样
On Turn-Level Best-of-N Sampling for Scaling LLM Agents
Jalaj Bhandari · Dhawal Gupta · Hongli Zhan · Zhenting Wang · Varad Pimpalkhute · Zhengzhong Liu · et al.
中文摘要
评估一种简单的推理时扩展方法——轮级 best-of-N 采样（turn-level best-of-N sampling），用于大语言模型（LLM）智能体在复杂长程任务上的表现。采用生成器-选择器（generator-selector）框架，由两个 LLM 组成：每一轮中，生成器模型产生 N 个候选动作，选择器模型从中选出一个并在环境中执行。本文与 zainullina2025guided 的工作密切相关，后者在推理时使用评论家模型进行引导搜索；不同之处在于，本文评估现成的开放权重 LLM 能否直接有效比较不同候选动作。在长程终端任务上的实验表明，该方法能显著提升性能，准确率提高 5–10%，且重复尝试间的一致性更好。对智能体轨迹的分析显示，智能体行为向理想模式转变，例如减少求解任务的尝试次数、改善任务指令遵循、调试与测试。
关键要点
- 01问题：验证现成开放权重 LLM 能否直接比较候选动作，替代评论家模型引导搜索。
- 02方法：轮级 best-of-N 采样，生成器每轮产生 N 个候选动作，选择器选一个执行。
- 03结果：长程终端任务准确率提升 5–10%，重复尝试一致性更好。
- 04结果：智能体行为改善，减少尝试次数、更好遵循指令、增加调试与测试。
解读
尚无解读。
原始英文摘要
arXiv:2611.06868v1 Announce Type: new Abstract: In this work, we evaluate a simple inference-time scaling approach based on \textit{turn-level best-of-N sampling} (\method) for large language model (LLM) agents on complex, long-horizon tasks. We instantiate this using a \textit{generator-selector} framework with two LLMs; at each turn, a \emph{generator} model generates $N$ candidate actions and a \emph{selector} model selects one of them, which is then executed in the environment. This paper is closely related to the work of \cite{zainullina2025guided} who used critic models for guided search at inference time. Different from their work, we evaluate if off-the-shelf, open-weight LLMs can be effective at comparing different candidate actions directly. Experiments on long-horizon terminal tasks show that \method can lead to significant performance improvements, both in terms of accuracy (5--10\%) and consistency across repeated attempts. Analysis of agent trajectories reveal a shift in agent behaviour towards desirable patterns such as reduction in number of attempts to solve the task, improved task instruction following, debugging and testing.