TSBPO：面向交互式语言智能体策略优化的树结构分支方法
TSBPO: Tree-Structured Branching for Policy Optimization in Interactive Language Agents
Rui Yu · Chong Mu · Jinchuan Zhang
中文摘要
从终端结果训练交互式语言智能体面临三个耦合问题：采样哪些任务与状态延续、如何将延迟反馈分配给决策、以及如何聚合多 token 动作的 token 损失。TSBPO 是一种树结构采样到更新范式，在单一训练循环内解决上述问题。基于似然拟合的 Root/Branch 分配器利用保留的任务与状态证据，在固定 rollout 预算下定向采样更可能产生对比性终端结果的延续。在回放得到的物理树上，轨迹优势比较相邻兄弟片段，而观测组步优势比较匹配任务/观测状态下的结果。actor 随后对每个完整响应的裁剪 token 替代项求和，并在活跃轮次上归一化，避免策略梯度项中的响应长度平均。使用 Qwen2.5-1.5B actor 时，固定检查点在 ALFWorld 上获得 97.79% 已见和 96.64% 未见随机成功率；对 500 个 WebShop 测试目标各跑六个随机回合，每回合成功率为 81.07%。WebShop 步项消融在相同 update-100 评估下为 75.93%；ALFWorld 聚合比较显示完整轮次加权下观测到的命令有效性更高。这些结果支持所评估的训练配置在两个交互式基准上的表现。
关键要点
- 01问题：交互式语言智能体从终端结果训练时，采样、延迟反馈分配与多 token 动作损失聚合三者耦合。
- 02方法：树结构采样到更新范式，用 Root/Branch 分配器定向采样，并以兄弟片段和观测组优势分配反馈。
- 03结果：Qwen2.5-1.5B 在 ALFWorld 达 97.79%/96.64% 成功率，WebShop 每回合成功率 81.07%。
- 04消融：WebShop 步项消融为 75.93%，完整轮次加权在 ALFWorld 上命令有效性更高。
- 05局限：结论仅基于两个交互式基准上的评估训练配置。
解读
尚无解读。
原始英文摘要
arXiv:2610.41697v1 Announce Type: cross Abstract: Training interactive language agents from terminal outcomes poses three coupled problems: which task and state continuations to sample, how to assign delayed feedback to decisions, and how to aggregate token losses for multi-token actions. We present TSBPO, a tree-structured sampling-to-update paradigm that addresses these problems within one training loop. A likelihood-fitted Root/Branch allocator uses retained task and state evidence to target continuations likely to produce contrasting terminal outcomes under fixed rollout budgets. On the resulting replayed physical trees, trajectory advantages compare immediate sibling segments, while observation-group step advantages compare outcomes at matched task/observation states. The actor then sums clipped token surrogates over each complete response and normalizes across active turns, avoiding response-length averaging in the policy-gradient term. With a Qwen2.5-1.5B actor, fixed checkpoints obtain 97.79% seen and 96.64% unseen stochastic success on ALFWorld and an 81.07% per-episode success rate over six stochastic episodes for each of the 500 WebShop test goals. A WebShop step-term ablation yields 75.93% at the same update-100 evaluation; an ALFWorld aggregation comparison shows higher observed command validity with complete-turn weighting. These results support the evaluated training configuration on two interactive benchmarks.