SCOPE: Selective Critical-Step On-Policy Self-Distillation for Agentic Reinforcement Learning
Qirui Liu · Wei Cao · Yu Mi · Yichen Sun · Xuekun Zhang · Lei Liu · et al.
基于结果的强化学习为长程 LLM 智能体提供的监督较为粗糙：失败轨迹中可能包含正确步骤，成功轨迹中也可能包含错误。轨迹级技能还会在无关步骤上改变教师偏好，引入偏离目标的蒸馏信号。为此提出 SCOPE 框架，即选择性关键步骤同策略自蒸馏（Selective Critical-Step On-Policy Self-Distillation）。在监督预热后，策略分析自身轨迹，生成针对具体步骤的事后技能，并仅在选定的错误步骤上蒸馏这些技能的指导。有界带符号的教师-学生系数稳定辅助更新，同时保留正向强化与负向纠正。成功相对成对长度奖励进一步鼓励高效完成任务。在 ALFWorld 与 WebShop 上的实验表明，该框架在多个模型骨干上均取得一致提升。使用 Qwen2.5-3B 时，SCOPE 在 ALFWorld 上达到 97.7% 成功率，在 WebShop 上达到 85.2%，分别比 SEED 高出 5.9 与 6.3 个百分点。在未见过的 ALFWorld 任务上，SCOPE 达到 89.6% 成功率，比 GRPO 提升 23.2 个百分点，交互次数减少 46.7%。扩展至 Qwen3-32B 时，SCOPE 在已见 ALFWorld 任务上达到 99.2% 成功率，在未见任务上达到 94.0%。
尚无解读。
arXiv:2610.38391v1 Announce Type: new Abstract: Outcome-based reinforcement learning provides coarse supervision for long-horizon LLM agents: failed trajectories may contain correct steps, while successful trajectories can contain errors. Trajectory-level skills can also shift teacher preferences at unrelated steps, introducing off-target distillation signals. We introduce SCOPE, a framework for Selective Critical-Step On-Policy Self-Distillation. After a supervised warm-up, the policy analyzes its own trajectories, generates step-specific hindsight skills, and distills their guidance only at selected error steps. A bounded signed teacher-student coefficient stabilizes auxiliary updates while preserving positive reinforcement and negative correction. A success-relative pairwise length reward further encourages efficient task completion. Experiments on ALFWorld and WebShop demonstrate consistent improvements across multiple model backbones. With Qwen2.5-3B, SCOPE achieves 97.7% success on ALFWorld and 85.2% on WebShop, outperforming SEED by 5.9 and 6.3 percentage points, respectively. On unseen ALFWorld tasks, SCOPE reaches 89.6% success, improving over GRPO by 23.2 percentage points with 46.7% fewer interactions. Scaling to Qwen3-32B, SCOPE achieves 99.2% success on seen ALFWorld tasks and 94.0% on unseen tasks.
I wanna build an AI that can learning a deck in Master duel called Kewl Tune that can win against pro players with differents decks , This AI can build combos and evo his level with a tool called ygo agent on git hub and with Kaggle Noteboo…