SkillSandbox:基于动态场景合成(scenario synthesis)的技能验证
SkillSandbox: Skill Verification via Dynamic Scenario Synthesis
Serin Kim · Kwangwook Seo · Dokyung Song · Jinyoung Yeo · Dongha Lee
中文摘要
自进化(self-evolving)智能体将任务求解经验提炼为技能(skill)以便未来复用,但这些技能可能编码了错误的操作步骤或不可迁移的知识。因此,必须验证每个技能的可复用性:其指导在新任务中是否依然有效。此类验证需要观察技能在新任务执行中的实际作用,但现有任务未必能暴露目标技能真正可被调用的情境。为构造这样的情境,提出 SkillSandbox 框架,为每个技能动态合成一个任务及其环境,使其与技能相关且新颖。框架由三个组件构成:Proposer 指定需保留的条件与应变化的源特定细节;Builder 据此构建可执行场景;Verifier 对比有/无该技能两种条件下的执行情况,从可执行性(executability)、效用(utility)与效率(efficiency)三个维度评估,给出 Keep 或 Reject 判定,决定技能是否入库。在 ALFWorld 与 WebShop 上使用三种模型测试,SkillSandbox 一致带来最优的下游任务性能与更高的执行效率。进一步分析考察上述增益是否反映对技能可复用性的准确评估,并识别 SkillSandbox 中起关键作用的组件。
关键要点
- 01问题:智能体提炼的技能可能包含错误步骤或不可迁移经验,需要逐技能验证其在新任务中的可复用性,而现有任务往往难以暴露技能适用的情境
- 02方法:SkillSandbox 通过 Proposer 指定保留条件与可变细节、Builder 构建可执行场景、Verifier 评估可执行性/效用/效率并给出 Keep/Reject 判定
- 03结果:在 ALFWorld 与 WebShop 上、采用三种模型,均取得最强下游任务性能并提升执行效率
- 04分析:进一步检验增益是否源于对技能可复用性的准确评估,并剖析各组件对最终收益的贡献
解读
尚无解读。
原始英文摘要
arXiv:2610.10088v1 Announce Type: cross Abstract: Self-evolving agents distill task-solving experience into skills for future reuse, but these skills can encode incorrect procedures or non-transferable knowledge. It is therefore critical to verify each skill's reusability: whether its guidance remains useful beyond the experience from which it was distilled. Such verification requires observing how a skill affects execution in new tasks, yet existing tasks may not expose the situations where the target skill can actually be exercised. To construct such situations, we propose SkillSandbox, a framework that dynamically synthesizes a task and its environment for each skill that are skill-relevant yet novel. A Proposer specifies the conditions to preserve and the source-specific details to vary, a Builder constructs an executable scenario, and a Verifier compares executions with and without the skill. The Verifier assesses executability, utility, and efficiency to assign a Keep or Reject verdict, determining whether the skill enters the library. Across ALFWorld and WebShop with three models, SkillSandbox consistently yields the strongest downstream performance and improved execution efficiency. Further analyses examine whether these gains reflect accurate assessment of skill reusability and identify which components of SkillSandbox contribute to them.