跳到正文
返回论文列表
cs.LG提交于 已译

通过残差学习分配改进多阶段任务的模仿策略

Improving Imitation Policies for Multi-Phase Tasks through Residual Learning Allocation

Clinton Enwerem · Saleh Kemal · John S. Baras · Calin Belta

中文摘要

残差强化学习(residual reinforcement learning)利用有限的环境交互预算来修正模仿学习训练的基础策略。在多阶段任务中,该预算需在各阶段间分配,而基础策略失败的阶段未必是修正后最能提升任务完成率的阶段。通过完成增益(completion gain)形式化这一区别:在单个阶段内,用状态反馈控制器替换基础策略动作的选定分量时,全任务完成概率的变化。将完成增益分解为到达该阶段的概率与该阶段起始处完成的条件改进,并从配对回合中估计。正估计值决定残差策略修正哪些动作分量以及训练回合在各阶段间的分布。在随机点质量对接实验中,多数失败发生在最终对接阶段,而最大完成增益出现在前一通道阶段;基于增益的分配相比按失败计数比例的分配提升了完成率,在更长接近过程的变体中,以一半预算优于均匀分配。在模拟双臂机器人装配中,完成增益估计仅在插入和释放阶段为正,尽管释放阶段失败很少;在第二种装配配置中,在释放阶段训练的残差策略使从释放起始处留出状态开始的完成率提升了16.9%。

关键要点

  1. 01提出完成增益指标,量化单阶段修正对全任务完成概率的提升
  2. 02将完成增益分解为到达阶段概率与条件改进,并用配对回合估计
  3. 03点质量对接实验中最大增益在前一通道阶段而非失败最多的对接阶段
  4. 04基于增益的分配优于按失败计数比例分配,半预算下优于均匀分配
  5. 05双臂装配中释放阶段失败少但增益为正,残差策略使完成率提升16.9%

解读

尚无解读。

原始英文摘要

arXiv:2611.25320v1 Announce Type: cross Abstract: Residual reinforcement learning corrects an imitation-trained base policy using a limited budget of environment interaction. In a task with several phases, this budget must be divided among the phases, and the phase in which the base policy fails is not necessarily the phase in which a correction most improves task completion. We formalize this distinction through the completion gain: the change in full-task completion probability when, during a single phase, a state-feedback controller replaces selected components of the base policy's action. We decompose the completion gain into the probability of reaching the phase and the conditional improvement in completion from the start of the phase, and we estimate it from paired episodes. The positive estimates determine which action components the residual policy corrects and how training episodes are distributed across phases. In stochastic point-mass docking experiments, most failures occurred in the final docking phase, whereas the largest completion gain occurred in the preceding channel phase; gain-based allocation improved completion over allocation proportional to failure counts and, in a variant with a longer approach, over uniform allocation at half of the budget. In simulated bimanual robotic assembly, the completion-gain estimates were positive only for insertion and release, although few failures occurred during release; in a second assembly configuration, a residual policy trained in the release phase improved completion from held-out states at the start of release by 16.9%.

同方向论文 · cs.LG

查看全部 →