应对合作多智能体强化学习中的外生变异性
Addressing Exogenous Variability in Cooperative Multi-Agent Reinforcement Learning
Seongmin Kim · Woohyeon Byeon · Jiwon Jeon · Seungyul Han · Youngchul Sung
中文摘要
合作多智能体强化学习(MARL)在对手策略或环境机制发生偏移等外生变异性下常失效,这类变化不受团队控制但会改变转移动态。该工作将其形式化为外生 Dec-POMDP(ED-POMDP):由独立于团队选择的回合级外生机制索引动态,全局状态被划分为直接响应当前团队动作的分量与当前步不直接响应动作的分量。该形式化揭示中介影响:团队动作当前步不直接更新后者,但可通过前者诱导的变化塑造其后续演化。据此提出 LEICA,一种兼容 CTDE 的算法,为两类分量学习以历史为条件的上下文表示,并用逐智能体、影响加权的反事实内在奖励塑造策略更新。在含对手策略偏移的 SMAX 基准上,LEICA 相比现有基线持续提升训练性能与对未见对手策略的泛化能力,支持外生训练-测试机制偏移下影响加权塑形的有效性。
关键要点
- 01问题:合作 MARL 在对手或环境机制偏移等外生变异性下性能下降,现有方法缺乏形式化刻画。
- 02方法:提出 ED-POMDP,将状态分为动作直接响应与不直接响应两类,并揭示中介影响机制。
- 03方法:LEICA 学习历史条件上下文表示,并用逐智能体影响加权反事实内在奖励塑造策略更新。
- 04结果:在 SMAX 对手策略偏移基准上,训练性能与对未见策略的泛化均优于现有基线。
解读
尚无解读。
原始英文摘要
arXiv:2611.22067v1 Announce Type: cross Abstract: Cooperative multi-agent reinforcement learning (MARL) often fails under exogenous variability such as shifts in opponents or environment regimes that cannot be controlled by the team but alter the transition dynamics. We formalize this challenge as Exogenous Dec-POMDP (ED-POMDP), in which an episode-level exogenous regime, selected independently of the team, indexes the dynamics, and the global state is partitioned into components that are directly responsive to the current team action and components that are not directly action-responsive at the current step. This formulation exposes mediated influence: while team actions do not directly update the latter at the current step, they can shape their later evolution through induced changes in the former. Based on this, we propose LEICA, a CTDE-compatible algorithm that learns history-conditioned context representations for both components and shapes policy updates using agent-wise, influence-weighted counterfactual intrinsic rewards. Across SMAX benchmarks with opponent strategy shifts, LEICA consistently improves both training performance and generalization to unseen opponents' strategies over existing baselines, supporting the usefulness of influence-weighted shaping under exogenous train-test regime shifts.