跳到正文
返回论文列表
cs.MA提交于 已译

MANGO Arena:相同种子,不同链条——在中断的供应链中评估 LLM 智能体

MANGO Arena: Same Seeds, Different Chains Evaluating LLM Agents in a Disrupted Supply Chain

Sara Amar

中文摘要

MANGO Arena 是一个开源测试平台,四个 LLM 智能体各自经营一条串行供应链中的一家公司,该供应链改编自课堂工厂管理游戏 MANGO。环境强制执行硬性规则(卡车数量限制、无信贷现金约束和生产能力),记录每一次违规尝试,并提供一个公开留言板,其中明确禁止私下付款。该环境可同时测量三件事:经济结果(牛鞭效应)、压力下的规则遵守情况以及串谋提议。在试点中,使用三个小型推理模型、两个脚本化参考策略、两种条件(有/无供应商火灾)和三个需求种子,所有 LLM 智能体都放大了需求波动性:在零售商处放大 7 到 25 倍,在供应商处放大 13 到 179 倍,通常超过教科书式的 order-up-to 策略。它们主要通过批量订货实现这一点(最多 60% 的决策为零订单)。真正的规则违规很少(每个模型最多 288 次决策中有 7 次),且没有智能体提出金钱私下交易。试点的主要教训涉及评估方法:API 故障(速率限制和失控的隐藏推理)以“无订单”决策的形式静默进入指标;并且当用相同种子且无失败调用重新运行游戏时,上游牛鞭效应在运行之间变化高达 17 倍。仅用三个种子,关于中断如何影响特定模型的结论无法在重跑中保持。我们完整报告了这些失败,并提出了此类多智能体评估的报告实践。代码、提示和每个记录的决策均已公开。

关键要点

  1. 01构建开源测试平台 MANGO Arena,让四个 LLM 智能体在串行供应链中各经营一家公司,同时测量牛鞭效应、规则遵守和串谋提议。
  2. 02试点中所有 LLM 智能体都放大了需求波动性:零售商处 7-25 倍,供应商处 13-179 倍,常超过教科书式 order-up-to 策略。
  3. 03智能体主要通过批量订货放大波动(最多 60% 决策为零订单),但真实规则违规很少(每模型最多 7/288 次),且无金钱私下交易提议。
  4. 04API 故障(速率限制、隐藏推理失控)被静默记为“无订单”决策,且相同种子重跑时上游牛鞭效应变化高达 17 倍,三个种子下结论不可复现。
  5. 05论文完整报告这些评估失败,并提出多智能体评估的报告实践建议,代码、提示和所有决策日志公开。

解读

尚无解读。

原始英文摘要

arXiv:2611.23613v1 Announce Type: cross Abstract: We introduce MANGO Arena, an open-source testbed in which four LLM agents each run one company in a serial supply chain adapted from MANGO, a classroom factory-management game. The environment enforces hard rules (a truck limit, a no-credit cash constraint and production capacity), logs every blocked attempt to break them, and gives agents a public message board on which side payments are explicitly forbidden. This lets one environment measure three things at once: an economic outcome (the bullwhip effect), rule adherence under pressure, and proposals to collude. In a pilot with three small reasoning models, two scripted reference policies, two conditions (with and without a supplier fire) and three demand seeds, all LLM agents amplified demand variability as it moved upstream, by 7 to 25 times at the Retailer and 13 to 179 times at the Supplier, often more than a textbook order-up-to policy. They did so largely by ordering in batches (zero orders in up to 60% of decisions). Real rule violations were rare (at most 7 of 288 decisions per model) and no agent proposed a money side deal. The pilot's main lesson concerns evaluation method. API failures (rate limits and runaway hidden reasoning) entered the metrics silently as "no order" decisions. And when we reran games with the same seeds and no failed calls, upstream bullwhip changed by up to 17 times between runs. With three seeds, conclusions about how a disruption affects a given model did not survive a rerun. We report these failures in full and propose reporting practices for multi-agent evaluations of this kind. Code, prompts and every logged decision are public.

同方向论文 · cs.MA

查看全部 →