GENSTRAT：迈向大语言模型战略推理的科学
GENSTRAT: Toward a Science of Strategic Reasoning in Large Language Models
Vartan Shadarevian · Kia Ghods · Alex Kenich · Anany Kotawala
中文摘要
固定游戏套件用于评估大语言模型（LLM）的战略推理时已饱和且难以避免污染。GENSTRAT 是一个由程序化生成双人零和不完全信息下注游戏驱动的 LLM 战略能力评估系统，允许评估者按需抽取新颖、新鲜的游戏。每个游戏在五个复杂度轴（状态空间、时间偶然性、对手依赖、私有信息、精度）上被刻画，并通过蒙特卡洛模拟测量。利用该生成器构建了 50 个游戏的基准，并举办十二个前沿与开放权重模型的锦标赛，收集了超过 40,000 场 LLM 对 LLM 对局数据。进一步构建复合复杂度轴，发现在八个最难游戏上，最佳与最差模型之间的差距是整个游戏集上该值的 1.7 倍。消融实验表明，更高的思考预算、公开对手策略、以及与游戏特定需求（私有信息和规划深度）匹配的更高推理，与 LLM 玩家更高的胜率边际相关。
关键要点
- 01固定游戏套件评估 LLM 战略推理已饱和且易受污染，需程序化生成新游戏。
- 02GENSTRAT 按五个复杂度轴生成双人零和不完全信息下注游戏，并用蒙特卡洛模拟测量。
- 03构建 50 游戏基准，十二个模型进行超 40,000 场对局，最难八游戏上模型差距达整体 1.7 倍。
- 04更高思考预算、公开对手策略、匹配游戏需求的推理与更高胜率边际相关。
解读
尚无解读。
原始英文摘要
arXiv:2605.23238v2 Announce Type: replace Abstract: Fixed game suites for evaluating strategic reasoning in large language models (LLMs) saturate and are hard to keep free of contamination. We introduce GENSTRAT, a system for evaluating the strategic capabilities of LLMs driven by procedural generation of two-player zero-sum imperfect-information betting games, allowing an evaluator to draw novel, fresh games on demand. Each game is characterized on a scale of five complexity axes (state space, temporal contingency, opponent dependence, private information, precision) and measured by Monte Carlo simulation. Using this generator, we construct a 50-game benchmark and hold a tournament of twelve frontier and open-weight models, gathering data from over 40,000 LLM vs. LLM matches. We further construct a composite complexity axis, and find that, on the eight hardest games, the gap between the best and worst model is 1.7x its value across the whole set of games. Ablations show that a higher thinking budget, disclosing the opponent's strategy, and higher reasoning matched to a game's specific demands (for private information and planning depth) were associated with higher margins for the LLM player.