Anthropic 发布 Claude Fable 5.2 与 Mythos 5.2
Anthropic 发布 Claude Fable 5.2 与 Mythos 5.2,官方称 1000 多个 Agent 连续运行 8 天,从空文件夹自主写出可开机的完整操作系统。
推荐理由:材料给出多档位跑分与单任务成本对照,可据此判断新模型在哪些任务上更划算。
E2E-SWE: Benchmarking LLMs on Building Working Codebases from Scratch
Hantian Ding · Chloe Bi · Jiacheng Zhu · John Yang · Matt Deitke · Pengcheng Yin · et al.
由大语言模型驱动的编码代理正从进行局部代码修改发展到开发完整的软件代码库。然而,评估代码库级生成仍具挑战性:任务必须要求系统级推理,同时确保所有评估行为被精确指定且独立于任何特定实现。我们推出E2E-SWE,一个用于评估编码代理能否端到端构建完整功能软件代码库的基准测试。E2E-SWE包含186个整体仓库生成任务,涵盖11种编程语言。给定自然语言规格说明和空工作区后,代理必须实现满足隐藏测试套件的完整可安装项目。每个任务由软件工程师与LLM协作构建,共同开发测试套件和相应实现无关的规格说明。为确保任务定义良好且实际可解,我们进一步对其进行迭代验证过程,其中自主代理使用静态检查和真实模型 rollout 观察到的失败来审计和修复任务缺陷。评估13个前沿模型,发现端到端仓库生成能力存在显著差异,pass@1从11.7%到67.7%,提供了强模型区分度,同时为未来进展留下相当大的提升空间。代理轨迹分析进一步揭示了长前加载推理模式,突出了从零构建可运行代码库所需的规划和系统级推理。
尚无解读。
arXiv:2609.38335v1 Announce Type: cross Abstract: Coding agents powered by large language models (LLMs) are evolving from making localized code changes to developing complete software repositories. However, evaluating repository-scale generation remains challenging: tasks must demand system-level reasoning while ensuring that all evaluated behaviors are precisely specified and independent of any particular implementation. We introduce E2E-SWE, a benchmark for evaluating whether coding agents can build complete, functional software repositories end to end. E2E-SWE contains 186 whole-repository generation tasks spanning 11 programming languages. Given only a natural-language specification and an empty workspace, an agent must implement a complete, installable project that satisfies a comprehensive suite of hidden tests. Each task is constructed by a software engineer in collaboration with an LLM; together, they develop the test suite and a corresponding implementation-independent specification. To ensure that tasks are well specified and practically solvable, we further subject them to an iterative verification process in which autonomous agents audit and repair task defects using static inspection and failures observed from real model rollouts. Evaluating 13 frontier models, we find substantial variation in end-to-end repository generation ability, with pass@1 ranging from 11.7% to 67.7%, providing strong model differentiation while leaving considerable headroom for future progress. Analysis of agent trajectories further reveals long, front-loaded reasoning patterns, highlighting the planning and system-level reasoning required to construct working codebases from scratch.
Anthropic 发布 Claude Fable 5.2 与 Mythos 5.2,官方称 1000 多个 Agent 连续运行 8 天,从空文件夹自主写出可开机的完整操作系统。
推荐理由:材料给出多档位跑分与单任务成本对照,可据此判断新模型在哪些任务上更划算。
Anthropic 在一项内部实验中让 Claude Fable 5.2 仅凭一个基础提示词和空项目文件夹,调度 1000 多个子智能体在八天内完成操作系统的编写、测试与审查,成品可在笔记本上启动,包含自研内核、硬件驱动、持久化文件存储、图形界面、登录界面和安装程序。
推荐理由:Anthropic 内部实验称 Claude Fable 5.2 调度千余子智能体八天写出可启动操作系统,同时点出算力成本与可靠性未披露。