BuggyMLE:通过缺陷注入扩展机器学习工程难度
BuggyMLE: Scaling Machine Learning Engineering Difficulty with Bug Injection
John Wu · Rushi Qiang · Yuchen Zhuang · Guowang Li · Girish Narayanswamy · Max Xu · et al.
中文摘要
让 AI 系统自主改进 AI 模型,需要智能体具备机器学习工程(MLE)能力。实践中,设计、训练和评估复杂模型要求修改多文件仓库以复现结果、调试训练运行、集成数据集和移植框架。然而,现有 MLE 基准将智能体限制在单文件编辑,缺乏完整仓库或 shell 访问权限,而常见的仓库级软件工程基准已迅速饱和。为此,引入代码复杂度指数(CCI),一种基于十个结构特征的难度度量,涵盖仓库规模、依赖深度和补丁大小。CCI 能强预测八个公开编码基准的平均解决率(R² = 0.951),识别出补丁大小和复杂度为最重要预测因子。在 CCI 指导下,开发了 BuggyMLE,一个可扩展框架,智能体向可工作的 ML 仓库注入受控突变,基于 222 个源仓库合成超过 6,600 个缺陷注入的 MLE 仓库变体和近 260K 行注入代码。利用 BuggyMLE 做出两项贡献。第一,构建 BuggyMLE-Bench-Lite,首个难度分级的仓库级 MLE-Bench Lite 变体;在 14 个评估模型中,随着难度从最简单到最难层级扩展,最高解决率从 95% 降至 9%。第二,在 200 多个额外 ML 仓库上合成三个难度级别的训练课程,微调 Qwen3.5-9B 模型,使其在 BuggyMLE-Bench-Lite 上媲美前沿模型。训练后的模型可迁移至 ML 之外,将其 SWE-bench Lite 解决率从 24% 提升至 33%,LiveCodeBench v6 pass@1 从 69% 提升至 71%。
关键要点
- 01现有 MLE 基准仅支持单文件编辑且缺乏仓库级访问,仓库级软件工程基准已饱和。
- 02提出代码复杂度指数(CCI),基于十个结构特征预测编码基准解决率,R² 达 0.951。
- 03BuggyMLE 框架基于 222 个源仓库合成超 6,600 个缺陷注入变体和近 260K 行代码。
- 04BuggyMLE-Bench-Lite 上,14 个模型最高解决率从 95% 降至 9%(难度递增)。
- 05微调 Qwen3.5-9B 在 SWE-bench Lite 和 LiveCodeBench v6 上分别提升至 33% 和 71%。
解读
尚无解读。
原始英文摘要
arXiv:2611.25088v1 Announce Type: cross Abstract: Enabling AI systems to autonomously improve AI models requires agents capable of performing machine learning engineering (MLE). In practice, designing, training, and evaluating complex models demands modifying multi-file repositories to reproduce results, debug training runs, integrate datasets, and port frameworks. However, existing MLE benchmarks limit agents to single-file edits without full repository or shell access, and common repository-level software engineering benchmarks have rapidly saturated. To address this, we introduce the Code Complexity Index (CCI), a difficulty metric based on ten structural features spanning repository scale, dependency depth, and patch size. CCI strongly predicts average solve rates across eight public coding benchmarks ($R^2 = 0.951$), identifying patch size and complexity as top predictors. Guided by CCI, we develop BuggyMLE, a scalable framework where agents inject controlled mutations into working ML repositories, synthesizing over 6,600 bug-injected MLE repository variants and nearly 260K lines of injected code based on 222 source repositories. Using BuggyMLE, we make two contributions. First, we construct BuggyMLE-Bench-Lite, the first difficulty-graded, repository-level variant of MLE-Bench Lite; across 14 evaluated models, the top solve rate drops from 95% to 9% as difficulty is scaled from the easiest to the hardest tier. Second, we synthesize a training curriculum across three difficulty levels over 200 additional ML repositories, fine-tuning a Qwen3.5-9B model to rival frontier models on BuggyMLE-Bench-Lite. Our trained model transfers beyond ML, raising its SWE-bench Lite resolve rate from 24% to 33% and its LiveCodeBench v6 pass@1 from 69% to 71%.