跳到正文
返回论文列表
cs.AI提交于 已译

从意图到改进:面向更好模型训练的智能体数据集搜索

From Intent to Improvement: Agentic Dataset Search for Better Model Training

Qiaosheng Chen · Kaijia Huang · Fanqi Zhou · Gong Cheng

中文摘要

在百万级机器学习仓库中寻找训练数据是一个 Web 搜索问题,仅靠文本相关性不够。查询同时耦合语义训练目标与任务、许可证、训练阶段等约束,而用于验证这些约束的元数据常常缺失。文本检索按相似度对数据集卡片排序,但无法强制执行硬约束,也无法沿关系找到描述稀疏的数据集。自由形式的工具调用智能体更灵活,但选择数据集的同一模型也自行宣称匹配,没有任何机制检查查询约束是否真正满足。两种方法都无法保证返回的数据能改进模型训练。为此提出 AgenticDS,一个契约治理的智能体框架,每个语言模型决策先实例化为类型化对象,再由确定性代码消费。该框架将查询编译为类型化搜索计划,通过混合检索与有界图扩展上的可允许确定性算子执行,并在接受排序或输出仅允许局部修复的类型化诊断前,验证基于来源的证据。在 HuDS-Bench 上——一个包含 200 条查询、15 个系统汇总的 24,908 条分级判断的新基准——AgenticDS 的 NDCG@10 比最强稠密-稀疏混合方法提升 25.7%,比商业 LLM 智能体提升 18.9%。一项受控的搜索到训练研究进一步表明,其检索到的数据集在数学、代码和医学领域给出最佳下游准确率。代码与数据公开于 https://github.com/nju-websoft/AgenticDS。

关键要点

  1. 01问题:百万级仓库中按训练目标找数据,文本相关性不足,元数据缺失且约束难验证
  2. 02方法:AgenticDS 将查询编译为类型化搜索计划,用确定性算子执行并验证来源证据
  3. 03结果:HuDS-Bench 上 NDCG@10 比最强混合检索高 25.7%,比商业 LLM 智能体高 18.9%
  4. 04结果:受控搜索到训练研究显示,检索数据在数学、代码、医学下游准确率最佳
  5. 05局限:依赖类型化契约与确定性算子,自由形式智能体的灵活性有所牺牲

解读

尚无解读。

原始英文摘要

arXiv:2610.39608v1 Announce Type: new Abstract: Finding training data in million-scale machine learning repositories is a Web search problem where text relevance is not enough. A query couples a semantic training goal with constraints such as task, license, and training stage, and the metadata that would verify them are often missing. Text retrieval ranks dataset cards by similarity but cannot enforce hard constraints or follow the relations that reach a sparsely described dataset. Free-form tool-calling agents are more flexible, but the same model that picks a dataset also declares it a match, and nothing checks that the query's constraints are actually met. Neither approach establishes that the data it returns will improve model training. To address it, we propose AgenticDS, a contract-governed agentic framework in which every language model decision instantiates a typed object before deterministic code consumes it. The framework compiles a query into a typed search plan, executes it through admissible deterministic operators over hybrid retrieval and bounded graph expansion, and verifies the provenance-grounded evidence before accepting a ranking or emitting a typed diagnosis that permits only local repairs. On HuDS-Bench, a new benchmark of 200 queries with 24,908 graded judgments pooled over 15 systems, AgenticDS improves NDCG@10 by 25.7\% over the strongest dense--sparse hybrid and by 18.9\% over a commercial LLM agent. A controlled search-to-train study further shows that the datasets it retrieves give the best downstream accuracy in mathematics, code, and medicine. Code and data are publicly available at https://github.com/nju-websoft/AgenticDS.

同方向论文 · cs.AI

查看全部 →

反向发现 · 这篇论文相关的工具与提示词

去交叉发现 →

关联提示词 · 6

智能体github.com/f/awesome-chatgpt-prompts·外部收录·

Weekend Planner

- Read the weekend weather, calendar and budget. - Find local events, hikes, exhibitions and restaurants. - Propose 3 plans: adventure, cozy, social, with times and costs. - Prepare reservation steps; book nothing without approval. - Rememb…

研究github.com/f/awesome-chatgpt-prompts·外部收录·

Paper to Practice

You turn research papers into practical takeaways for the user. - Topics are set by the user (edit this list): e.g. retrieval, agents, evaluation. - Each run, find new papers on arXiv and well-known lab blogs. Pick at most 5 that are likel…

智能体github.com/f/awesome-chatgpt-prompts·外部收录·

Task Planner

--- name: sa-plan description: Structured Autonomy Planning Prompt model: Claude Sonnet 4.5 (copilot) agent: agent --- You are a Project Planning Agent that collaborates with users to design development plans. A development plan defines a…