跳到正文
返回论文列表
cs.AI提交于 已译

无限制 CTI-REALM 与开放权重模型

Unconstrained CTI-REALM with Open-Weights

Sahil Garg · Ruhi Gottumukkala · Vishal Agarwal · Shubham Vishwakarma

中文摘要

针对安全领域语言模型代理在开放权重与昂贵闭源前沿模型间的选择,本研究在 CTI-REALM 基准上评估了 26 个模型(19 个开放权重,7 个闭源)。主要发现是,若给予足够轮次,开放权重模型能以不到闭源模型十分之一的成本匹敌最强闭源模型。基准发布的消息限制构成混淆变量,导致部分结果仅反映模型是否及时完成任务。放宽限制后,被截断的模型性能恢复,最高奖励归属于开放权重模型。此外,价格并非能力的良好指标,相似奖励的模型成本差异超一个数量级。研究引入阶段级指标,通过执行检测查询评估代理提交内容,发现不同模型在不同阶段表现各异,建议按阶段路由至最佳模型,而非依据总分选择单一模型。输出中用于思考的 token 占比与奖励无相关性。

关键要点

  1. 01在 CTI-REALM 基准上,开放权重模型在足够轮次下能以极低成本匹敌最强闭源模型。
  2. 02基准的消息限制导致结果混淆,放宽限制后开放权重模型获得最高奖励。
  3. 03引入阶段级指标评估代理提交内容,发现不同模型在不同工作流阶段表现各异。
  4. 04模型价格与能力不匹配,相似奖励的模型成本差异巨大,且思考 token 占比与奖励无关。

解读

尚无解读。

原始英文摘要

arXiv:2610.14180v1 Announce Type: new Abstract: Organisations that want language-model agents for security work face a choice between open-weight models, which they can run and audit themselves, and far more expensive closed frontier models. We study that choice on CTI-REALM (arXiv:2603.13517), a benchmark in which an agent turns a threat-intelligence report into a working detection by querying a live telemetry store. We evaluate twenty-six models, nineteen open-weight and seven closed, under a common harness, scorer and panel of LLM judges. Our main finding is that an open-weight model can match the strongest closed models at less than a tenth of their cost, provided it is given enough turns. The benchmark's published limit on agent messages is a confound: a task that exhausts the limit scores zero on every checkpoint, including those it had already passed, so results at that limit partly measure which models finish in time. At the published limit the leading open-weight and closed models cannot be distinguished. With the limit relaxed, the models it was truncating recover, and the highest rewards belong to open-weight models. Among the leading models, price is a poor guide to capability: models with similar rewards differ in cost by more than an order of magnitude. Because the benchmark's checkpoint scores largely record what an agent did or mentioned, we also introduce stage-level metrics that score what the agent submits, by executing its detection queries in the telemetry store and judging its outputs against the answer key. Rather than combining them into one more overall reward, we show that different models lead different stages, which suggests routing each stage of the workflow to the model that performs it best rather than choosing a single model by its overall score. We also report how many tokens each model generates, and find that the share of output spent on thinking does not track the reward.

同方向论文 · cs.AI

查看全部 →

反向发现 · 这篇论文相关的工具与提示词

去交叉发现 →

关联工具 · 1

Google DeepMind89

Google DeepMind 发布 Gemini 4 Carbon 前沿模型

Google DeepMind 向开发者、企业和消费者开放 Gemini 4 Carbon,称其在网络安全与长程智能体工作流上达到前沿水平。

推荐理由:官方披露了 Carbon 在编码、网络安全与科研推理上的基准表现和定价,可据此判断它与 Argon 的分工。

关联提示词 · 5

智能体github.com/f/awesome-chatgpt-prompts·外部收录·

Weekend Planner

- Read the weekend weather, calendar and budget. - Find local events, hikes, exhibitions and restaurants. - Propose 3 plans: adventure, cozy, social, with times and costs. - Prepare reservation steps; book nothing without approval. - Rememb…

智能体github.com/f/awesome-chatgpt-prompts·外部收录·

Update Agent Permissions

# Task: Update Agent Permissions Please analyse our entire conversation and identify all specific commands used. Update permissions for both Claude Code and Gemini CLI. ## Reference Files - Claude: ~/.claude/settings.json - Gemini polic…

智能体github.com/f/awesome-chatgpt-prompts·外部收录·

AI Workflow Automation Specialist

Act as an AI Workflow Automation Specialist. You are an expert in automating business processes, workflow optimization, and AI tool integration. Your task is to help users: - Identify processes that can be automated - Design efficient work…

智能体github.com/f/awesome-chatgpt-prompts·外部收录·

AI Agent Security Evaluation Checklist

Act as an AI Security and Compliance Expert. You specialize in evaluating the security of AI agents, focusing on privacy compliance, workflow security, and knowledge base management. Your task is to create a comprehensive security evaluati…