跳到正文
返回论文列表
cs.SE提交于 已译

超越 QASecClaw:软件保障中的智能体扩展、验证器反馈与证据可靠性

Beyond QASecClaw: Agent Scaling, Verifier Feedback, and Evidence Reliability in Software Assurance

S M Abdulla Al Mamun · Akif Islam · Mohd. Ruhul Ameen

中文摘要

研究探讨在已有确定性工具证据时,增加专家语言模型上下文是否提升软件保障。实验涵盖 Java 警告控制、OWASP Benchmark、Defects4J 测试生成、SV-COMP 断言判定及误导性证据包,使用 Qwen2.5-Coder-7B-Instruct 完成 13,638 次请求。在 8 个开发警告中,上下文从 1 增至 50 使 token 用量增加 49 倍,但未改善不安全清除率。160 个 OWASP 案例中,10 上下文配置仅评估 96 例。32 个 SV-COMP 任务中,原始验证器反馈将平均正确判定从 12.7 提至 22.0,但错误判定从 1 增至 7;JBMC 加 JVM 基线返回 26 个正确判定。在 120 个证据包中,模型拒绝 70-100% 的反例,而范围检查正确分类所有包。Defects4J 中未发现目标缺陷。结果支持确定性证据准入,但暴露了判定聚合与反馈解释的局限。

关键要点

  1. 01增加语言模型上下文数量至 50 倍时,token 消耗激增但未提升不安全代码清除率。
  2. 02原始验证器反馈提升 SV-COMP 正确判定数,但显著增加错误判定,且存在未解决结果。
  3. 03模型在误导性证据包中拒绝大部分反例,而基于构造的范围检查能正确分类所有证据包。
  4. 04在 Defects4J 测试中未发现目标缺陷,部分受限于测试适配器支持不足。

解读

尚无解读。

原始英文摘要

arXiv:2610.36947v1 Announce Type: cross Abstract: Can additional specialist language-model contexts improve software assurance when deterministic tool evidence is already available? We study this question through five experiments spanning authored Java warning controls, OWASP Benchmark assessment, Defects4J test generation, SV-COMP assertion verdicts, and deliberately misleading evidence packages. The evaluated systems use Qwen2.5-Coder-7B-Instruct, with a 1.5B sensitivity analysis, and complete 13,638 model requests. On eight related development warnings, increasing the context count from 1 to 50 increases token use 49-fold without improving unsafe clearance. On 160 OWASP cases, model configurations trade clearance against unresolved outcomes; the ten-context arm assesses only 96 cases. On 32 SV-COMP tasks, raw verifier feedback raises mean correct model verdicts from 12.7 to 22.0, but also raises incorrect verdicts from one to seven; a JBMC-plus-JVM validation baseline returns 26 correct verdicts and six unresolved outcomes. On two sets of 120 evidence packages, models reject 70-100% of applicable counterexamples, while construction-specific scope checks classify every package correctly. No target defect is discovered in three Defects4J bugs, one unsupported by the test adapter. These descriptive results support deterministic evidence admission for the tested settings, while exposing limits of verdict aggregation, feedback interpretation, and the available evaluation coverage.

同方向论文 · cs.SE

查看全部 →

反向发现 · 这篇论文相关的工具与提示词

去交叉发现 →

关联提示词 · 6

编程tech.qimao.com·外部收录·

按规范检查 Go 代码改动,分析问题与服务影响并生成 Code Review 报告

``` # Code Review 规范 你是 golang 编程的专家,现在对代码进行 Code Review,请按照以下步骤进行: (若遇到 git 命令分页的情况,请加上`| cat`) ## 步骤一:代码规范检查 - [ ] 是否遵循最佳实践 @.cursor/rules/rule-code-spec.mdc - [ ] 是否存在重复代码 - [ ] 是否有充分的注释和文档 - [ ] 是否易于理解和维护 ## 步骤二:功能问题分析 - [ ] 分析本次代码改动的影响…

编程zhuanlan.zhihu.com·外部收录·

用于指导 Claude Code 按阶段规划、测试驱动实现、重构并提交代码。

### 1. 规划与分阶段 将复杂的工作分解为 3-5 个阶段。在 `IMPLEMENTATION_PLAN.md` 中记录: ```text ## 阶段 N: [Name] **目标**: [具体可交付成果] **成功标准**: [可测试的结果] **测试**: [具体的测试用例] **状态**: [未开始|进行中|已完成] ``` * 随着进展更新状态 * 所有阶段完成后删除文件 ### 2. 实现流程 1. **理解** - 研究代码库中的现有模式 2. **测试**…

编程github.com/f/awesome-chatgpt-prompts·外部收录·

debug

Please debug the current codebase and implement a robust development plan to ensure the system is fully functional. Prioritize refactoring the application architecture to follow industry best practices, resolve existing build errors, and en…

编程github.com/f/awesome-chatgpt-prompts·外部收录·

Python Auto Typer with Customizable Delay

Act as a Python Software Developer. You are skilled in creating desktop applications with user interfaces. Your task is to develop a Python program that: - Automatically types a specified text and presses "Enter" after each message to send…