跳到正文
返回论文列表
cs.CL提交于 已译

FinVeriQA: 金融研究问答智能体:路由、检索、MCP工具使用与来源感知证据验证

FinVeriQA: Financial Research Agents for Question Answering: Routing, Retrieval, MCP Tool Use, and Provenance-Aware Evidence Verification

Meenakshi Rajpurohit · Anna Phan · Ravikumar Komandur Narayanan · Gunanidhi Ramakrishnan · Nathan Howland

中文摘要

大型语言模型(LLM)能写出流畅的金融分析,但会算错比率、回忆过时价格,并自信地将问题路由到错误工具。我们研究金融问答助手各组件应承担的任务,并用FinVeriQA——一个基于SEC文件和实时市场数据的开源多智能体研究原型——进行验证:紧凑的类型化决策模型(Laya)负责路由问题并过滤建议请求,Python计算所有数字,检索和工具获取证据,Mistral-7B撰写带引用的答案,代码验证器检查引用ID存在且数字与计算笔记匹配,模型上下文协议(MCP)服务器暴露流水线。在实时实验中,Laya路由优于Mistral-7B(0.85 vs. 0.52;总体0.86 vs. 0.68,McNemar p=0.0003);12B Mistral决策无改善,DeepSeek-R1-8B决策更好(0.79 vs. 0.68,p=0.0192)但慢约9.9倍。Mistral-7B在100次金融计算中正确率29%,Python为100%;实时报价将价格误差从210%降至0.00%;混合检索在50个文件问题上达到76% Hit@5。然而在实时案例研究中,两个智能体答案均通过验证,却将AMD样本数据描述为NVIDIA和Alphabet的数据。FinRank基准上的受控上限研究解释了原因:当共享证据中一个数字被破坏时,自一致性保持自信(一致性0.55至0.59,准确率从0.50降至0.00),共享证据检查(如我们的验证器)捕获35%的错误,而对照原始段落检查捕获77%(p=5.96×10⁻⁸);对于DeepSeek-R1-8B,分别为68%和90%。验证必须独立于写作者所见的证据。

关键要点

  1. 01问题:LLM金融分析常算错比率、用旧价格、错误路由工具。
  2. 02方法:多智能体原型FinVeriQA,Laya路由、Python计算、MCP暴露流水线。
  3. 03结果:Laya路由优于Mistral-7B(0.85 vs 0.52),Python计算100%正确。
  4. 04局限:验证器依赖共享证据,仅捕获35%错误,需独立验证。

解读

尚无解读。

原始英文摘要

arXiv:2611.25071v1 Announce Type: new Abstract: Large language models (LLMs) write fluent financial analyses but miscompute ratios, recall stale prices, and route questions to the wrong tool with high confidence. We ask which component should do each job in a financial question-answering assistant, and study it with FinVeriQA, an open-source multi-agent research prototype over SEC filings and live market data: a compact typed-decision model (Laya) routes each question and screens out advice requests, Python computes every number, retrieval and tools fetch evidence, Mistral-7B writes a cited answer, a code validator checks that cited IDs exist and that numbers match computed notes, and a Model Context Protocol (MCP) server exposes the pipeline. In live experiments, Laya routed better than Mistral-7B (0.85 vs. 0.52; overall 0.86 vs. 0.68, McNemar p = 0.0003); a 12B Mistral decided no better, and DeepSeek-R1-8B decided better (0.79 vs. 0.68, p = 0.0192) but about 9.9 times more slowly. Mistral-7B computed 29% of 100 financial calculations correctly against 100% for Python, a live quote cut price error from 210% to 0.00%, and hybrid retrieval reached 76% Hit@5 on 50 filing questions. Yet in a live case study, both agent answers passed validation while describing AMD sample data under NVIDIA's and Alphabet's names. A controlled upper-bound study on the FinRank benchmark explains why: when one figure in the shared evidence is corrupted, self-consistency stays confident (agreement 0.55 to 0.59 while accuracy falls from 0.50 to 0.00), and a shared-evidence check (the kind our validator performs) catches 35% of the errors, whereas a check against the original passage catches 77% ($p = 5.96 \times 10^{-8}$); for DeepSeek-R1-8B, 68% vs. 90%. Verification must be independent of the evidence the writer saw.

同方向论文 · cs.CL

查看全部 →

反向发现 · 这篇论文相关的工具与提示词

去交叉发现 →

关联工具 · 1

Mistral AI71

Mistral 发布 Agents API

Mistral 推出新的 Agents API,将 Mistral 强大的语言模型与内置连接器(代码执行、网络搜索、图像生成、文档库)、MCP 工具支持、持久记忆和智能体编排能力相结合。SimpleQA 基准测试显示,加入 web search 后,Mistral Large 和 Mistral Medium 的得分分别从 23% 和 22.08% 提升至 75% 和 82.32%。

推荐理由:原文给出了 SimpleQA 基准测试的具体分数对比(有无 web search),读者可以据此评估搜索增强对模型实际能力的提升幅度。