cs.LG提交于 2026-10-23已译

面向成本感知的 LLM 拒答式 IT 工单分类

Cost-Aware Abstention for LLM-Based IT Ticket Classification

Sachin Gupta

中文摘要

将 IT 支持工单错误自动路由的成本，可能高于人工分流。研究将成本感知拒答（cost-aware abstention）作为离线、按成本条件进行的就绪度评估：先在明确成本下用校准数据选择路由策略，再将冻结后的策略仅在留出测试数据上评估一次。在一个公开的 47,837 条、八分类工单语料库上，对比了四种训练型非 LLM 分类器与四种大语言模型，并采用零样本和少样本提示。在示例成本（c_ok, c_triage, c_wrong）=（0, 1, 5）下，TF-IDF 配合逻辑回归及校准集选定的 0.60 阈值，将测试成本从 0.763 降至 0.472，相对降低 38.1%（配对比例 bootstrap 95% CI 为 [35.7%, 40.4%]），错误路由率从 15.3% 降至 3.5%。事后成本参数分析中，冻结的校准选定策略在 16 种设置中的 14 种优于始终路由和始终转人工。最强的提示式配置在分层抽取的 1,000 条工单测试子集上达到 0.477 准确率，而同一批工单上的微调 DistilBERT 为 0.879。等渗校准（isotonic）降低了期望校准误差，但未可靠改善区分能力：GPT-4o-mini 少样本在校准前后仅有一个置信度值，正确性 AUROC 为 0.500。因此，研究同时报告正确性 AUROC 和考虑并列情况的风险—覆盖面积。在解析推导出的盈亏平衡概率 0.80 处，七种提示式配置会将全部工单转交人工；Haiku 少样本仅路由 19/1,000 条。仅依赖 LLM 一致性的对照，以及基于边际和熵的拒答基线，进一步区分了校准与区分能力。结论仅适用于该数据集、提示设计、模型端点和示例成本；该工作是评估框架，而非新的拒答算法或通用部署认证。

关键要点

01错误自动路由可能比人工分流成本更高，需按成本条件评估拒答策略。
02TF-IDF 与逻辑回归将测试成本从 0.763 降至 0.472，错误路由率降至 3.5%。
03冻结的校准选定策略在 16 种设置中的 14 种优于始终路由和始终转人工。
04等渗校准降低 ECE，但未可靠改善区分能力，需同时报告 AUROC 与风险—覆盖面积。
05结论受限于特定数据集、提示设计、模型端点和示例成本，不构成通用部署认证。

解读

尚无解读。

原始英文摘要

arXiv:2610.31657v1 Announce Type: new Abstract: Wrong automatic routing of an IT support ticket can cost more than human triage. We evaluate cost-aware abstention as an offline, cost-conditional readiness assessment: select a routing policy on calibration data under explicit costs, then evaluate that frozen policy once on held-out test data. On a public 47,837-ticket, eight-class corpus, we compare four trained non-LLM classifiers with four large language models under zero- and few-shot prompting. Under the illustrative cost model (c_ok, c_triage, c_wrong) = (0, 1, 5), TF-IDF with logistic regression and a threshold of 0.60 selected on calibration reduces test cost from 0.763 to 0.472, a 38.1% relative reduction (paired ratio-bootstrap 95% CI [35.7%, 40.4%]), while lowering wrong routes from 15.3% to 3.5%. Frozen calibration-selected policies also beat both always-route and always-defer on held-out test data in 14 of 16 settings in a post-hoc cost-parameter analysis. The strongest prompted configuration reaches 0.477 accuracy on a stratified 1,000-ticket test subset, versus 0.879 for fine-tuned DistilBERT on the same tickets. Isotonic lowers expected calibration error without reliably improving discrimination: GPT-4o-mini few-shot has one confidence value before and after recalibration and correctness AUROC 0.500. We therefore report correctness AUROC and tie-aware risk-coverage area alongside ECE. At the analytically derived break-even probability 0.80, seven prompted configurations defer all tickets; Haiku few-shot routes only 19/1,000. An LLM-only agreement control and margin/entropy rejection baselines further separate calibration from discrimination. The findings are scoped to this dataset, prompt design, model endpoints, and illustrative costs; the work is an evaluation framework, not a new abstention algorithm or universal deployment certification.