跳到正文
返回论文列表
cs.CL提交于 已译

LLMs 已是置信度感知的零样本分类器

LLMs Are Already Confidence-Aware Zero-Shot Classifiers

Owen Cook · Xingyi Song

中文摘要

零样本大型语言模型(LLM)越来越多地被用作文本分类器,但只有当知道其标签可信程度时,这些标签才有用。比较黑盒方法(言语化置信度和采样)与读取 token 概率的白盒方法(P(True),以及使用约束生成的多种 logit 评分方法)。在五个分类数据集(3 到 151 个标签)上评估 11 个开放权重 LLM(1B-32B 参数),测量分类性能(准确率、F1-macro)、区分度(AUROC)和校准(ECE 与 Brier 分数)。置信度方法的选择很少显著影响准确率,但基于 logit 的评分方法比黑盒方法带来好得多的区分度(AUROC 0.74-0.77 对 0.57-0.66)。用单 token 标识符替换标签可保持区分度,但平均损失 3-6 个准确率点。还引入仅决策评分,仅对前缀树(trie)中标签分叉的位置评分。在三个最大标签集上减少 88-96% 的解码步骤,性能与全标签评分相当。最佳开放权重模型-方法配置(Gemma 4 31B 和 Qwen 3.5 9B)与商业 Jev 1.13.0 分类器具有竞争力。

关键要点

  1. 01比较黑盒与白盒置信度方法在零样本 LLM 分类中的表现
  2. 02基于 logit 的评分方法区分度(AUROC 0.74-0.77)显著优于黑盒方法(0.57-0.66)
  3. 03单 token 标识符替换标签保持区分度但平均损失 3-6 个准确率点
  4. 04仅决策评分在前缀树分叉处评分,减少 88-96% 解码步骤且性能相当
  5. 05最佳开放权重配置与商业 Jev 1.13.0 分类器具有竞争力

解读

尚无解读。

原始英文摘要

arXiv:2611.25289v1 Announce Type: cross Abstract: Zero-shot large language models (LLMs) are increasingly used as text classifiers, but their labels are only useful when we know to what extent we can trust them. We compare black-box methods (verbalised confidence and sampling) with white-box methods that read token probabilities (P(True), and multiple logit-scoring methods using constrained generation). We evaluate eleven open-weight LLMs (1B-32B parameters) on five classification datasets with 3 to 151 labels, measuring classification performance (accuracy, F1-macro), discrimination (AUROC), and calibration (ECE and Brier score). The choice of confidence method rarely impacts accuracy significantly, but logit-based scoring methods lead to much better discrimination (AUROC 0.74-0.77) than black-box methods (AUROC 0.57-0.66). Replacing labels with single-token identifiers preserves discrimination but costs 3-6 accuracy points on average. We also introduce decision-only scoring, which only scores positions where labels diverge in the prefix tree (trie). It reduces decode steps by 88-96% on the three largest label sets and performs comparably to full-label scoring. The best open-weight model-method configurations (Gemma 4 31B and Qwen 3.5 9B) are competitive with the commercial Jev 1.13.0 classifier.

同方向论文 · cs.CL

查看全部 →