跳到正文
返回论文列表
cs.AI提交于 已译

同一问题,不同答案:审计六个 LLM 在戒烟建议中依赖人设的差异

Same Question, Different Answers: Auditing Persona-Dependent Variation in Six LLMs on Smoking-Cessation Advice

Artin Yousefi · Aizierjiang Aiersilan · Lorien C Abroms · David A Broniatowski

中文摘要

数亿人向大型语言模型(LLM)聊天机器人咨询健康建议,由此引出一个简单却难以回答的问题:每个人得到的建议是否相同?针对戒烟这一已有成文临床指南且公共卫生意义重大的领域,审计来自四家提供商的六个广泛部署的 LLM,向十二个在生命阶段、自述专业水平和吸烟史上各不相同的用户人设提出同样的四个戒烟问题,共获得 2,880 条回答。模型给出的内容取决于看起来是谁在提问。每个模型向青少年人设提及 FDA 批准处方戒烟药物的频率都低于成人人设,差距为 32 至 67 个百分点;六个模型中有五个,向自称具备医疗专业知识的人设提供的事实少于不声称具备专业知识的人设。随后评估一种可部署的干预:一段简短的系统提示,规定事实底线,即每个用户在相关时应获得的核心医学事实集合,同时允许语气、词汇和表述自由调整。该提示缩小了每个模型中的人设差异,但其对总体事实覆盖率的影响在幅度和方向上因模型而异,因此在一个系统上验证过的提示不能假定可迁移到另一个系统。两点警示框定这些结果:提及率相等并不总是正确的临床目标,因为处方戒烟药物疗法不建议用于未成年人,所以可辩护的事实底线必须考虑年龄而非统一;且由于人设同时变化多个属性,报告的是场景级差异而非年龄或专业水平的孤立效应。发布全部 2,880 条回答和分析代码。

关键要点

  1. 01审计六个 LLM 对十二种人设的戒烟建议,共 2,880 条回答,发现答案随提问者人设变化。
  2. 02所有模型向青少年人设提及 FDA 批准处方戒烟药的频率比成人低 32 至 67 个百分点。
  3. 03六模型中有五个对自称医疗专业的人设提供的事实更少。
  4. 04规定事实底线的系统提示能缩小人设差异,但对事实覆盖率的影响因模型而异,不可跨系统迁移。
  5. 05事实底线需考虑年龄(处方药不推荐未成年人),且人设多属性同时变化,只能报告场景级差异。

解读

尚无解读。

原始英文摘要

arXiv:2610.42039v1 Announce Type: new Abstract: Hundreds of millions of people now ask large language model (LLM) chatbots for health advice, which raises a question that is simple to pose and hard to answer: does everyone get the same advice? We audit six widely deployed LLMs from four providers on smoking cessation, a domain with codified clini- cal guidance and substantial public-health stakes, by putting the same four cessation questions to twelve user personas that differ in life stage, self-described expertise, and smoking history, for a total of 2,880 responses. What the models say depends on who appears to be asking. Every model named FDA-approved prescription cessation medications less often to teen personas than to adult personas, by 32 to 67 percentage points, and in five of six models a persona presenting health- care expertise received fewer facts than a persona presenting none. We then evaluate one deployable intervention: a short system prompt specifying a factual floor, the set of core med- ical facts every user should receive when they are relevant, while leaving tone, vocabulary, and framing free to adapt. The prompt narrows persona differences in every model, but its effect on overall fact coverage varies in both magnitude and direction across models, so a prompt validated on one system cannot be assumed to transfer to another. Two cautions frame these results. Equal mention rates are not always the correct clinical target, because prescription cessation pharmacother- apy is not recommended for minors, so a defensible factual floor must be age-aware rather than uniform. And because our personas vary on several attributes at once, we report scenario- level differences rather than isolated effects of age or expertise. We release all 2,880 responses and the analysis code.

同方向论文 · cs.AI

查看全部 →

反向发现 · 这篇论文相关的工具与提示词

去交叉发现 →

关联提示词 · 1