人类能分辨吗？人类对 LLM 生成新闻感知的双轴研究
Can Humans Tell? A Dual-Axis Study of Human Perception of LLM-Generated News
Alexander Loth · Martin Kappes · Marc-Oliver Pahl
中文摘要
人类能否分辨一篇新闻文章由人还是大语言模型（LLM）撰写？通过 JudgeGPT 平台，在连续量表上独立测量来源归因（人类 vs. 机器）与真实性判断（合法 vs. 虚假）。从 1,054 名参与者对六个 LLM 生成内容的 2,318 次判断中，报告五项发现：(1) 参与者无法可靠区分机器生成与人类撰写文本（p > .05，Welch's t 检验）；(2) 该无法区分现象在所有测试模型上均成立，包括参数低至 7B 的开源权重模型；(3) 自报领域专业知识预测判断准确率（r = .35，p < .001），而政治倾向不预测（r = -.10，n.s.）；(4) 聚类揭示出不同响应策略（“怀疑者” vs. “相信者”）；(5) 约 30 次连续评估后，因认知疲劳准确率下降。简言之，答案是否定的：人类无法可靠分辨。结果表明用户侧检测不是可行防御，需推动系统级对策如加密内容溯源。
关键要点
- 01人类无法可靠区分 LLM 生成与人类撰写的新闻（p > .05）
- 02该无法区分现象在所有测试模型上成立，包括 7B 参数开源模型
- 03自报领域专业知识预测判断准确率（r = .35），政治倾向不预测
- 04聚类揭示“怀疑者”与“相信者”两种响应策略
- 05约 30 次连续评估后因认知疲劳准确率下降
解读
尚无解读。
原始英文摘要
arXiv:2604.03755v2 Announce Type: replace-cross Abstract: Can humans tell whether a news article was written by a person or a large language model (LLM)? We investigate this question using JudgeGPT, a study platform that independently measures source attribution (human vs. machine) and authenticity judgment (legitimate vs. fake) on continuous scales. From 2,318 judgments collected from 1,054 participants across content generated by six LLMs, we report five findings: (1) participants cannot reliably distinguish machine-generated from human-written text (p > .05, Welch's t-test); (2) this inability holds across all tested models, including open-weight models with as few as 7B parameters; (3) self-reported domain expertise predicts judgment accuracy (r = .35, p < .001) whereas political orientation does not (r = -.10, n.s.); (4) clustering reveals distinct response strategies ("Skeptics" vs. "Believers"); and (5) accuracy degrades after approximately 30 sequential evaluations due to cognitive fatigue. The answer, in short, is no: humans cannot reliably tell. These results indicate that user-side detection is not a viable defense and motivate system-level countermeasures such as cryptographic content provenance.