cs.IR提交于 2026-10-21已译

MOSS: 用于监控和诊断搜索与推荐系统中语义故障的生产框架

MOSS: A Production Framework for Monitoring and Diagnosing Semantic Failures in Search and Recommendation Systems

Alina Nesen · Prathyusha Kanmanth Reddy · David Relyea · Rohit Avinash Mujumdar · Samaneh Karimi · Bo Shen · et al.

中文摘要

生产环境中推荐与搜索系统的语义故障检测困难，行为指标滞后且人工审查难以扩展。MOSS 框架将基于 LLM 的相关性评分嵌入持续监控循环，结合定向采样、策略感知提示及结构化告警，将故障归类为可操作的根因。该框架在美国大型零售商的生产环境中运行超过六个月，覆盖推荐、搜索及优惠界面，并发现两个已确认的服务缺陷。针对其中一项回填混合缺陷，MOSS 离线量化修复方案使告警率相对降低 76.4%。在 498 对人工标注的推荐基准上，前沿 LLM 评判器在 F1 和 AUC 上优于非 LLM 基线（内部生产嵌入和 SBERT），在需要意图级关系推理的任务上增益最大。同一评判器、评分尺度及阈值仅需表面特定提示适配即可迁移至涵盖 21 个查询的 466 对搜索基准，无需重新调参即实现强 F1 和 AUC 表现。

关键要点

01解决生产环境语义故障检测滞后及人工审查不可扩展的问题，填补离线评估框架在持续监控中的空白。
02构建包含定向采样、策略感知提示及诊断跟进的 LLM 监控循环，将故障归类为可操作的根因。
03在大型零售商生产环境运行超六个月，发现两个服务缺陷，其中一项修复使告警率相对降低 76.4%。
04LLM 评判器在推荐基准上 F1 和 AUC 优于非 LLM 基线，且无需重新调参即可迁移至搜索基准。

解读

尚无解读。

原始英文摘要

arXiv:2610.26902v1 Announce Type: cross Abstract: Detecting semantic failures in production recommendation and search systems is hard: behavioral metrics lag by days, manual review does not scale, and most LLM evaluation frameworks are designed for offline, fixed datasets rather than continuous operational monitoring. We present Model Sanity Service (MOSS), a production framework that addresses this operational gap by embedding LLM-based relevance scoring into a recurring monitoring loop with targeted sampling, strategy-aware prompting, structured alerting, and diagnostic follow-up prompting that categorizes failures into actionable root causes. MOSS has been running in production at a large U.S. retailer for over six months across recommendation, search, and offer surfaces, and has surfaced two confirmed serving defects. For one of these, a backfill blending defect, MOSS quantified the impact of the proposed fix offline as a 76.4\% relative reduction in alert rate. On a 498-pair human-annotated recommendation benchmark, frontier LLM judges achieve higher F1 and AUC point estimates than non-LLM baselines (internal production embeddings and SBERT), with the largest gains on strategy-dependent tasks requiring intent-level relational reasoning. The same judge, score scale, and threshold transfer to a 466-pair search benchmark spanning 21 queries with only a surface-specific prompt adaptation, achieving strong F1 and AUC without retuning. These results show that a single LLM monitoring framework can provide useful, actionable quality signals across multiple product discovery surfaces in a large-scale e-commerce setting.