AI systems fail silently far more often than they fail visibly. In a large-scale quantitative analysis of human-AI interactions from the WildChat dataset, we find that 78% of AI failures are invisible: something went wrong but the user gave no overt indication that there was a problem. These invisible failures cluster into eight archetypes that help us characterize where and how AI systems are failing to meet users' needs. In addition, the archetypes show systematic co-occurrence patterns indicating higher-level failure types. To address the question of whether these archetypes will remain relevant as AI systems become more capable, we also assess failures for whether they are primarily interactional or capability-driven, finding that 91% involve interactional dynamics, and we estimate that 94% of such failures would persist even with a more capable model. Finally, we illustrate how the archetypes help us to identify systematic and variable AI limitations across different usage domains. Overall, we argue that our invisible failure taxonomy can be a key component in reliable failure monitoring for product developers, scientists, and policy makers. Our code and data are available at https://github.com/bigspinai/bigspin-invisible-failure-archetypes

翻译：人工智能系统发生静默故障的频率远高于可见故障。通过对WildChat数据集中人机交互的大规模定量分析，我们发现78%的AI故障属于隐形故障：系统出现异常但用户未表现出明显的问题迹象。这些隐形故障可归纳为八种原型，有助于我们刻画AI系统在哪些方面以及如何未能满足用户需求。此外，这些原型显示出系统性的共现模式，揭示了更高层级的故障类型。为探究这些原型在AI系统能力提升后是否仍具相关性，我们进一步评估了故障主要是由交互机制还是能力缺陷驱动，发现91%涉及交互动态，并估算出即使采用更强大的模型，此类故障中仍有94%会持续存在。最后，我们通过案例说明这些原型如何帮助识别不同使用领域中系统性和可变性的AI局限。总体而言，我们认为隐形故障分类体系可为产品开发者、科研人员及政策制定者构建可靠的故障监测机制提供关键支撑。代码与数据详见https://github.com/bigspinai/bigspin-invisible-failure-archetypes