Speech language models (SLMs) are systems of systems: independent components that unite to achieve a common goal. Despite their heterogeneous nature, SLMs are often studied end-to-end; how information flows through the pipeline remains obscure. We investigate this question through the lens of backdoor attacks. We first establish that backdoors can propagate through the SLM, leaving all tasks highly vulnerable. From this, we design a component analysis to reveal the role each component takes in backdoor learning. We find that backdoor persistence or erasure is highly dependent on the targeted component. Beyond propagation, we examine how backdoors are encoded in shared multitask embeddings, showing that poisoned samples are not directly separable from benign ones, challenging a common separability assumption used in filtering defenses. Our findings emphasize the need to treat multimodal pipelines as intricate systems with unique vulnerabilities, not solely extensions of unimodal ones.

翻译：语音语言模型（SLMs）是系统的系统：多个独立组件共同协作以实现统一目标。尽管具有异构特性，SLMs 通常以端到端方式被研究，信息在流水线中的流动路径仍不明确。本文通过后门攻击的视角探究该问题。我们首先证实后门可在 SLM 中传播，导致所有任务高度脆弱。基于此，我们设计组件分析揭示各组件在后门学习中的作用，发现后门的持续存在或消除高度依赖于目标组件。除传播外，我们还探讨了后门在共享多任务嵌入中的编码方式，表明中毒样本无法直接与良性样本分离，挑战了过滤防御中常用的可分离性假设。研究结果强调，需将多模态流水线视为具有独特漏洞的复杂系统，而非单模态系统的简单延伸。