Speech language models (SLMs) are systems of systems: independent components that unite to achieve a common goal. Despite their heterogeneous nature, SLMs are often studied end-to-end; how information flows through the pipeline remains obscure. We investigate this question through the lens of backdoor attacks. We first establish that backdoors can propagate through the SLM, leaving all tasks highly vulnerable. From this, we design a component analysis to discover the role each component takes in backdoor learning. We find that backdoor persistence or erasure is highly dependent on the targeted component. Beyond propagation, we examine how backdoors are encoded in shared multitask embeddings, showing that poisoned samples are not directly separable from benign ones, challenging a common separability assumption used in filtering defenses. Our findings emphasize the need to treat multimodal pipelines as intricate systems with unique vulnerabilities, not solely extensions of unimodal ones.

翻译：语音语言模型（SLM）是由独立组件构成的系统之系统，这些组件协同工作以实现共同目标。尽管具有异构特性，SLM常被以端到端的方式研究，但信息在流水线中的流动机制依然模糊。我们通过后门攻击的视角探究这一问题。首先，我们证实后门可在SLM中传播，使所有任务均高度脆弱。基于此，我们设计组件分析以揭示各组件在后门学习中的作用。研究发现，后门的持久性与消除性高度依赖于目标组件。除了传播机制外，我们进一步探究后门如何编码于共享的多任务嵌入中，发现中毒样本与良性样本无法直接分离，挑战了过滤防御中常用的可分离性假设。我们的研究强调，需将多模态流水线视为具有独特脆弱性的复杂系统，而非单模态系统的简单扩展。