Machine learning systems face diverse threats that undermine robustness, privacy, and fairness. Although many defenses have been proposed, each typically addresses a single risk in isolation. Real-world deployments, however, require these defenses to be composed to meet multiple guarantees simultaneously. The process of composing defenses is complex and not well understood, and its impact on performance and security remains unclear. We present Landseer, a modular framework for integrating machine learning (ML) defenses into the ML lifecycle and systematically evaluating their composition. Landseer encapsulates defenses as containerized modules, allowing existing and new techniques to be plugged in with minimal effort. Its evaluation engine automates experiments across multiple metrics, supporting the study of defenses both individually and in combination. In a preliminary study, we identified 35 state-of-the-art machine learning defenses. After filtering for reproducibility, we analyzed their performance using Landseer's unified evaluation process. Our findings reveal gaps in replicability across defense families and provide insights into the challenges and opportunities in integrating multiple defenses, establishing a foundation for improving the reliability of machine learning systems.

翻译：机器学习系统面临着破坏鲁棒性、隐私性和公平性的多种威胁。尽管已提出众多防御机制，但每种通常仅孤立地应对单一风险。然而，实际部署需要组合这些防御机制以同时满足多重保障。防御机制的组合过程复杂且尚未被充分理解，其对性能和安全性的影响仍不明确。我们提出了Landseer，一个用于将机器学习（ML）防御集成到ML生命周期中并系统评估其组合效果的模块化框架。Landseer将防御机制封装为容器化模块，使现有及新技术能够以最小代价即插即用。其评估引擎可自动化执行跨多指标的实验，支持对防御机制进行单独及组合研究。在初步研究中，我们识别了35种前沿的机器学习防御机制。经可复现性筛选后，我们利用Landseer的统一评估流程分析了其性能。我们的发现揭示了不同防御族在可复现性方面的差距，并为整合多种防御机制面临的挑战与机遇提供了洞见，为提升机器学习系统的可靠性奠定了基础。