In today's machine learning (ML) models, any part of the training data can affect its output. This lack of control for information flow from training data to model output is a major obstacle in training models on sensitive data when access control only allows individual users to access a subset of data. To enable secure machine learning for access controlled data, we propose the notion of information flow control for machine learning, and develop a secure Transformer-based language model based on the Mixture-of-Experts (MoE) architecture. The secure MoE architecture controls information flow by limiting the influence of training data from each security domain to a single expert module, and only enabling a subset of experts at inference time based on an access control policy. The evaluation using a large corpus of text data shows that the proposed MoE architecture has minimal (1.9%) performance overhead and can significantly improve model accuracy (up to 37%) by enabling training on access-controlled data.

翻译：当今机器学习（ML）模型中，训练数据的任何部分都可能影响其输出结果。当访问控制机制仅允许单个用户访问数据子集时，这种训练数据到模型输出的信息流缺乏控制，成为在敏感数据上训练模型的主要障碍。为实现受控数据的安全机器学习，我们提出了针对机器学习的信息流控制概念，并基于混合专家（MoE）架构开发了安全的Transformer语言模型。该安全MoE架构通过将每个安全域的训练数据影响力限制在单一专家模块内，并在推理阶段根据访问控制策略仅启用部分专家模块，从而实现信息流控制。基于大型文本数据语料的评估表明，所提出的MoE架构仅产生1.9%的性能开销，同时通过支持在受控数据上训练，可使模型准确率显著提升（最高达37%）。