Machine learning (ML) models memorize and leak training data, causing serious privacy issues to data owners. Training algorithms with differential privacy (DP), such as DP-SGD, have been gaining attention as a solution. However, DP-SGD adds a noise at each training iteration, which degrades the accuracy of the trained model. To improve accuracy, a new family of approaches adds carefully designed correlated noises, so that noises cancel out each other across iterations. We performed an extensive characterization study of these new mechanisms, for the first time to the best of our knowledge, and show they incur non-negligible overheads when the model is large or uses large embedding tables. Motivated by the analysis, we propose Cocoon, a hardware-software co-designed framework for efficient training with correlated noises. Cocoon accelerates models with embedding tables through pre-computing and storing correlated noises in a coalesced format (Cocoon-Emb), and supports large models through a custom near-memory processing device (Cocoon-NMP). On a real system with an FPGA-based NMP device prototype, Cocoon improves the performance by 2.33-10.82x(Cocoon-Emb) and 1.55-3.06x (Cocoon-NMP).

翻译：机器学习（ML）模型会记忆并泄露训练数据，给数据所有者带来严重的隐私问题。采用差分隐私（DP）的训练算法（如DP-SGD）作为一种解决方案日益受到关注。然而，DP-SGD在每次训练迭代时都会添加噪声，这会降低训练模型的准确性。为提高准确性，一类新方法通过添加精心设计的相关噪声，使得噪声在多次迭代间相互抵消。据我们所知，我们首次对这些新机制进行了广泛的特性研究，结果表明当模型规模较大或使用大型嵌入表时，它们会产生不可忽略的开销。基于该分析，我们提出了Cocoon，一个软硬件协同设计的框架，用于高效实现带相关噪声的训练。Cocoon通过以聚合格式预计算并存储相关噪声来加速带有嵌入表的模型（Cocoon-Emb），并通过定制的近内存处理设备支持大型模型（Cocoon-NMP）。在一个配备基于FPGA的NMP设备原型的真实系统上，Cocoon将性能提升了2.33-10.82倍（Cocoon-Emb）和1.55-3.06倍（Cocoon-NMP）。