Machine learning models are famously vulnerable to adversarial attacks: small ad-hoc perturbations of the data that can catastrophically alter the model predictions. While a large literature has studied the case of test-time attacks on pre-trained models, the important case of attacks in an online learning setting has received little attention so far. In this work, we use a control-theoretical perspective to study the scenario where an attacker may perturb data labels to manipulate the learning dynamics of an online learner. We perform a theoretical analysis of the problem in a teacher-student setup, considering different attack strategies, and obtaining analytical results for the steady state of simple linear learners. These results enable us to prove that a discontinuous transition in the learner's accuracy occurs when the attack strength exceeds a critical threshold. We then study empirically attacks on learners with complex architectures using real data, confirming the insights of our theoretical analysis. Our findings show that greedy attacks can be extremely efficient, especially when data stream in small batches.

翻译：机器学习模型以其易受对抗性攻击而闻名：数据的微小特定扰动可能灾难性地改变模型预测。尽管大量文献研究了预训练模型在测试时攻击的情况，但在在线学习环境中攻击的重要场景迄今未得到足够关注。本研究采用控制理论视角，探讨攻击者可能通过扰动数据标签来操纵在线学习者学习动态的场景。我们在教师-学生框架下对该问题进行了理论分析，考虑了不同的攻击策略，并获得了简单线性学习者稳态的解析结果。这些结果使我们能够证明，当攻击强度超过临界阈值时，学习者的准确率会出现不连续跃迁。随后，我们使用真实数据对具有复杂架构的学习者进行了实证攻击研究，验证了理论分析的洞见。研究结果表明，贪婪攻击可能极为高效，尤其是在数据以小批量流式传输时。