In this work, we investigate the interplay between memorization and learning in the context of \emph{stochastic convex optimization} (SCO). We define memorization via the information a learning algorithm reveals about its training data points. We then quantify this information using the framework of conditional mutual information (CMI) proposed by Steinke and Zakynthinou (2020). Our main result is a precise characterization of the tradeoff between the accuracy of a learning algorithm and its CMI, answering an open question posed by Livni (2023). We show that, in the $L^2$ Lipschitz--bounded setting and under strong convexity, every learner with an excess error $\varepsilon$ has CMI bounded below by $\Omega(1/\varepsilon^2)$ and $\Omega(1/\varepsilon)$, respectively. We further demonstrate the essential role of memorization in learning problems in SCO by designing an adversary capable of accurately identifying a significant fraction of the training samples in specific SCO problems. Finally, we enumerate several implications of our results, such as a limitation of generalization bounds based on CMI and the incompressibility of samples in SCO problems.

翻译：本文研究了在随机凸优化（SCO）背景下记忆与学习之间的相互作用。我们通过学习算法所揭示的训练数据点信息来定义记忆，并利用Steinke和Zakynthinou（2020）提出的条件互信息（CMI）框架量化该信息。我们的主要结果是对学习算法精度与其CMI之间权衡的精确刻画，回答了Livni（2023）提出的开放问题。我们证明，在$L^2$ Lipschitz有界条件下以及强凸性假设下，每个超额误差为$\varepsilon$的学习器分别具有下界为$\Omega(1/\varepsilon^2)$和$\Omega(1/\varepsilon)$的CMI。通过设计一个能在特定SCO问题中准确识别显著比例训练样本的对抗器，我们进一步展示了记忆在SCO学习问题中的本质作用。最后，我们列举了结果的若干启示，例如基于CMI的泛化界局限性以及SCO问题中样本的不可压缩性。