Online prediction from experts is a fundamental problem in machine learning and several works have studied this problem under privacy constraints. We propose and analyze new algorithms for this problem that improve over the regret bounds of the best existing algorithms for non-adaptive adversaries. For approximate differential privacy, our algorithms achieve regret bounds of $\tilde{O}(\sqrt{T \log d} + \log d/\varepsilon)$ for the stochastic setting and $\tilde O(\sqrt{T \log d} + T^{1/3} \log d/\varepsilon)$ for oblivious adversaries (where $d$ is the number of experts). For pure DP, our algorithms are the first to obtain sub-linear regret for oblivious adversaries in the high-dimensional regime $d \ge T$. Moreover, we prove new lower bounds for adaptive adversaries. Our results imply that unlike the non-private setting, there is a strong separation between the optimal regret for adaptive and non-adaptive adversaries for this problem. Our lower bounds also show a separation between pure and approximate differential privacy for adaptive adversaries where the latter is necessary to achieve the non-private $O(\sqrt{T})$ regret.

翻译：基于专家意见的在线预测是机器学习中的一个基本问题，已有若干研究在隐私约束下对该问题进行了探讨。我们提出并分析了该问题的新算法，这些算法针对非适应性对手改进了现有最佳算法的遗憾界。在近似差分隐私条件下，我们的算法在随机设置中实现了$\tilde{O}(\sqrt{T \log d} + \log d/\varepsilon)$的遗憾界，在遗忘对手设置中实现了$\tilde O(\sqrt{T \log d} + T^{1/3} \log d/\varepsilon)$（其中$d$为专家数量）。在纯差分隐私条件下，我们的算法首次在高维场景$d \ge T$中针对遗忘对手实现了次线性遗憾。此外，我们证明了针对适应性对手的新下界。结果表明，与非隐私设置不同，该问题在适应性与非适应性对手之间存在显著的遗憾最优分离。我们的下界还揭示了适应性对手场景下纯差分隐私与近似差分隐私之间的分离，其中后者是实现非隐私设置中$O(\sqrt{T})$遗憾界的必要条件。