Decentralized bilevel optimization has been actively studied in the past few years since it has widespread applications in machine learning. However, existing algorithms suffer from large communication complexity caused by the estimation of stochastic hypergradient, limiting their application to real-world tasks. To address this issue, we develop a novel decentralized stochastic bilevel gradient descent algorithm under the heterogeneous setting, which enjoys a small communication cost in each round and small communication rounds. As such, it can achieve a much better communication complexity than existing algorithms. Moreover, we extend our algorithm to the more challenging decentralized multi-level optimization. To the best of our knowledge, this is the first time achieving these theoretical results under the heterogeneous setting. At last, the experimental results confirm the efficacy of our algorithm.

翻译：去中心化双层优化在过去几年中受到积极研究，因其在机器学习领域具有广泛的应用。然而，现有算法由于随机超梯度估计导致通信复杂度过高，限制了其在实际任务中的应用。为解决这一问题，我们在异质性设置下提出了一种新型去中心化随机双层梯度下降算法，该算法每轮通信成本低且通信轮次少，因此能实现比现有算法更优的通信复杂度。此外，我们将该算法扩展至更具挑战性的去中心化多层优化。据我们所知，这是首次在异质性设置下实现上述理论结果。最后，实验结果验证了我们算法的有效性。