Bilevel optimization refers to scenarios whereby the optimal solution of a lower-level energy function serves as input features to an upper-level objective of interest. These optimal features typically depend on tunable parameters of the lower-level energy in such a way that the entire bilevel pipeline can be trained end-to-end. Although not generally presented as such, this paper demonstrates how a variety of graph learning techniques can be recast as special cases of bilevel optimization or simplifications thereof. In brief, building on prior work we first derive a more flexible class of energy functions that, when paired with various descent steps (e.g., gradient descent, proximal methods, momentum, etc.), form graph neural network (GNN) message-passing layers; critically, we also carefully unpack where any residual approximation error lies with respect to the underlying constituent message-passing functions. We then probe several simplifications of this framework to derive close connections with non-GNN-based graph learning approaches, including knowledge graph embeddings, various forms of label propagation, and efficient graph-regularized MLP models. And finally, we present supporting empirical results that demonstrate the versatility of the proposed bilevel lens, which we refer to as BloomGML, referencing that BiLevel Optimization Offers More Graph Machine Learning. Our code is available at https://github.com/amberyzheng/BloomGML. Let graph ML bloom.

翻译：双层优化指的是这样一种场景：下层能量函数的最优解作为上层目标函数的输入特征。这些最优特征通常依赖于下层能量的可调参数，从而使得整个双层管道能够以端到端的方式进行训练。尽管通常不以此形式呈现，本文展示了多种图学习技术如何可以被重新解释为双层优化的特例或其简化形式。简言之，基于先前工作，我们首先推导出一类更灵活的能量函数，当与各种下降步骤（如梯度下降、近端方法、动量等）配对时，这些函数构成了图神经网络（GNN）的消息传递层；关键的是，我们还仔细解析了任何残差近似误差在底层消息传递函数中的具体位置。接着，我们探讨了该框架的若干简化形式，以建立与非GNN基图学习方法之间的紧密联系，包括知识图谱嵌入、多种形式的标签传播以及高效的图正则化MLP模型。最后，我们提供了支持性实证结果，展示了所提出的双层视角（我们称之为BloomGML，寓意“双层优化为图机器学习带来更多可能”）的通用性。我们的代码已开源在https://github.com/amberyzheng/BloomGML。愿图机器学习蓬勃发展。