This thesis addresses the challenge of understanding Neural Networks through the lens of their most fundamental component: the weights, which encapsulate the learned information and determine the model behavior. At the core of this thesis is a fundamental question: Can we learn general, task-agnostic representations from populations of Neural Network models? The key contribution of this thesis to answer that question are hyper-representations, a self-supervised method to learn representations of NN weights. Work in this thesis finds that trained NN models indeed occupy meaningful structures in the weight space, that can be learned and used. Through extensive experiments, this thesis demonstrates that hyper-representations uncover model properties, such as their performance, state of training, or hyperparameters. Moreover, the identification of regions with specific properties in hyper-representation space allows to sample and generate model weights with targeted properties. This thesis demonstrates applications for fine-tuning, and transfer learning to great success. Lastly, it presents methods that allow hyper-representations to generalize beyond model sizes, architectures, and tasks. The practical implications of that are profound, as it opens the door to foundation models of Neural Networks, which aggregate and instantiate their knowledge across models and architectures. Ultimately, this thesis contributes to the deeper understanding of Neural Networks by investigating structures in their weights which leads to more interpretable, efficient, and adaptable models. By laying the groundwork for representation learning of NN weights, this research demonstrates the potential to change the way Neural Networks are developed, analyzed, and used.

翻译：本论文从神经网络最基本组成部分——权重的角度出发，探讨理解神经网络的挑战，这些权重封装了学习到的信息并决定了模型行为。本论文的核心是一个根本性问题：我们能否从神经网络模型群体中学习到通用的、与任务无关的表示？为回答该问题，本论文的关键贡献是超表示——一种学习神经网络权重表示的自监督方法。本研究发现，训练后的神经网络模型确实在权重空间中占据有意义的结构，这些结构可以被学习和利用。通过大量实验，本论文证明超表示能够揭示模型特性，例如其性能、训练状态或超参数。此外，在超表示空间中识别具有特定属性的区域，使得采样和生成具有目标属性的模型权重成为可能。本研究展示了超表示在微调和迁移学习中的成功应用。最后，本文提出了使超表示能够泛化到不同模型规模、架构和任务的方法。其实际意义深远，因为它为构建神经网络基础模型打开了大门，这些模型能够跨模型和架构聚合并实例化知识。最终，本论文通过研究神经网络权重中的结构，促进了对神经网络的深入理解，从而产生更具可解释性、高效性和适应性的模型。通过为神经网络权重的表示学习奠定基础，本研究展示了改变神经网络开发、分析和使用方式的潜力。