Hypernetworks, or hypernets in short, are neural networks that generate weights for another neural network, known as the target network. They have emerged as a powerful deep learning technique that allows for greater flexibility, adaptability, dynamism, faster training, information sharing, and model compression etc. Hypernets have shown promising results in a variety of deep learning problems, including continual learning, causal inference, transfer learning, weight pruning, uncertainty quantification, zero-shot learning, natural language processing, and reinforcement learning etc. Despite their success across different problem settings, currently, there is no review available to inform the researchers about the developments and to help in utilizing hypernets. To fill this gap, we review the progress in hypernets. We present an illustrative example to train deep neural networks using hypernets and propose categorizing hypernets based on five design criteria as inputs, outputs, variability of inputs and outputs, and architecture of hypernets. We also review applications of hypernets across different deep learning problem settings, followed by a discussion of general scenarios where hypernets can be effectively employed. Finally, we discuss the challenges and future directions that remain under-explored in the field of hypernets. We believe that hypernetworks have the potential to revolutionize the field of deep learning. They offer a new way to design and train neural networks, and they have the potential to improve the performance of deep learning models on a variety of tasks. Through this review, we aim to inspire further advancements in deep learning through hypernetworks.

翻译：超网络（hypernetwork，简称hypernet）是一种为另一神经网络（称为目标网络）生成权重的神经网络。它们已成为一种强大的深度学习技术，能够实现更高的灵活性、适应性、动态性、更快的训练速度、信息共享和模型压缩等。超网络在各种深度学习问题中展现出前景广阔的结果，包括持续学习、因果推断、迁移学习、权重剪枝、不确定性量化、零样本学习、自然语言处理和强化学习等。尽管超网络在不同问题设定下取得了成功，但目前尚无综述性文章向研究者介绍其发展并帮助利用超网络。为填补这一空白，我们回顾了超网络的进展。我们提供了一个利用超网络训练深度神经网络的示例，并基于五个设计标准（即输入、输出、输入输出的可变性以及超网络的架构）对超网络进行分类。我们还回顾了超网络在不同深度学习问题设定中的应用，随后讨论了可有效运用超网络的通用场景。最后，我们探讨了超网络领域尚待探索的挑战和未来方向。我们相信，超网络有潜力彻底改变深度学习领域。它们提供了一种设计和训练神经网络的新方法，并有潜力提升深度学习模型在各种任务上的性能。通过这篇综述，我们旨在通过超网络激发深度学习领域的进一步进展。