We develop a thermodynamic theory for machine learning (ML) systems. Similar to physical thermodynamic systems which are characterized by energy and entropy, ML systems possess these characteristics as well. This comparison inspire us to integrate the concept of temperature into ML systems grounded in the fundamental principles of thermodynamics, and establish a basic thermodynamic framework for machine learning systems with non-Boltzmann distributions. We introduce the concept of states within a ML system, identify two typical types of state, and interpret model training and refresh as a process of state phase transition. We consider that the initial potential energy of a ML system is described by the model's loss functions, and the energy adheres to the principle of minimum potential energy. For a variety of energy forms and parameter initialization methods, we derive the temperature of systems during the phase transition both analytically and asymptotically, highlighting temperature as a vital indicator of system data distribution and ML training complexity. Moreover, we perceive deep neural networks as complex heat engines with both global temperature and local temperatures in each layer. The concept of work efficiency is introduced within neural networks, which mainly depends on the neural activation functions. We then classify neural networks based on their work efficiency, and describe neural networks as two types of heat engines.

翻译：我们为机器学习系统发展了一种热力学理论。与以能量和熵为特征的物理热力学系统类似，机器学习系统也具有这些特征。这一比较启发我们基于热力学基本定律将温度概念引入机器学习系统，并为具有非玻尔兹曼分布的机器学习系统建立了基本的热力学框架。我们引入了机器学习系统中状态的概念，识别了两种典型的状态类型，并将模型训练与刷新解释为状态相变过程。我们认为机器学习系统的初始势能由模型的损失函数描述，且能量遵循最小势能原理。针对多种能量形式和参数初始化方法，我们通过解析和渐近方式推导了系统在相变过程中的温度，强调了温度作为系统数据分布和机器学习训练复杂性的关键指标。此外，我们将深度神经网络视为具有全局温度和每层局部温度的复杂热机。在神经网络中引入了工作效率的概念，该效率主要取决于神经激活函数。随后，我们根据工作效率对神经网络进行分类，并将神经网络描述为两种类型的热机。