The significance of statistical physics concepts such as entropy extends far beyond classical thermodynamics. We interpret the similarity between partitions in statistical mechanics and partitions in Bayesian inference as an articulation of a result by Jaynes (1957), who clarified that thermodynamics is in essence a theory of information. In this, every sampling process has a mechanical analogue. Consequently, the divide between ensembles of samplers in parameter space and sampling from a mechanical system in thermodynamic equilibrium would be artificial. Based on this realisation, we construct a continuous modelling of a Bayes update akin to a transition between thermodynamic ensembles. This leads to an information theoretic interpretation of Jazinsky's equality, relating the expenditure of work to the influence of data via the likelihood. We propose one way to transfer the vocabulary and the formalism of thermodynamics (energy, work, heat) and statistical mechanics (partition functions) to statistical inference, starting from Bayes' law. Different kinds of inference processes are discussed and relative entropies are shown to follow from suitably constructed partitions as an analytical formulation of sampling processes. Lastly, we propose an effective dimension as a measure of system complexity. A numerical example from cosmology is put forward to illustrate these results.

翻译：统计物理学中熵等概念的重要性远超出经典热力学范畴。我们将统计力学中的配分与贝叶斯推断中的划分之间的相似性，阐释为对Jaynes（1957）研究成果的具体表述——该研究阐明热力学本质上是信息理论。在此框架下，每个抽样过程都存在对应的力学类比。因此，参数空间中抽样器系综与热力学平衡态机械系统抽样之间的分野实属人为划分。基于这一认识，我们构建了类似于热力学系综间转变的贝叶斯更新连续模型。由此推导出Jazinsky等式的信息论诠释，通过似然函数将功的消耗与数据影响联系起来。我们提出一种从贝叶斯定律出发，将热力学（能量、功、热）与统计力学（配分函数）的术语体系及形式化方法移植到统计推断的途径。本文讨论了不同类型的推断过程，并证明相对熵可通过适当构建的配分函数作为抽样过程的解析表述而推导得出。最后，我们提出用有效维度作为系统复杂度的度量标准，并以宇宙学数值算例佐证上述结论。