Active inference is a theory of perception, learning and decision making, which can be applied to neuroscience, robotics, and machine learning. Recently, reasearch has been taking place to scale up this framework using Monte-Carlo tree search and deep learning. The goal of this activity is to solve more complicated tasks using deep active inference. First, we review the existing literature, then, we progresively build a deep active inference agent. For two agents, we have experimented with five definitions of the expected free energy and three different action selection strategies. According to our experiments, the models able to solve the dSprites environment are the ones that maximise rewards. Finally, we compare the similarity of the representation learned by the layers of various agents using centered kernel alignment. Importantly, the agent maximising reward and the agent minimising expected free energy learn very similar representations except for the last layer of the critic network (reflecting the difference in learning objective), and the variance layers of the transition and encoder networks. We found that the reward maximising agent is a lot more certain than the agent minimising expected free energy. This is because the agent minimising expected free energy always picks the action down, and does not gather enough data for the other actions. In contrast, the agent maximising reward, keeps on selecting the actions left and right, enabling it to successfully solve the task. The only difference between those two agents is the epistemic value, which aims to make the outputs of the transition and encoder networks as close as possible. Thus, the agent minimising expected free energy picks a single action (down), and becomes an expert at predicting the future when selecting this action. This makes the KL divergence between the output of the transition and encoder networks small.

翻译：主动推理是一种关于感知、学习和决策的理论，可应用于神经科学、机器人学和机器学习。近期，研究人员开始利用蒙特卡洛树搜索和深度学习来扩展该框架，旨在通过深度主动推理解决更复杂的任务。首先，我们回顾了现有文献，然后逐步构建了一个深度主动推理智能体。针对两个智能体，我们实验了五种期望自由能的定义和三种不同的动作选择策略。实验结果表明，能够解决dSprites环境的模型是那些最大化奖励的模型。最后，我们使用中心核对齐比较了各智能体网络层所学表示的相似性。重要的是，最大化奖励的智能体与最小化期望自由能的智能体学到了非常相似的表示，除了评论家网络的最后一层（反映了学习目标的差异）以及转换网络和编码网络的方差层。我们发现，最大化奖励的智能体比最小化期望自由能的智能体具有更高的确定性。这是因为最小化期望自由能的智能体总是选择向下动作，未能收集其他动作的足够数据。相反，最大化奖励的智能体持续选择左右动作，使其能够成功完成任务。这两个智能体之间的唯一区别在于认知价值——其目标是使转换网络和编码网络的输出尽可能接近。因此，最小化期望自由能的智能体仅选择单一动作（向下），并成为预测该动作未来状态的专家，从而使得转换网络与编码网络输出之间的KL散度较小。