Active inference is a theory of perception, learning and decision making, which can be applied to neuroscience, robotics, and machine learning. Recently, reasearch has been taking place to scale up this framework using Monte-Carlo tree search and deep learning. The goal of this activity is to solve more complicated tasks using deep active inference. First, we review the existing literature, then, we progresively build a deep active inference agent. For two agents, we have experimented with five definitions of the expected free energy and three different action selection strategies. According to our experiments, the models able to solve the dSprites environment are the ones that maximise rewards. Finally, we compare the similarity of the representation learned by the layers of various agents using centered kernel alignment. Importantly, the agent maximising reward and the agent minimising expected free energy learn very similar representations except for the last layer of the critic network (reflecting the difference in learning objective), and the variance layers of the transition and encoder networks. We found that the reward maximising agent is a lot more certain than the agent minimising expected free energy. This is because the agent minimising expected free energy always picks the action down, and does not gather enough data for the other actions. In contrast, the agent maximising reward, keeps on selecting the actions left and right, enabling it to successfully solve the task. The only difference between those two agents is the epistemic value, which aims to make the outputs of the transition and encoder networks as close as possible. Thus, the agent minimising expected free energy picks a single action (down), and becomes an expert at predicting the future when selecting this action. This makes the KL divergence between the output of the transition and encoder networks small.

翻译：主动推理是一种关于感知、学习和决策的理论，可应用于神经科学、机器人学和机器学习。近期，研究人员正尝试利用蒙特卡洛树搜索和深度学习来扩展这一框架，其目标是借助深度主动推理解决更复杂的任务。本文首先回顾现有文献，然后逐步构建一个深度主动推理智能体。对于两个智能体，我们实验了五种预期自由能定义和三种不同的动作选择策略。实验表明，能够解决dSprites环境的模型是那些最大化奖励的模型。最后，我们使用中心核对齐比较了各智能体网络层所学表征的相似性。重要的是，除了评论家网络的最后一层（反映学习目标的差异）以及转移网络和编码网络的方差层外，最大化奖励的智能体与最小化预期自由能的智能体所学表征高度相似。我们发现，最大化奖励的智能体比最小化预期自由能的智能体确定性高得多。这是因为最小化预期自由能的智能体总是选择向下动作，未获取其他动作的足够数据。相比之下，最大化奖励的智能体持续选择左右动作，使其能够成功完成任务。这两个智能体间的唯一区别在于认知价值——后者旨在使转移网络和编码网络的输出尽可能接近。因此，最小化预期自由能的智能体只选择单一动作（向下），并在选择该动作时成为预测未来的专家，这促使转移网络与编码网络输出间的KL散度变小。