To generalize across tasks, an agent should acquire knowledge from past tasks that facilitate adaptation and exploration in future tasks. We focus on the problem of in-context adaptation and exploration, where an agent only relies on context, i.e., history of states, actions and/or rewards, rather than gradient-based updates. Posterior sampling (extension of Thompson sampling) is a promising approach, but it requires Bayesian inference and dynamic programming, which often involve unknowns (e.g., a prior) and costly computations. To address these difficulties, we use a transformer to learn an inference process from training tasks and consider a hypothesis space of partial models, represented as small Markov decision processes that are cheap for dynamic programming. In our version of the Symbolic Alchemy benchmark, our method's adaptation speed and exploration-exploitation balance approach those of an exact posterior sampling oracle. We also show that even though partial models exclude relevant information from the environment, they can nevertheless lead to good policies.

翻译：为了在不同任务间进行泛化，智能体应从过往任务中获取有助于未来任务适应与探索的知识。我们重点关注上下文适应与探索问题，即智能体仅依赖上下文（状态、动作和/或奖励的历史记录），而非基于梯度的更新进行学习。后验采样（汤普森采样的扩展）是一种有前景的方法，但它需要贝叶斯推理和动态规划，这往往涉及未知因素（例如先验分布）和昂贵的计算成本。为应对这些挑战，我们使用Transformer从训练任务中学习推理过程，并将部分模型的假设空间视为小型马尔可夫决策过程（便于动态规划计算）。在我们的Symbolic Alchemy基准测试中，该方法的适应速度和探索-利用平衡逼近了精确后验采样基准。我们还证明，尽管部分模型排除了环境中的相关信息，但它们仍能产生良好的策略。