Incorporating the successful paradigm of pretraining and finetuning from Computer Vision and Natural Language Processing into decision-making has become increasingly popular in recent years. In this paper, we study Imitation Learning from Observation with pretrained models and find existing approaches such as BCO and AIME face knowledge barriers, specifically the Embodiment Knowledge Barrier (EKB) and the Demonstration Knowledge Barrier (DKB), greatly limiting their performance. The EKB arises when pretrained models lack knowledge about unseen observations, leading to errors in action inference. The DKB results from policies trained on limited demonstrations, hindering adaptability to diverse scenarios. We thoroughly analyse the underlying mechanism of these barriers and propose AIME-v2 upon AIME as a solution. AIME-v2 uses online interactions with data-driven regulariser to alleviate the EKB and mitigates the DKB by introducing a surrogate reward function to enhance policy training. Experimental results on tasks from the DeepMind Control Suite and Meta-World benchmarks demonstrate the effectiveness of these modifications in improving both sample-efficiency and converged performance. The study contributes valuable insights into resolving knowledge barriers for enhanced decision-making in pretraining-based approaches. Code will be available at https://github.com/argmax-ai/aime-v2.

翻译：将计算机视觉和自然语言处理领域成功的预训练与微调范式引入决策制定过程，近年来日益受到关注。本文研究了基于预训练模型的观察模仿学习，发现BCO和AIME等现有方法存在知识壁垒，具体表现为具身知识壁垒（EKB）和示范知识壁垒（DKB），严重限制了其性能。EKB源于预训练模型缺乏对未见观察的知识，导致动作推理出现误差；DKB则因策略仅基于有限示范进行训练，阻碍了其对多样化场景的适应能力。我们深入分析了这些壁垒的底层机制，并提出基于AIME的改进版本AIME-v2。AIME-v2通过在线交互结合数据驱动正则化器缓解EKB，同时引入替代奖励函数增强策略训练以削弱DKB。在DeepMind控制套件和Meta-World基准测试任务上的实验结果表明，这些改进在提升样本效率和收敛性能方面均具有有效性。本研究为解决基于预训练方法中决策性能提升的知识壁垒问题提供了重要见解。相关代码将在https://github.com/argmax-ai/aime-v2 开放获取。