Meta-Reinforcement Learning (Meta-RL) agents can struggle to operate across tasks with varying environmental features that require different optimal skills (i.e., different modes of behaviours). Using context encoders based on contrastive learning to enhance the generalisability of Meta-RL agents is now widely studied but faces challenges such as the requirement for a large sample size, also referred to as the $\log$-$K$ curse. To improve RL generalisation to different tasks, we first introduce Skill-aware Mutual Information (SaMI), an optimisation objective that aids in distinguishing context embeddings according to skills, thereby equipping RL agents with the ability to identify and execute different skills across tasks. We then propose Skill-aware Noise Contrastive Estimation (SaNCE), a $K$-sample estimator used to optimise the SaMI objective. We provide a framework for equipping an RL agent with SaNCE in practice and conduct experimental validation on modified MuJoCo and Panda-gym benchmarks. We empirically find that RL agents that learn by maximising SaMI achieve substantially improved zero-shot generalisation to unseen tasks. Additionally, the context encoder equipped with SaNCE demonstrates greater robustness to reductions in the number of available samples, thus possessing the potential to overcome the $\log$-$K$ curse.

翻译：元强化学习（Meta-RL）智能体在处理具有不同环境特征且需要不同最优技能（即不同行为模式）的任务时可能面临困难。当前基于对比学习的上下文编码器被广泛研究用于提升元强化学习智能体的泛化能力，但仍面临诸如需要大量样本（即$\log$-$K$诅咒）等挑战。为提升强化学习在不同任务间的泛化性能，我们首先提出技能感知互信息（SaMI）优化目标，该目标能根据技能差异区分上下文嵌入表示，从而使强化学习智能体具备跨任务识别与执行不同技能的能力。随后我们提出技能感知噪声对比估计（SaNCE），这是一种用于优化SaMI目标的$K$样本估计器。我们构建了为强化学习智能体实际配备SaNCE的框架，并在改进的MuJoCo和Panda-gym基准测试中进行实验验证。实证研究表明，通过最大化SaMI进行学习的强化学习智能体在未见任务上实现了显著提升的零样本泛化能力。此外，配备SaNCE的上下文编码器对可用样本数量减少表现出更强的鲁棒性，因而具备克服$\log$-$K$诅咒的潜力。