The authors of 'Unsupervised Reinforcement Learning in Multiple environments' propose a method, alpha-MEPOL, to tackle unsupervised RL across multiple environments. They pre-train a task-agnostic exploration policy using interactions from an entire environment class and then fine-tune this policy for various tasks using supervision. We expanded upon this work, with the goal of improving performance. We primarily propose and experiment with five new modifications to the original work: sampling trajectories using an entropy-based probability distribution, dynamic alpha, higher KL Divergence threshold, curiosity-driven exploration, and alpha-percentile sampling on curiosity. Dynamic alpha and higher KL-Divergence threshold both provided a significant improvement over the baseline from the earlier work. PDF-sampling failed to provide any improvement due to it being approximately equivalent to the baseline method when the sample space is small. In high-dimensional environments, the addition of curiosity-driven exploration enhances learning by encouraging the agent to seek diverse experiences and explore the unknown more. However, its benefits are limited in low-dimensional and simpler environments where exploration possibilities are constrained and there is little that is truly unknown to the agent. Overall, some of our experiments did boost performance over the baseline and there are a few directions that seem promising for further research.

翻译：《多环境无监督强化学习》的作者提出了一种名为alpha-MEPOL的方法，用于解决跨多个环境的无监督强化学习问题。该方法利用整个环境类中的交互预训练一个任务无关的探索策略，随后通过监督信号对该策略进行微调以适应各类任务。我们在该工作的基础上进行了扩展，旨在提升性能。主要提出了五项创新性改进并进行了实验验证：基于熵的概率分布轨迹采样、动态alpha值、更高的KL散度阈值、好奇心驱动探索，以及基于好奇心指标的alpha百分位采样。动态alpha值与更高KL散度阈值均显著优于基线结果。PDF采样因在样本空间较小时与基线方法近似等效而未见性能提升。在高维环境中，好奇心驱动探索通过激励智能体寻求多样化经验并探索未知领域来增强学习效果；但在低维简单环境中，由于探索可能性受限且智能体实际未知内容较少，其优势有限。总体而言，部分实验相较基线实现了性能提升，且存在若干值得进一步研究的方向。