Adversarial Inverse Reinforcement Learning (AIRL) has shown promise in addressing the sparse reward problem in reinforcement learning (RL) by inferring dense reward functions from expert demonstrations. However, its performance in highly complex, imperfect-information settings remains largely unexplored. To explore this gap, we evaluate AIRL in the context of Heads-Up Limit Hold'em (HULHE) poker, a domain characterized by sparse, delayed rewards and significant uncertainty. In this setting, we find that AIRL struggles to infer a sufficiently informative reward function. To overcome this limitation, we contribute Hybrid-AIRL (H-AIRL), an extension that enhances reward inference and policy learning by incorporating a supervised loss derived from expert data and a stochastic regularization mechanism. We evaluate H-AIRL on a carefully selected set of Gymnasium benchmarks and the HULHE poker setting. Additionally, we analyze the learned reward function through visualization to gain deeper insights into the learning process. Our experimental results show that H-AIRL achieves higher sample efficiency and more stable learning compared to AIRL. This highlights the benefits of incorporating supervised signals into inverse RL and establishes H-AIRL as a promising framework for tackling challenging, real-world settings.

翻译：对抗性逆强化学习（AIRL）通过从专家演示中推断密集奖励函数，在解决强化学习（RL）中的稀疏奖励问题方面展现出潜力。然而，其在高度复杂、信息不完善环境中的性能仍很大程度上未被探索。为填补这一空白，我们在单挑有限注德州扑克（HULHE）这一具有稀疏、延迟奖励及显著不确定性的领域中评估了AIRL。在此环境下，我们发现AIRL难以推断出足够信息量的奖励函数。为克服这一局限，我们提出了混合对抗性逆强化学习（H-AIRL），该扩展通过引入源自专家数据的监督损失和随机正则化机制，增强了奖励推断与策略学习。我们在精心挑选的Gymnasium基准测试集及HULHE扑克环境中评估了H-AIRL。此外，我们通过可视化分析学习到的奖励函数，以更深入理解学习过程。实验结果表明，与AIRL相比，H-AIRL实现了更高的样本效率和更稳定的学习性能。这凸显了将监督信号融入逆强化学习的优势，并确立了H-AIRL作为应对具有挑战性的现实世界场景的有前景框架。