The use of reinforcement learning algorithms in financial trading is becoming increasingly prevalent. However, the autonomous nature of these algorithms can lead to unexpected outcomes that deviate from traditional game-theoretical predictions and may even destabilize markets. In this study, we examine a scenario in which two autonomous agents, modelled with Double Deep Q-Learning, learn to liquidate the same asset optimally in the presence of market impact, under the Almgren-Chriss (2000) framework. We show that the strategies learned by the agents deviate significantly from the Nash equilibrium of the corresponding market impact game. Notably, the learned strategies exhibit supra-competitive solution, {which might be compatible with a tacit collusive behaviour}, closely aligning with the Pareto-optimal solution. We further explore how different levels of market volatility influence the agents' performance and the equilibria they discover, including scenarios where volatility differs between the training and testing phases.

翻译：强化学习算法在金融交易中的应用日益普遍。然而，这些算法的自主性可能导致偏离传统博弈论预测的意外结果，甚至可能扰乱市场稳定。在本研究中，我们考察了在Almgren-Chriss（2000）框架下，两个采用双重深度Q学习建模的自主代理在存在市场影响的情况下学习最优清算同一资产的场景。我们证明，代理学习到的策略显著偏离了相应市场影响博弈的纳什均衡。值得注意的是，学习到的策略表现出超竞争解，{这可能与默示共谋行为相容}，且与帕累托最优解高度吻合。我们进一步探讨了不同市场波动水平如何影响代理的表现及其发现的均衡，包括训练与测试阶段波动率存在差异的情景。