Reinforcement learning controllers have made impressive progress in humanoid locomotion and light load manipulation. However, achieving robust and precise motion with strong force interaction remains a significant challenge. Based on the above limitations, this paper proposes HAFO, a dual-agent reinforcement learning control framework that simultaneously optimizes both a robust locomotion strategy and a precise upper-body manipulation strategy through coupled training under external force interaction environments. Simultaneously, we explicitly model the external pulling disturbances through a spring-damper system and achieve fine-grained force control by manipulating the virtual spring. During this process, the reinforcement-learning policy spontaneously generates disturbance-rejection response by exploiting environmental feedback. Moreover, HAFO employs an asymmetric Actor-Critic framework in which the Critic-network access to privileged spring-damping forces guides the actor-network to learn a generalizable, robust policy for resisting external disturbances. The experimental results demonstrate that HAFO achieves stable control of humanoid robot under various strong force interactions, showing remarkable performance in load tasks and ensuring stable robot operation under rope tension disturbances. Project website: hafo-robot.github.io.

翻译：强化学习控制器在人形机器人运动与轻负载操作方面已取得显著进展，然而在强外力交互条件下实现鲁棒且精确的运动控制仍面临重大挑战。基于上述局限性，本文提出HAFO——一种双智能体强化学习控制框架，通过在外力交互环境下的耦合训练，同步优化鲁棒的运动策略与精确的上半身操作策略。同时，我们通过弹簧-阻尼系统显式建模外部拉力扰动，并借助虚拟弹簧的调控实现精细的力控制。在此过程中，强化学习策略通过利用环境反馈自发产生抗干扰响应。此外，HAFO采用非对称的Actor-Critic框架，其中Critic网络通过访问特权弹簧阻尼力信息，引导Actor网络学习可泛化的鲁棒抗干扰策略。实验结果表明，HAFO能在多种强外力交互场景下实现人形机器人的稳定控制，在负载任务中表现优异，并能在绳索拉力扰动下保持机器人稳定运行。项目网站：hafo-robot.github.io。