Reinforcement learning (RL) controllers have made impressive progress in humanoid locomotion and light-weight object manipulation. However, achieving robust and precise motion control with intense force interaction remains a significant challenge. To address these limitations, this paper proposes HAFO, a dual-agent reinforcement learning framework that concurrently optimizes both a robust locomotion strategy and a precise upper-body manipulation strategy via coupled training in environments with external disturbances. The external pulling disturbances are explicitly modeled using a spring-damper system, allowing for fine-grained force control through manipulation of the virtual spring. In this process, the reinforcement learning policy autonomously generates a disturbance-rejection response by utilizing environmental feedback. Furthermore, HAFO employs an asymmetric Actor-Critic framework in which the Critic network's access to privileged external forces guides the actor network to acquire generalizable force adaptation for resisting external disturbances. The experimental results demonstrate that HAFO achieves whole-body control for humanoid robots across diverse force-interaction environments, delivering outstanding performance in load-bearing tasks and maintaining stable operation even under rope suspension state.

翻译：强化学习控制器在人形机器人运动与轻量级物体操控方面取得了显著进展。然而，在强外力交互场景下实现鲁棒且精确的运动控制仍面临重大挑战。为突破这些局限，本文提出HAFO——一种双智能体强化学习框架，通过在存在外部扰动的环境中进行耦合训练，同步优化鲁棒的运动策略与精确的上半身操控策略。外部拉力扰动通过弹簧-阻尼系统进行显式建模，使得通过调控虚拟弹簧实现精细的力控制成为可能。在此过程中，强化学习策略利用环境反馈自主生成抗扰响应。此外，HAFO采用非对称的Actor-Critic框架，其中Critic网络对特权外力信息的访问引导Actor网络学习可泛化的力适应能力以抵抗外部扰动。实验结果表明，HAFO能够在多样化的力交互环境中实现人形机器人的全身控制，在负重任务中表现优异，并在绳索悬吊状态下仍能保持稳定运行。