Industry is rapidly moving towards fully autonomous and interconnected systems that can detect and adapt to changing conditions, including machine hardware faults. Traditional methods for adding hardware fault tolerance to machines involve duplicating components and algorithmically reconfiguring a machine's processes when a fault occurs. However, the growing interest in reinforcement learning-based robotic control offers a new perspective on achieving hardware fault tolerance. However, limited research has explored the potential of these approaches for hardware fault tolerance in machines. This paper investigates the potential of two state-of-the-art reinforcement learning algorithms, Proximal Policy Optimization (PPO) and Soft Actor-Critic (SAC), to enhance hardware fault tolerance into machines. We assess the performance of these algorithms in two OpenAI Gym simulated environments, Ant-v2 and FetchReach-v1. Robot models in these environments are subjected to six simulated hardware faults. Additionally, we conduct an ablation study to determine the optimal method for transferring an agent's knowledge, acquired through learning in a normal (pre-fault) environment, to a (post-)fault environment in a continual learning setting. Our results demonstrate that reinforcement learning-based approaches can enhance hardware fault tolerance in simulated machines, with adaptation occurring within minutes. Specifically, PPO exhibits the fastest adaptation when retaining the knowledge within its models, while SAC performs best when discarding all acquired knowledge. Overall, this study highlights the potential of reinforcement learning-based approaches, such as PPO and SAC, for hardware fault tolerance in machines. These findings pave the way for the development of robust and adaptive machines capable of effectively operating in real-world scenarios.

翻译：工业界正迅速迈向完全自主且互联的系统，这些系统能够检测并适应不断变化的条件，包括机器硬件故障。传统的机器硬件容错方法通常涉及复制组件，并在故障发生时通过算法重新配置机器流程。然而，基于强化学习的机器人控制日益受到关注，这为实现硬件容错提供了新的视角。然而，目前对这些方法在机器硬件容错方面潜力的探索仍有限。本文研究了两种先进的强化学习算法——近端策略优化（PPO）和柔性演员-评论家（SAC）——在增强机器硬件容错能力方面的潜力。我们在两个OpenAI Gym模拟环境（Ant-v2和FetchReach-v1）中评估了这些算法的性能。这些环境中的机器人模型被施加了六种模拟硬件故障。此外，我们进行了消融研究，以确定在持续学习设置中，将智能体在正常（故障前）环境中学习获得的知识迁移到（故障后）环境中的最优方法。我们的结果表明，基于强化学习的方法能够在几分钟内实现适应，从而增强模拟机器的硬件容错能力。具体而言，当保留模型内知识时，PPO表现出最快的适应速度；而当丢弃所有已获得知识时，SAC表现最佳。总体而言，本研究凸显了基于强化学习的方法（如PPO和SAC）在机器硬件容错方面的潜力。这些发现为开发能够在现实场景中有效运行的鲁棒且自适应的机器铺平了道路。