Reaching disability limits an individual's ability in performing daily tasks. Surface Functional Electrical Stimulation (FES) offers a non-invasive solution to restore the lost abilities. However, inducing desired movements using FES is still an open engineering problem. This problem is accentuated by the complexities of human arms' neuromechanics and the variations across individuals. Reinforcement Learning (RL) emerges as a promising approach to govern customised control rules for different subjects and settings. Yet, one remaining challenge of using RL to control FES is unobservable muscle fatigue that progressively changes as an unknown function of the stimulation, breaking the Markovian assumption of RL. In this work, we present a method to address the unobservable muscle fatigue issue, allowing our RL controller to achieve higher control performances. Our method is based on a Gaussian State-Space Model (GSSM) that utilizes recurrent neural networks to learn Markovian state-spaces from partial observations. The GSSM is used as a filter that converts the observations into the state-space representation for RL to preserve the Markovian assumption. Here, we start with presenting the modification of the original GSSM to address an overconfident issue. We then present the interaction between RL and the modified GSSM, followed by the setup for FES control learning. We test our RL-GSSM system on a planar reaching setting in simulation using a detailed neuromechanical model and show that the GSSM can help RL maintain its control performance against the fatigue.

翻译：伸手障碍限制了个体完成日常任务的能力。表面功能性电刺激（FES）提供了一种非侵入性解决方案来恢复丧失的能力。然而，利用FES诱导期望运动仍是一个未解决的工程问题。人类手臂神经力学的复杂性及个体差异进一步加剧了这一难题。强化学习（RL）作为一种有前景的方法，可针对不同受试者和场景制定定制化控制规则。但使用RL控制FES的一个剩余挑战在于不可观测的肌肉疲劳——该疲劳随刺激呈未知函数渐进变化，破坏了RL的马尔可夫性假设。本文提出了一种应对不可观测肌肉疲劳问题的方法，使我们的RL控制器能够实现更高的控制性能。该方法基于高斯状态空间模型（GSSM），利用循环神经网络从部分观测中学习马尔可夫状态空间。GSSM作为滤波器，将观测转换为状态空间表征以供RL使用，从而维持马尔可夫性假设。我们首先阐述了为应对过度自信问题而对原始GSSM进行的修改，随后介绍RL与改进GSSM的交互机制，接着构建FES控制学习框架。我们在仿真环境中使用详细神经力学模型对平面伸手任务进行测试，结果表明GSSM能有效帮助RL在疲劳干扰下维持其控制性能。