Partially observable Markov decision processes (POMDPs) provide a flexible representation for real-world decision and control problems. However, POMDPs are notoriously difficult to solve, especially when the state and observation spaces are continuous or hybrid, which is often the case for physical systems. While recent online sampling-based POMDP algorithms that plan with observation likelihood weighting have shown practical effectiveness, a general theory characterizing the approximation error of the particle filtering techniques that these algorithms use has not previously been proposed. Our main contribution is bounding the error between any POMDP and its corresponding finite sample particle belief MDP (PB-MDP) approximation. This fundamental bridge between PB-MDPs and POMDPs allows us to adapt any sampling-based MDP algorithm to a POMDP by solving the corresponding particle belief MDP, thereby extending the convergence guarantees of the MDP algorithm to the POMDP. Practically, this is implemented by using the particle filter belief transition model as the generative model for the MDP solver. While this requires access to the observation density model from the POMDP, it only increases the transition sampling complexity of the MDP solver by a factor of $\mathcal{O}(C)$, where $C$ is the number of particles. Thus, when combined with sparse sampling MDP algorithms, this approach can yield algorithms for POMDPs that have no direct theoretical dependence on the size of the state and observation spaces. In addition to our theoretical contribution, we perform five numerical experiments on benchmark POMDPs to demonstrate that a simple MDP algorithm adapted using PB-MDP approximation, Sparse-PFT, achieves performance competitive with other leading continuous observation POMDP solvers.

翻译：部分可观测马尔可夫决策过程（POMDPs）为现实世界中的决策与控制问题提供了灵活的表示方法。然而，POMDPs 的求解尤为困难，尤其是当状态和观测空间为连续或混合类型时（这在物理系统中十分常见）。尽管近期基于在线采样的 POMDP 算法（通过观测似然加权进行规划）已展现出实际有效性，但此前尚未有通用理论刻画这些算法所采用的粒子滤波技术的近似误差。我们的主要贡献在于界定了任意 POMDP 与其对应的有限样本粒子信念 MDP（PB-MDP）近似之间的误差。这一连接 PB-MDP 与 POMDP 的基础桥梁使得我们能够通过求解对应的粒子信念 MDP，将任意基于采样的 MDP 算法适配至 POMDP，从而将 MDP 算法的收敛性保证扩展至 POMDP。在实际应用中，这通过将粒子滤波信念转移模型作为 MDP 求解器的生成模型来实现。虽然该方法需要访问 POMDP 的观测密度模型，但仅将 MDP 求解器的转移采样复杂度增加因子 $\mathcal{O}(C)$（其中 $C$ 为粒子数）。因此，当与稀疏采样 MDP 算法结合时，该方法可生成对状态和观测空间大小无直接理论依赖的 POMDP 算法。除理论贡献外，我们在基准 POMDP 上进行了五项数值实验，结果表明：采用 PB-MDP 近似的简单 MDP 算法 Sparse-PFT 在性能上可与其他领先的连续观测 POMDP 求解器相媲美。