As hardware failures such as node losses become increasingly common, MPI programmers may want to save vulnerable data in a resilient store. While third-party storage solutions such as Redis or the Hazelcast IMap exist, a tailored, MPI-based store may be easier to integrate and can be optimized for particular application needs. This paper considers the implementation of such a store, which is intended as a component in a resilient task-based runtime system written in MPI. The store holds redundant data copies as key-value pairs in the main memories of multiple processes. Since store access operations, such as reads and writes, are naturally one-sided, we implemented the store with passive target MPI RMA functions. Process aborts are detected with the user-level failure mitigation (ULFM) extension of Open MPI. After failures, the program recovers on the surviving processes and continues with the intact data copies. Our implementation proved difficult, since several proposed ULFM functionalities for RMA have not yet been implemented. Even assuming their existence, we think that the programming task could be simplified. This paper describes our experiences, lists functionalities that we missed, and explains a workaround that we adopted in our implementation.
翻译:随着节点丢失等硬件故障日益常见,MPI程序员可能需要将易丢失数据保存至弹性存储中。尽管存在Redis或Hazelcast IMap等第三方存储解决方案,但基于MPI的定制化存储更易于集成,并可针对特定应用需求进行优化。本文探讨了此类存储的实现方案——该存储旨在作为基于MPI的弹性任务运行时系统的组件。该存储以键值对形式将冗余数据副本保存在多个进程的主存中。鉴于读写等存储访问操作天然具有单边特性,我们采用被动目标MPI RMA函数实现了该存储。进程中止通过Open MPI的用户级故障缓解扩展检测。故障发生后,程序在幸存进程上恢复运行,并利用完整数据副本继续执行。由于RMA的多项ULFM提议功能尚未实现,我们的实施过程面临挑战。即便假定这些功能存在,我们认为编程任务仍可简化。本文详述了我们的实践经验,列出了缺失的功能,并解释了在实现中采用的变通方案。