This work aims to generate realistic anatomical deformations from static patient scans. Specifically, we present a method to generate these deformations/augmentations via deep learning driven respiratory motion simulation that provides the ground truth for validating deformable image registration (DIR) algorithms and driving more accurate deep learning based DIR. We present a novel 3D Seq2Seq deep learning respiratory motion simulator (RMSim) that learns from 4D-CT images and predicts future breathing phases given a static CT image. The predicted respiratory patterns, represented by time-varying displacement vector fields (DVFs) at different breathing phases, are modulated through auxiliary inputs of 1D breathing traces so that a larger amplitude in the trace results in more significant predicted deformation. Stacked 3D-ConvLSTMs are used to capture the spatial-temporal respiration patterns. Training loss includes a smoothness loss in the DVF and mean-squared error between the predicted and ground truth phase images. A spatial transformer deforms the static CT with the predicted DVF to generate the predicted phase image. 10-phase 4D-CTs of 140 internal patients were used to train and test RMSim. The trained RMSim was then used to augment a public DIR challenge dataset for training VoxelMorph to show the effectiveness of RMSim-generated deformation augmentation. We validated our RMSim output with both private and public benchmark datasets (healthy and cancer patients). The proposed approach can be used for validating DIR algorithms as well as for patient-specific augmentations to improve deep learning DIR algorithms. The code, pretrained models, and augmented DIR validation datasets will be released at https://github.com/nadeemlab/SeqX2Y.

翻译：本研究旨在从静态患者扫描图像中生成真实的解剖形变。具体而言，我们提出了一种通过深度学习驱动的呼吸运动模拟来生成这些形变/增广的方法，该方法可为验证可变形图像配准（DIR）算法提供真值，并驱动更精确的基于深度学习的DIR。我们提出了一种新颖的3D Seq2Seq深度学习呼吸运动模拟器（RMSim），该模拟器从4D-CT图像中学习，并能在给定静态CT图像的情况下预测未来的呼吸相位。预测的呼吸模式（由不同呼吸相位时随时间变化的位移矢量场（DVF）表示）通过一维呼吸迹线的辅助输入进行调制，使得迹线中更大的幅度对应更显著的预测形变。采用堆叠的3D-ConvLSTM来捕捉时空呼吸模式。训练损失包括DVF中的平滑损失以及预测相位图像与真值相位图像之间的均方误差。空间变换器利用预测的DVF对静态CT进行形变，以生成预测的相位图像。使用140例内部患者的10相位4D-CT数据对RMSim进行训练和测试。随后，利用训练好的RMSim对一个公共DIR挑战数据集进行增广，以训练VoxelMorph，从而展示RMSim生成的形变增广的有效性。我们使用私有和公共基准数据集（包括健康患者和癌症患者）对RMSim输出进行了验证。所提出的方法可用于验证DIR算法，以及进行患者特异性增广以改进深度学习DIR算法。相关代码、预训练模型及增广后的DIR验证数据集将在https://github.com/nadeemlab/SeqX2Y 发布。