The COVID-19 pandemic highlighted the urgent need for effective vaccine policies, but traditional clinical trials often lack sufficient data to capture the diverse population characteristics necessary for comprehensive public health strategies. Ethical concerns around randomized trials during a pandemic further complicate policy development for public health. Reinforcement Learning (RL) offers a promising alternative for vaccine policy development. However, direct online RL exploration in real-world scenarios can result in suboptimal and potentially harmful decisions. This study proposes a novel framework combining tabular Q-learning with microsimulation, where a Recurrent Neural Network (RNN) serves as a digital twin environment simulator of the target population. This digital twin captures temporal associations between infection and patient characteristics to generate realistic individual disease trajectories, enabling safe and efficient policy learning without real-world interaction. Our tabular Q-learning model produces an interpretable policy table that balances the risks of severe infection against vaccination side effects. Applied to COVID-19 booster policies, the learned Q-learning-based policy outperforms current practices, offering a path toward more effective vaccination strategies. A project webpage introducing our work, including links to the software, a brief introductory video, and a step-by-step tutorial video, is available at https://public.websites.umich.edu/~jiankang/software/dtpl_website_umich/index.html.

翻译：COVID-19疫情凸显了制定有效疫苗政策的迫切需求，但传统临床试验往往缺乏足够的数据来捕捉制定全面公共卫生策略所需的多样化人群特征。疫情期间对随机试验的伦理关切进一步增加了公共卫生政策制定的复杂性。强化学习为疫苗政策制定提供了有前景的替代方案，然而在现实场景中直接进行在线RL探索可能导致次优甚至有害的决策。本研究提出了一种融合表格型Q学习与微观模拟的新框架，其中循环神经网络作为目标人群的数字孪生环境模拟器。该数字孪生模型通过捕捉感染与患者特征之间的时间关联性，生成逼真的个体疾病发展轨迹，从而在无需真实世界交互的情况下实现安全高效的政策学习。我们的表格型Q学习模型可生成具有可解释性的策略表，该策略表在重症感染风险与疫苗接种副作用之间取得平衡。应用于COVID-19加强针政策时，基于Q学习习得的策略优于当前实践，为制定更有效的疫苗接种策略提供了可行路径。介绍本工作的项目网页（含软件链接、简短介绍视频及分步教程视频）可见于：https://public.websites.umich.edu/~jiankang/software/dtpl_website_umich/index.html