Modeling the joint distribution of data samples and their properties allows to construct a single model for both data generation and property prediction, with synergistic benefits reaching beyond purely generative or predictive models. However, training joint models presents daunting architectural and optimization challenges. Here, we propose Hyformer, a transformer-based joint model that successfully blends the generative and predictive functionalities, using an alternating attention mechanism and a joint pre-training scheme. We show that Hyformer is simultaneously optimized for molecule generation and property prediction, while exhibiting synergistic benefits in conditional sampling, out-of-distribution property prediction and representation learning. Finally, we demonstrate the benefits of joint learning in a drug design use case of discovering novel antimicrobial~peptides.

翻译：通过建模数据样本及其性质的联合分布，可构建兼具数据生成与性质预测功能的单一模型，其协同增益超越纯生成或预测模型。然而，联合模型的训练面临架构设计与优化策略上的重大挑战。本文提出Hyformer——一种基于Transformer的联合模型，通过交替注意力机制与联合预训练方案，成功融合生成与预测功能。实验表明，Hyformer可在分子生成与性质预测任务中同步优化，并在条件采样、分布外性质预测及表征学习场景中展现协同增益效应。最后，我们通过发现新型抗菌肽的药物设计案例，验证了联合学习在实际应用中的优势。