Proteins inherently possess a consistent sequence-structure duality. The abundance of protein sequence data, which can be readily represented as discrete tokens, has driven fruitful developments in protein language models (pLMs). A key remaining challenge, however, is how to effectively integrate continuous structural knowledge into pLMs. Current methods often discretize protein structures to accommodate the language modeling framework, which inevitably results in the loss of fine-grained information and limits the performance potential of multimodal pLMs. In this paper, we argue that such concerns can be circumvented: a sequence-based pLM can be extended to incorporate the structure modality through continuous tokens, i.e., high-fidelity protein structure latents that avoid vector quantization. Specifically, we propose a hybrid diffusion protein language model, HD-Prot, which embeds a continuous-valued diffusion head atop a discrete pLM, enabling seamless operation with both discrete and continuous tokens for joint sequence-structure modeling. It captures inter-token dependencies across modalities through a unified absorbing diffusion process, and estimates per-token distributions via categorical prediction for sequences and continuous diffusion for structures. Extensive results demonstrate that HD-Prot achieves competitive performance in unconditional sequence-structure co-generation, motif-scaffolding, protein structure prediction, and inverse folding tasks. Furthermore, our method can perform on par with state-of-the-art multimodal pLMs, despite being developed under limited computational resources (i.e., less than one-tenth the budget for modality extension fine-tuning). It highlights the viability of simultaneously estimating categorical and continuous distributions within a unified language model architecture, offering a promising alternative direction for multimodal pLMs.

翻译：蛋白质天然具有一致的序列-结构二象性。大量可表示为离散令牌的蛋白质序列数据，推动了蛋白质语言模型（pLMs）的丰硕发展。然而，一个尚存的关键挑战是如何将连续结构知识有效整合到pLMs中。当前方法通常将蛋白质结构离散化以适应语言建模框架，这不可避免地导致细粒度信息的丢失，并限制了多模态pLMs的性能潜力。本文认为，此类问题可以规避：基于序列的pLM可通过连续令牌（即避免矢量量化的高保真蛋白质结构潜在表示）扩展以融入结构模态。具体而言，我们提出一种混合扩散蛋白质语言模型HD-Prot，其在离散pLM之上嵌入连续值扩散头，从而支持使用离散令牌和连续令牌无缝进行序列-结构联合建模。该模型通过统一的吸收扩散过程捕捉跨模态的令牌间依赖关系，并通过序列的分类预测和结构的连续扩散估计每个令牌的分布。大量结果表明，HD-Prot在无条件序列-结构协同生成、基序支架构建、蛋白质结构预测和反向折叠任务中均取得了具有竞争力的性能。此外，尽管在有限计算资源下开发（即模态扩展微调预算不足十分之一），我们的方法仍可达到与最先进多模态pLMs相当的性能。这凸显了在统一语言模型架构内同时估计分类分布和连续分布的可行性，为多模态pLMs提供了一种有前景的替代方向。