Proteins inherently possess a consistent sequence-structure duality. The abundance of protein sequence data, which can be readily represented as discrete tokens, has driven fruitful developments in protein language models (pLMs). A key remaining challenge, however, is how to effectively integrate continuous structural knowledge into pLMs. Current methods often discretize protein structures to accommodate the language modeling framework, which inevitably results in the loss of fine-grained information and limits the performance potential of multimodal pLMs. In this paper, we argue that such concerns can be circumvented: a sequence-based pLM can be extended to incorporate the structure modality through continuous tokens, i.e., high-fidelity protein structure latents that avoid vector quantization. Specifically, we propose a hybrid diffusion protein language model, HD-Prot, which embeds a continuous-valued diffusion head atop a discrete pLM, enabling seamless operation with both discrete and continuous tokens for joint sequence-structure modeling. It captures inter-token dependencies across modalities through a unified absorbing diffusion process, and estimates per-token distributions via categorical prediction for sequences and continuous diffusion for structures. Extensive results demonstrate that HD-Prot achieves competitive performance in unconditional sequence-structure co-generation, motif-scaffolding, protein structure prediction, and inverse folding tasks. Furthermore, our method can perform on par with state-of-the-art multimodal pLMs, despite being developed under limited computational resources (i.e., less than one-tenth the budget for modality extension fine-tuning). It highlights the viability of simultaneously estimating categorical and continuous distributions within a unified language model architecture, offering a promising alternative direction for multimodal pLMs.

翻译：蛋白质天然具有一致的序列-结构双重性。丰富的蛋白质序列数据可便捷地表示为离散标记，这推动了蛋白质语言模型（pLMs）的蓬勃发展。然而，如何有效整合连续结构知识仍是一个关键挑战。现有方法通常将蛋白质结构离散化以适应语言建模框架，这不可避免地导致细粒度信息的丢失，并限制了多模态pLMs的性能潜力。本文论证了此类问题可被规避：基于序列的pLM可通过连续标记（即避免向量量化的高保真蛋白质结构隐变量）扩展以融入结构模态。具体而言，我们提出混合扩散蛋白质语言模型HD-Prot，其在离散pLM之上嵌入连续值扩散头，实现离散与连续标记的无缝协同，用于联合序列-结构建模。该模型通过统一吸收扩散过程捕捉跨模态的标记间依赖关系，并分别通过序列的分类预测和结构的连续扩散估计各标记分布。大量结果表明，HD-Prot在无条件序列-结构协同生成、基序支架设计、蛋白质结构预测及逆折叠任务中均取得了具有竞争力的性能。此外，尽管在有限计算资源下开发（即模态扩展微调预算不足十分之一），该方法仍能与最先进的多模态pLMs性能相当。这证实了在统一语言模型架构中同时估计分类分布与连续分布的可行性，为多模态pLMs提供了富有前景的替代方向。