Transformers are arguably the preferred architecture for language generation. In this paper, inspired by continued fractions, we introduce a new function class for generative modeling. The architecture family implementing this function class is named CoFrGeNets - Continued Fraction Generative Networks. We design novel architectural components based on this function class that can replace Multi-head Attention and Feed-Forward Networks in Transformer blocks while requiring much fewer parameters. We derive custom gradient formulations to optimize the proposed components more accurately and efficiently than using standard PyTorch-based gradients. Our components are a plug-in replacement requiring little change in training or inference procedures that have already been put in place for Transformer-based models thus making our approach easy to incorporate in large industrial workflows. We experiment on two very different transformer architectures GPT2-xl (1.5B) and Llama3 (3.2B), where the former we pre-train on OpenWebText and GneissWeb, while the latter we pre-train on the docling data mix which consists of nine different datasets. Results show that the performance on downstream classification, Q\& A, reasoning and text understanding tasks of our models is competitive and sometimes even superior to the original models with $\frac{2}{3}$ to $\frac{1}{2}$ the parameters and shorter pre-training time. We believe that future implementations customized to hardware will further bring out the true potential of our architectures.

翻译：Transformer 无疑是语言生成的首选架构。本文受连分式启发，为生成式建模引入了一类新的函数族。实现该函数族的架构家族被命名为 CoFrGeNet（连分式生成网络）。我们基于此函数族设计了新颖的架构组件，这些组件可以替代 Transformer 块中的多头注意力机制和前馈网络，同时所需参数量大幅减少。我们推导了定制化的梯度公式，以比基于标准 PyTorch 的梯度更准确、更高效地优化所提出的组件。我们的组件可作为即插即用的替换模块，对基于 Transformer 模型已建立的训练或推理流程几乎无需改动，从而便于融入大型工业工作流。我们在两种差异显著的 Transformer 架构 GPT2-xl（1.5B）和 Llama3（3.2B）上进行了实验：前者在 OpenWebText 和 GneissWeb 上进行预训练，后者则在包含九个不同数据集的 docling 数据混合集上进行预训练。结果表明，我们的模型在下游分类、问答、推理和文本理解任务上的性能具有竞争力，有时甚至优于原始模型，而参数量仅为原模型的 $\frac{2}{3}$ 到 $\frac{1}{2}$，且预训练时间更短。我们相信，未来针对硬件定制的实现将进一步释放我们架构的真正潜力。