The widespread use of spreadsheet environments by billions of users presents a unique opportunity for formula-authoring assistance. Although large language models, such as Codex, can assist in general-purpose languages, they are expensive to train and challenging to deploy due to their large model sizes (up to billions of parameters). Moreover, they require hundreds of gigabytes of training data. We present FLAME, a T5-based model trained on Excel formulas that leverages domain insights to achieve competitive performance with a substantially smaller model (60M parameters) and two orders of magnitude less training data. We curate a training dataset using sketch deduplication, introduce an Excel-specific formula tokenizer for our model, and use domain-specific versions of masked span prediction and noisy auto-encoding as pretraining objectives. We evaluate FLAME on formula repair, formula auto-completion, and a novel task called syntax reconstruction. FLAME (60M) can outperform much larger models, such as Codex-Davinci (175B), Codex-Cushman (12B), and CodeT5 (220M), in 6 out of 10 settings.

翻译：数十亿用户对电子表格环境的广泛使用，为公式编写辅助功能提供了独特机遇。尽管Codex等大型语言模型可支持通用编程语言开发，但其模型规模高达数十亿参数，导致训练成本高昂且部署困难，且需要数百吉字节的训练数据。我们提出FLAME——一种基于T5架构、专为Excel公式训练的模型。该模型通过利用领域知识，以显著更小的规模（6000万参数）和低两个数量级的训练数据实现竞争性能。我们采用草图去重技术构建训练数据集，引入面向Excel的公式分词器，并使用掩码跨度预测与噪声自编码的领域适配版本作为预训练目标。我们在公式修复、公式自动补全及新型任务"语法重构"上评估FLAME。在10项测试中，FLAME（6000万参数）在6项设置下可超越Codex-Davinci（1750亿）、Codex-Cushman（120亿）和CodeT5（2.2亿）等更大规模模型。