Large pre-trained models (LMs) and Large Language Models (LLMs) are typically effective at capturing language semantics and contextual relationships. However, these models encounter challenges in maintaining optimal performance on tasks involving numbers. Blindly treating numerical or structured data as terms is inadequate -- their semantics must be well understood and encoded by the models. In this paper, we propose CONE, a hybrid transformer encoder pre-trained model that encodes numbers, ranges, and gaussians into an embedding vector space preserving distance. We introduce a novel composite embedding construction algorithm that integrates numerical values, ranges or gaussians together with their associated units and attribute names to precisely capture their intricate semantics. We conduct extensive experimental evaluation on large-scale datasets across diverse domains (web, medical, finance, and government) that justifies CONE's strong numerical reasoning capabilities, achieving an F1 score of 87.28% on DROP, a remarkable improvement of up to 9.37% in F1 over state-of-the-art (SOTA) baselines, and outperforming major SOTA models with a significant Recall@10 gain of up to 25%.

翻译：大型预训练模型（LMs）与大型语言模型（LLMs）通常在捕捉语言语义和上下文关系方面表现优异。然而，这些模型在处理涉及数值的任务时难以保持最佳性能。将数值或结构化数据盲目视为普通词汇是不充分的——模型必须充分理解并编码其内在语义。本文提出CONE，一种混合Transformer编码器预训练模型，能够将数值、区间及高斯分布编码为保持距离关系的嵌入向量空间。我们引入了一种新颖的复合嵌入构建算法，该算法将数值、区间或高斯分布与其关联的单位和属性名称相结合，以精确捕捉其复杂语义。我们在跨多领域（网络、医疗、金融与政务）的大规模数据集上进行了广泛实验评估，结果验证了CONE强大的数值推理能力：在DROP数据集上取得87.28%的F1分数，较当前最优基准模型提升达9.37%的F1值，并在Recall@10指标上以最高25%的显著优势超越主流前沿模型。