Large language models (LLMs) have recently garnered significant interest. With in-context learning, LLMs achieve impressive results in various natural language tasks. However, the application of LLMs to sentence embeddings remains an area of ongoing research. In this work, we propose an in-context learning-based method aimed at improving sentence embeddings performance. Our approach involves adapting the previous prompt-based representation method for autoregressive models, constructing a demonstration set that enables LLMs to perform in-context learning, and scaling up the LLMs to different model sizes. Through extensive experiments, in-context learning enables LLMs to generate high-quality sentence embeddings without any fine-tuning. It helps LLMs achieve performance comparable to current contrastive learning methods. By scaling model size, we find scaling to more than tens of billion parameters harms the performance on semantic textual similarity (STS) tasks. However, the largest model outperforms other counterparts and achieves the new state-of-the-art result on transfer tasks. We also fine-tune LLMs with current contrastive learning approach, and the 2.7B OPT model, incorporating our prompt-based method, surpasses the performance of 4.8B ST5, achieving the new state-of-the-art results on STS tasks. Our code is available at https://github.com/kongds/scaling_sentemb.

翻译：近年来，大规模语言模型（LLM）引发了广泛关注。凭借上下文学习能力，LLM在各类自然语言任务中取得了显著成果。然而，将LLM应用于句子嵌入仍是研究前沿课题。本文提出一种基于上下文学习的优化方法，旨在提升句子嵌入性能。该方法包括：针对自回归模型改进基于提示的表示方法、构建支持LLM上下文学习的示例集，以及实现不同参数规模LLM的扩展。大量实验表明，无需微调即可通过上下文学习使LLM生成高质量句子嵌入，其性能可媲美当前对比学习方法。通过模型扩展我们发现，参数规模超过数百亿的模型在语义文本相似度（STS）任务上性能下降，但最大规模模型在迁移任务中超越其他模型，创下新纪录。此外，我们采用当前对比学习方法对LLM进行微调，其中2.7B参数规模的OPT模型结合我们提出的提示方法，在STS任务上超越4.8B参数的ST5模型，达到当前最佳结果。相关代码已开源：https://github.com/kongds/scaling_sentemb