While multilingual large language models have gained widespread adoption, their performance on non-English languages remains substantially inferior to English. This disparity is particularly evident in in-context learning scenarios, where providing demonstrations in English but testing on non-English inputs leads to significant performance degradation. In this paper, we hypothesize that LLMs develop a universal semantic space for understanding languages, where different languages are encoded as distinct directions within this space. Based on this hypothesis, we propose language vectors -- a training-free language steering approach that leverages activation differences between source and target languages to guide model behavior. We steer the model generations by adding the vector to the intermediate model activations during inference. This is done to make the model's internal representations shift towards the target language space without any parameter updates. We evaluate our method across three datasets and test on a total of 19 languages on three different models. Our results show consistent improvements on multilingual in-context learning over baselines across all tasks and languages tested. Beyond performance gains, hierarchical clustering of steering vectors reveals meaningful linguistic structure aligned with language families. These vectors also successfully transfer across tasks, demonstrating that these representations are task-agnostic.

翻译：尽管多语言大语言模型已获得广泛应用，但其在非英语语言上的性能仍显著低于英语。这种差异在上下文学习场景中尤为明显：当提供英语示例却测试非英语输入时，会导致性能显著下降。本文假设大语言模型为理解语言构建了一个通用的语义空间，不同语言在该空间中被编码为不同的方向。基于此假设，我们提出语言向量——一种无需训练的语言引导方法，该方法利用源语言与目标语言间的激活差异来引导模型行为。我们通过在推理过程中向中间层激活添加该向量来引导模型生成，旨在使模型的内部表示向目标语言空间偏移，而无需更新任何参数。我们在三个数据集上评估了该方法，并在三种不同模型上测试了总计19种语言。实验结果表明，在所有测试任务和语言中，我们的方法在多语言上下文学习上均较基线模型取得了一致性改进。除性能提升外，对引导向量的层次聚类分析揭示了与语系对齐的有意义的语言结构。这些向量还能成功跨任务迁移，表明其表征具有任务无关性。