Large language models (LLMs) have revolutionized natural language processing. However, effectively incorporating complex and potentially noisy user interaction data remains a challenge. To address this, we propose User-LLM, a novel framework that leverages user embeddings to contextualize LLMs. These embeddings, distilled from diverse user interactions using self-supervised pretraining, capture latent user preferences and their evolution over time. We integrate these user embeddings with LLMs through cross-attention and soft-prompting, enabling LLMs to dynamically adapt to user context. Our comprehensive experiments on MovieLens, Amazon Review, and Google Local Review datasets demonstrate significant performance gains across various tasks. Notably, our approach outperforms text-prompt-based contextualization on long sequence tasks and tasks that require deep user understanding while being computationally efficient. We further incorporate Perceiver layers to streamline the integration between user encoders and LLMs, reducing computational demands.

翻译：大型语言模型（LLM）已彻底改变了自然语言处理领域。然而，如何有效整合复杂且可能含有噪声的用户交互数据仍是一大挑战。为此，我们提出用户-LLM这一创新框架，通过利用用户嵌入实现LLM的上下文化。这些嵌入通过自监督预训练从多样化的用户交互中提取，能够捕捉潜在的用户偏好及其随时间演变的趋势。我们通过交叉注意力机制与软提示技术将用户嵌入集成到LLM中，使模型能够动态适应用户上下文。在MovieLens、Amazon Review和Google Local Review数据集上的全面实验表明，该方法在各类任务中均实现了显著的性能提升。值得注意的是，在长序列任务及需要深度理解用户的任务中，我们的方法不仅计算效率更高，且性能显著优于基于文本提示的上下文化方法。我们还引入了感知器层来优化用户编码器与LLM之间的集成，有效降低了计算需求。