This study analyzes changes in the attention mechanisms of large language models (LLMs) when used to understand natural conversations between humans (human-human). We analyze three use cases of LLMs: interactions over web content, code, and mathematical texts. By analyzing attention distance, dispersion, and interdependency across these domains, we highlight the unique challenges posed by conversational data. Notably, conversations require nuanced handling of long-term contextual relationships and exhibit higher complexity through their attention patterns. Our findings reveal that while language models exhibit domain-specific attention behaviors, there is a significant gap in their ability to specialize in human conversations. Through detailed attention entropy analysis and t-SNE visualizations, we demonstrate the need for models trained with a diverse array of high-quality conversational data to enhance understanding and generation of human-like dialogue. This research highlights the importance of domain specialization in language models and suggests pathways for future advancement in modeling human conversational nuances.

翻译：本研究分析了大型语言模型（LLMs）在理解人类间自然对话时注意力机制的变化。我们考察了LLMs的三种应用场景：网络内容交互、代码处理与数学文本分析。通过分析注意力距离、离散度及跨域互依性，揭示了对话数据带来的独特挑战。值得注意的是，对话需要精细处理长期上下文关系，其注意力模式展现出更高复杂度。研究结果表明，尽管语言模型具有领域特异性注意力行为，但在专精于人类对话方面仍存在显著差距。通过详细的注意力熵分析与t-SNE可视化，我们论证了使用多样化高质量对话数据训练的模型对于增强类人对话理解与生成能力的必要性。本研究凸显了语言模型领域专精化的重要性，并为未来建模人类对话细微差异提供了发展路径。