Large language models (LLMs) demonstrate strong performance, but they often lack transparency. We introduce GeoLAN, a training framework that treats token representations as geometric trajectories and applies stickiness conditions inspired by recent developments related to the Kakeya Conjecture. We have developed two differentiable regularizers, Katz-Tao Convex Wolff (KT-CW) and Katz-Tao Attention (KT-Attn), that promote isotropy and encourage diverse attention. Our experiments with Gemma-3 (1B, 4B, 12B) and Llama-3-8B show that GeoLAN frequently maintains task accuracy while improving geometric metrics and reducing certain fairness biases. These benefits are most significant in mid-sized models. Our findings reveal scale-dependent trade-offs between geometric precision and performance, suggesting that geometry-aware training is a promising approach to enhance mechanistic interpretability.

翻译：摘要：大语言模型（LLMs）展现了强大的性能，但通常缺乏透明度。我们提出GeoLAN训练框架，该方法将词元表征视为几何轨迹，并应用受Kakeya猜想最新进展启发的黏性条件。我们开发了两个可微分正则化器——Katz-Tao凸Wolff（KT-CW）和Katz-Tao注意力（KT-Attn），它们能促进各向同性并鼓励注意力多样化。在Gemma-3（1B、4B、12B）和Llama-3-8B上的实验表明，GeoLAN在保持任务精度的同时，常能改进几何度量并减少特定公平性偏差。这些优势在中等规模模型中最为显著。我们的发现揭示了几何精度与性能之间存在尺度依赖的权衡，表明几何感知训练是增强机制可解释性的一种有前景的方法。