Positional encoding (PE) is a core architectural component of Transformers, yet its impact on the Transformer's generalization and robustness remains unclear. In this work, we provide the first generalization analysis for a single-layer Transformer under in-context regression that explicitly accounts for a completely trainable PE module. Our result shows that PE systematically enlarges the generalization gap. Extending to the adversarial setting, we derive the adversarial Rademacher generalization bound. We find that the gap between models with and without PE is magnified under attack, demonstrating that PE amplifies the vulnerability of models. Our bounds are empirically validated by a simulation study. Together, this work establishes a new framework for understanding the clean and adversarial generalization in ICL with PE.

翻译：位置编码是变换器核心架构组件之一，但其对变换器泛化性与鲁棒性的影响尚未明确。本研究首次针对单层变换器在上下文回归场景下进行泛化分析，显式考虑了完全可训练的位置编码模块。结果表明，位置编码系统性扩大了泛化差距。将研究拓展至对抗性场景，我们推导出对抗性Rademacher泛化界，发现带与不带位置编码的模型差距在攻击下被放大，证实位置编码加剧了模型脆弱性。通过仿真实验验证了所提界的有效性。综上，本工作建立了理解上下文学习中位置编码对干净与对抗泛化影响的新框架。