Generating long, coherent text remains a challenge for large language models (LLMs), as they lack hierarchical planning and structured organization in discourse generation. We introduce Structural Alignment, a novel method that aligns LLMs with human-like discourse structures to enhance long-form text generation. By integrating linguistically grounded discourse frameworks into reinforcement learning, our approach guides models to produce coherent and well-organized outputs. We employ a dense reward scheme within a Proximal Policy Optimization framework, assigning fine-grained, token-level rewards based on the discourse distinctiveness relative to human writing. Two complementary reward models are evaluated: the first improves readability by scoring surface-level textual features to provide explicit structuring, while the second reinforces deeper coherence and rhetorical sophistication by analyzing global discourse patterns through hierarchical discourse motifs, outperforming both standard and RLHF-enhanced models in tasks such as essay generation and long-document summarization. All training data and code will be publicly shared at https://github.com/minnesotanlp/struct_align.

翻译：生成长篇连贯文本仍然是大型语言模型（LLMs）面临的一项挑战，因为它们在语篇生成中缺乏层次化规划和结构化组织。我们提出了结构对齐（Structural Alignment）这一新方法，通过将LLMs与类人语篇结构对齐来增强长文本生成能力。该方法将基于语言学的语篇框架融入强化学习中，引导模型生成连贯且组织良好的输出。我们在近端策略优化（Proximal Policy Optimization）框架内采用密集奖励机制，根据相对于人类写作的语篇区分度分配细粒度的词元级奖励。我们评估了两种互补的奖励模型：第一种通过评估表层文本特征来提供显式结构以提升可读性；第二种则通过分析层次化语篇主题（discourse motifs）来强化深层连贯性与修辞复杂度。该方法在议论文生成和长文档摘要等任务中均优于标准模型及经过RLHF增强的模型。所有训练数据与代码将在 https://github.com/minnesotanlp/struct_align 公开共享。