Language models deployed in online communities must adapt to norms that vary across social, cultural, and domain-specific contexts. Prior alignment approaches rely on explicit preference supervision or predefined principles, which are effective for well-resourced settings but exclude most online communities -- particularly those without institutional backing, annotation infrastructure, or organized around sensitive topics -- where preference elicitation is costly, ethically fraught, or culturally misaligned. We observe that communities already express preferences implicitly through what content they accept, engage with, and allow to persist. We show that this acceptance behavior induces measurable geometric structure in representation space: accepted responses occupy coherent, high-density regions that reflect community-specific norms, while rejected content falls in sparser or misaligned areas. We operationalize this structure as an implicit preference signal for alignment and introduce density-guided response optimization (DGRO), a method that aligns language models to community norms without requiring explicit preference labels. Using labeled preference data, we demonstrate that local density recovers pairwise community judgments, indicating that geometric structure encodes meaningful preference signal. We then apply DGRO in annotation-scarce settings across diverse communities spanning platform, topic, and language. DGRO-aligned models consistently produce responses preferred by human annotators, domain experts, and model-based judges over supervised and prompt-based baselines. We position DGRO as a practical alignment alternative for communities where explicit preference supervision is unavailable or misaligned with situated practices, and discuss the implications and risks of learning from emergent acceptance behavior.

翻译：部署于在线社区的语言模型必须适应随社会、文化和领域特定情境变化的规范。先前的对齐方法依赖于显式的偏好监督或预定义原则，这些方法在资源充足的环境中有效，但排除了大多数在线社区——尤其是那些缺乏机构支持、标注基础设施或围绕敏感话题组织的社区——在这些社区中，偏好获取成本高昂、存在伦理风险或文化错位。我们观察到，社区已通过其接受、参与并允许持续存在的内容来隐含地表达偏好。我们证明这种接受行为在表示空间中诱导出可测量的几何结构：被接受的响应占据连贯、高密度的区域，这些区域反映了社区特定的规范，而被拒绝的内容则落在稀疏或错位的区域。我们将这种结构操作化为对齐的隐含偏好信号，并引入密度引导的响应优化（DGRO），这是一种无需显式偏好标签即可将语言模型与社区规范对齐的方法。通过使用标注的偏好数据，我们证明局部密度能够恢复社区成对判断，表明几何结构编码了有意义的偏好信号。随后，我们在涵盖平台、主题和语言的多样化社区中，于标注稀缺的环境中应用DGRO。与监督式和基于提示的基线相比，经DGRO对齐的模型持续生成更受人工标注者、领域专家和基于模型的评判者偏好的响应。我们将DGRO定位为一种实用的对齐替代方案，适用于显式偏好监督不可用或与情境实践错位的社区，并讨论了从涌现的接受行为中学习的意义与风险。