Policy optimization methods like Group Relative Policy Optimization (GRPO) and its variants have achieved strong results on mathematical reasoning and code generation tasks. Despite extensive exploration of reward processing strategies and training dynamics, all existing group-based methods exclusively use KL divergence for policy regularization, leaving the choice of divergence function unexplored. We introduce Group-Based Mirror Policy Optimization (GBMPO), a framework that extends group-based policy optimization to flexible Bregman divergences, including hand-designed alternatives (L2 in probability space) and learned neural mirror maps. On GSM8K mathematical reasoning, hand-designed ProbL2-GRPO achieves 86.7% accuracy, improving +5.5 points over the Dr. GRPO baseline. On MBPP code generation, neural mirror maps reach 60.1-60.8% pass@1, with random initialization already capturing most of the benefit. While evolutionary strategies meta-learning provides marginal accuracy improvements, its primary value lies in variance reduction ($\pm$0.2 versus $\pm$0.6) and efficiency gains (15% shorter responses on MBPP), suggesting that random initialization of neural mirror maps is sufficient for most practical applications. These results establish divergence choice as a critical, previously unexplored design dimension in group-based policy optimization for LLM reasoning.

翻译：在数学推理与代码生成任务上，诸如分组相对策略优化（GRPO）及其变体等策略优化方法已取得显著成果。尽管对奖励处理策略与训练动态进行了广泛探索，现有所有基于分组的方法均仅采用KL散度进行策略正则化，而散度函数的选择尚未得到研究。本文提出基于分组的镜像策略优化（GBMPO）框架，将基于分组的策略优化扩展至灵活的Bregman散度，包括人工设计的替代方案（概率空间中的L2散度）以及习得的神经镜像映射。在GSM8K数学推理任务中，人工设计的ProbL2-GRPO达到86.7%的准确率，较Dr. GRPO基线提升+5.5个百分点。在MBPP代码生成任务中，神经镜像映射获得60.1-60.8%的pass@1指标，仅通过随机初始化即可捕获大部分性能增益。虽然进化策略元学习带来有限的准确率提升，但其主要价值在于降低方差（$\pm$0.2对比$\pm$0.6）与提升效率（MBPP任务中响应长度缩短15%），这表明神经镜像映射的随机初始化已能满足多数实际应用需求。这些结果确立了散度选择作为基于分组的LLM推理策略优化中一个关键且此前未被探索的设计维度。