Generative models frequently suffer miscalibration, wherein class probabilities and other statistics of the sampling distribution deviate from desired values. We frame calibration as a constrained optimization problem and seek the closest model in Kullback-Leibler divergence satisfying calibration constraints. To address the intractability of imposing these constraints exactly, we introduce two surrogate objectives for fine-tuning: (1) the relax loss, which replaces the constraint with a miscalibration penalty, and (2) the reward loss, which converts calibration into a reward fine-tuning problem. We demonstrate that these approaches substantially reduce calibration error across hundreds of simultaneous constraints and models with up to one billion parameters, spanning applications in protein design, image generation, and language modeling.

翻译：生成模型常存在校准不足的问题，即采样分布的类别概率及其他统计量与期望值存在偏差。本文将校准问题构建为约束优化问题，旨在寻找满足校准约束条件下Kullback-Leibler散度最小的最接近模型。为解决精确施加这些约束的难处理性，我们引入了两种微调替代目标：(1) 松弛损失——将约束替换为校准误差惩罚项；(2) 奖励损失——将校准转化为奖励微调问题。实验表明，这些方法能在数百个同步约束下显著降低校准误差，适用于参数规模达十亿级的模型，涵盖蛋白质设计、图像生成和语言建模等多个应用领域。