Generative models frequently suffer miscalibration, wherein statistics of the sampling distribution such as class probabilities deviate from desired values. We frame calibration as a constrained optimization problem and seek the closest model in Kullback-Leibler divergence satisfying calibration constraints. To address the intractability of imposing these constraints exactly, we introduce two surrogate objectives for fine-tuning: (1) the relax loss, which replaces the constraint with a miscalibration penalty, and (2) the reward loss, which converts calibration into a reward fine-tuning problem. We demonstrate that these approaches substantially reduce calibration error across hundreds of simultaneous constraints and models with up to one billion parameters, spanning applications in protein design, image generation, and language modeling.

翻译：生成模型常存在校准不足的问题，即采样分布的统计量（如类别概率）偏离期望值。本文将校准问题构建为约束优化问题，旨在寻找满足校准约束条件下Kullback-Leibler散度最小的最接近模型。为解决精确施加这些约束的难处理性，我们提出了两种微调替代目标：（1）松弛损失——将约束替换为校准误差惩罚项；（2）奖励损失——将校准问题转化为奖励微调任务。实验表明，在蛋白质设计、图像生成和语言建模等应用中，这些方法能显著降低涵盖数百个同步约束、参数量高达十亿级模型的校准误差。