We introduce iterative tilting, a gradient-free method for fine-tuning diffusion models toward reward-tilted distributions. The method decomposes a large reward tilt $\exp(λr)$ into $N$ sequential smaller tilts, each admitting a tractable score update via first-order Taylor expansion. This requires only forward evaluations of the reward function and avoids backpropagating through sampling chains. We validate on a two-dimensional Gaussian mixture with linear reward, where the exact tilted distribution is available in closed form.

翻译：我们提出了迭代倾斜方法，这是一种无需梯度的扩散模型微调技术，旨在使模型向奖励倾斜分布对齐。该方法将较大的奖励倾斜项 $\\exp(\\lambda r)$ 分解为 $N$ 个连续的小倾斜步骤，每一步通过一阶泰勒展开获得可处理的分数更新。该方法仅需对奖励函数进行前向计算，避免了在采样链中进行反向传播。我们在具有线性奖励的二维高斯混合模型上进行了验证，该场景下精确的倾斜分布具有闭式解。