Human feedback plays a critical role in learning and refining reward models for text-to-image generation, but the optimal form the feedback should take for learning an accurate reward function has not been conclusively established. This paper investigates the effectiveness of fine-grained feedback which captures nuanced distinctions in image quality and prompt-alignment, compared to traditional coarse-grained feedback (for example, thumbs up/down or ranking between a set of options). While fine-grained feedback holds promise, particularly for systems catering to diverse societal preferences, we show that demonstrating its superiority to coarse-grained feedback is not automatic. Through experiments on real and synthetic preference data, we surface the complexities of building effective models due to the interplay of model choice, feedback type, and the alignment between human judgment and computational interpretation. We identify key challenges in eliciting and utilizing fine-grained feedback, prompting a reassessment of its assumed benefits and practicality. Our findings -- e.g., that fine-grained feedback can lead to worse models for a fixed budget, in some settings; however, in controlled settings with known attributes, fine grained rewards can indeed be more helpful -- call for careful consideration of feedback attributes and potentially beckon novel modeling approaches to appropriately unlock the potential value of fine-grained feedback in-the-wild.

翻译：人类反馈在学习和优化文本到图像生成的奖励模型中起着关键作用，但反馈应采取何种最优形式以学习准确的奖励函数尚未得到明确结论。本文研究了捕捉图像质量与提示对齐细微差异的细粒度反馈相较于传统粗粒度反馈（例如点赞/点踩或选项间排序）的有效性。尽管细粒度反馈具有潜力，特别是在适应多样化社会偏好的系统中，但我们证明其相对于粗粒度反馈的优越性并非自动实现。通过在真实与合成偏好数据上的实验，我们揭示了由于模型选择、反馈类型以及人类判断与计算解释之间的对齐相互作用，构建有效模型所面临的复杂性。我们识别了在获取和利用细粒度反馈过程中的关键挑战，促使对其假定优势与实用性的重新评估。我们的发现——例如，在某些场景下，固定预算下细粒度反馈可能导致更差的模型；然而，在具有已知属性的受控环境中，细粒度奖励确实能提供更多帮助——呼吁对反馈属性进行审慎考量，并可能需要新颖的建模方法，以在真实场景中恰当释放细粒度反馈的潜在价值。