Large language models (LLMs) trained with Reinforcement Learning from Human Feedback (RLHF) have demonstrated remarkable capabilities, but their underlying reward functions and decision-making processes remain opaque. This paper introduces a novel approach to interpreting LLMs by applying inverse reinforcement learning (IRL) to recover their implicit reward functions. We conduct experiments on toxicity-aligned LLMs of varying sizes, extracting reward models that achieve up to 85% accuracy in predicting human preferences. Our analysis reveals key insights into the non-identifiability of reward functions, the relationship between model size and interpretability, and potential pitfalls in the RLHF process. We demonstrate that IRL-derived reward models can be used to fine-tune new LLMs, resulting in comparable or improved performance on toxicity benchmarks. This work provides a new lens for understanding and improving LLM alignment, with implications for the responsible development and deployment of these powerful systems.

翻译：通过人类反馈强化学习（RLHF）训练的大语言模型（LLM）已展现出卓越能力，但其底层的奖励函数与决策过程仍不透明。本文提出一种新颖的LLM解释方法，通过应用逆向强化学习（IRL）来恢复其隐式奖励函数。我们在不同规模的毒性对齐LLM上进行实验，提取的奖励模型在预测人类偏好方面达到高达85%的准确率。分析揭示了奖励函数不可识别性的关键见解、模型规模与可解释性之间的关系，以及RLHF过程中潜在的缺陷。我们证明，基于IRL推导的奖励模型可用于微调新LLM，在毒性基准测试中获得相当或更优的性能。这项工作为理解和改进LLM对齐提供了新视角，对负责任地开发和部署这些强大系统具有重要意义。