Recent strides in large language models (LLMs) have yielded remarkable performance, leveraging reinforcement learning from human feedback (RLHF) to significantly enhance generation and alignment capabilities. However, RLHF encounters numerous challenges, including the objective mismatch issue, leading to suboptimal performance in Natural Language Understanding (NLU) tasks. To address this limitation, we propose a novel Reinforcement Learning framework enhanced with Label-sensitive Reward (RLLR) to amplify the performance of LLMs in NLU tasks. By incorporating label-sensitive pairs into reinforcement learning, our method aims to adeptly capture nuanced label-sensitive semantic features during RL, thereby enhancing natural language understanding. Experiments conducted on five diverse foundation models across eight tasks showcase promising results. In comparison to Supervised Fine-tuning models (SFT), RLLR demonstrates an average performance improvement of 1.54%. Compared with RLHF models, the improvement averages at 0.69%. These results reveal the effectiveness of our method for LLMs in NLU tasks. Code and data available at: https://github.com/MagiaSN/ACL2024_RLLR.

翻译：近年来，大型语言模型（LLMs）取得了显著进展，通过利用人类反馈强化学习（RLHF）显著提升了生成与对齐能力。然而，RLHF面临诸多挑战，包括目标失配问题，导致其在自然语言理解（NLU）任务中表现欠佳。为应对这一局限，我们提出了一种新颖的、通过标签敏感奖励增强的强化学习框架（RLLR），以提升LLMs在NLU任务中的性能。通过将标签敏感对融入强化学习过程，我们的方法旨在RL过程中有效捕捉细微的标签敏感语义特征，从而增强自然语言理解能力。在八个任务上对五种不同的基础模型进行的实验展示了有希望的结果。与监督微调模型（SFT）相比，RLLR实现了平均1.54%的性能提升。与RLHF模型相比，平均提升为0.69%。这些结果揭示了我们的方法在LLMs处理NLU任务方面的有效性。代码与数据可见：https://github.com/MagiaSN/ACL2024_RLLR。