Fine-tuning pre-trained language models (LMs) is essential for enhancing their capabilities. Existing techniques commonly fine-tune on input-output pairs (e.g., instruction tuning) or with numerical rewards that gauge the output quality (e.g., RLHF). We explore LMs' potential to learn from textual interactions (LETI) that not only check their correctness with binary labels but also pinpoint and explain errors in their outputs through textual feedback. Our focus is the code generation task, where the model produces code based on natural language instructions. This setting invites a natural and scalable way to acquire textual feedback: the error messages and stack traces from code execution using a Python interpreter. LETI iteratively fine-tunes the model, using the LM objective, on a concatenation of natural language instructions, LM-generated programs, and textual feedback. Prepended to this fine-tuning text, a binary reward token is used to differentiate correct and buggy solutions. LETI requires no ground-truth outputs for training and even outperforms a fine-tuned baseline that does. LETI not only improves the performance of LMs on a code generation dataset MBPP, but also generalizes to other datasets. Trained on MBPP, it achieves comparable or better performance than the base LMs on unseen problems in HumanEval. Furthermore, compared to binary feedback, we observe that textual feedback leads to improved generation quality and sample efficiency, achieving the same performance with fewer than half of the gradient steps. LETI is equally applicable in natural language tasks when they can be formulated as code generation, which we empirically verified on event argument extraction.

翻译：对预训练语言模型（LM）进行微调是提升其能力的关键技术。现有方法通常基于输入-输出对（如指令微调）或衡量输出质量的数值奖励（如 RLHF）进行微调。我们探索语言模型从文本交互中学习（LETI）的潜力——该方法不仅通过二元标签检验模型输出的正确性，还能通过文本反馈精准定位并解释输出中的错误。我们聚焦代码生成任务，即模型需根据自然语言指令生成代码。该场景提供了一种自然且可扩展的文本反馈获取方式：使用 Python 解释器执行代码时产生的错误信息和堆栈跟踪。LETI 采用语言模型训练目标，将自然语言指令、LM 生成的程序与文本反馈串联后对模型进行迭代微调。在前置的微调文本中，我们使用二元奖励标记区分正确与错误解决方案。LETI 无需真实输出即可完成训练，甚至优于使用真实标签的微调基线。该方法不仅提升了 LM 在代码生成数据集 MBPP 上的性能，还能泛化至其他数据集。基于 MBPP 训练的模型在 HumanEval 的未见问题上表现达到或超越基础 LM 的基准水平。此外，相较于二元反馈，文本反馈可提升生成质量与样本效率，仅需不到一半的梯度更新步数即可达到相同性能。当自然语言任务可转化为代码生成形式时，LETI 同样适用——我们在事件论元抽取任务上通过实验验证了这一特性。