There are two main Parameter-Efficient Fine-Tuning (PEFT) techniques for Large Language Models (LLMs). While Low-Rank Adaptation (LoRA) introduces additional weights between the LLM layers, Soft Prompting introduces additional fine-tuning-specific raw tokens to an LLM input. However, both require modification to the computational graphs of precompiled, preoptimized LLMs. As a result, neither is fully supported in high-throughput engines like vLLM. We propose fine-tuning with ART (Art-based Reinforcement Training). The method injects information into a frozen Multimodal Large Language Model (MLLM) by optimizing only its raw visual input, thus enabling the soft-token approach on pre-compiled computational graphs. It relies on backpropagation of gradients back into a plain pixel array and thus supports any fine-tuning objective. Moreover, the optimized visual input can be stylized as task-relevant computational artworks. The approach's effectiveness is confirmed for different sizes of a popular open Qwen architecture and for several textual benchmarks. Specifically, ART reaches accuracy competitive with LoRA across mathematics and structured-tool-use benchmarks.

翻译：大语言模型主要有两种参数高效微调技术。低秩适应通过在LLM层之间引入额外权重，而软提示则向LLM输入添加特定于微调的原始标记。然而，这两种方法都需要修改预编译、预优化的LLM的计算图。因此，两者在vLLM等高吞吐量引擎中均未得到完全支持。我们提出基于ART（艺术强化训练）的微调方法。该方法通过仅优化冻结的多模态大语言模型的原始视觉输入来注入信息，从而在预编译的计算图上实现软标记方法。它依赖于梯度反向传播到普通像素阵列中，因此支持任何微调目标。此外，优化后的视觉输入可被风格化为与任务相关的计算艺术品。该方法在流行的开源Qwen架构的不同规模以及多个文本基准上的有效性得到了验证。具体而言，ART在数学和结构化工具使用基准测试中达到了与LoRA相当的准确率。