Despite the impressive generative abilities of black-box large language models (LLMs), their inherent opacity hinders further advancements in capabilities such as reasoning, planning, and personalization. Existing works aim to enhance LLM capabilities via domain-specific adaptation or in-context learning, which require additional training on accessible model parameters, an infeasible option for black-box LLMs. To address this challenge, we introduce Matryoshika, a lightweight white-box LLM controller that guides a large-scale black-box LLM generator by decomposing complex tasks into a series of intermediate outputs. Specifically, we consider the black-box LLM as an environment, with Matryoshika serving as a policy to provide intermediate guidance through prompts for driving the black-box LLM. Matryoshika is trained to pivot the outputs of the black-box LLM aligning with preferences during iterative interaction, which enables controllable multi-turn generation and self-improvement in optimizing intermediate guidance. Empirical evaluations on three diverse tasks demonstrate that Matryoshika effectively enhances the capabilities of black-box LLMs in complex, long-horizon tasks, including reasoning, planning, and personalization. By leveraging this pioneering controller-generator framework to mitigate dependence on model parameters, Matryoshika provides a transparent and practical solution for improving black-box LLMs through controllable multi-turn generation using white-box LLMs.

翻译：尽管黑盒大语言模型（LLM）展现出卓越的生成能力，其固有的不透明性阻碍了在推理、规划与个性化等能力上的进一步发展。现有研究主要通过领域适应性调整或上下文学习来增强LLM能力，这些方法需要对可访问的模型参数进行额外训练，这对黑盒LLM而言并不可行。为应对这一挑战，我们提出Matryoshika——一种轻量级白盒LLM控制器，通过将复杂任务分解为一系列中间输出来指导大规模黑盒LLM生成器。具体而言，我们将黑盒LLM视为环境，Matryoshika则作为策略通过提示提供中间指导以驱动黑盒LLM。Matryoshika经过训练，可在迭代交互过程中调整黑盒LLM的输出以符合预期偏好，从而实现可控的多轮生成及优化中间指导的自我改进能力。在三个多样化任务上的实证评估表明，Matryoshika能有效提升黑盒LLM在复杂长程任务（包括推理、规划与个性化）中的能力。通过采用这种开创性的控制器-生成器框架来降低对模型参数的依赖，Matryoshika为利用白盒LLM实现可控多轮生成提供了一种透明且实用的黑盒LLM增强方案。