Training or fine-tuning large language model (LLM)-based systems often requires costly human feedback, yet there is limited understanding of how to minimize such intervention while maintaining strong error guarantees. We study this problem for LLM-based classification systems in an active learning framework: an agent sequentially labels $d$-dimensional query embeddings drawn i.i.d. from an unknown distribution by either calling a costly expert or guessing with no feedback, with the goal of minimizing regret relative to an oracle with free expert access. When the horizon $T$ is at least exponential in the embedding dimension $d$, the geometry of the class regions can be learned. In this regime, we propose the Conservative Hull-based Classifier (CHC), which maintains convex hulls of expert-labeled queries and calls the expert when a query lands outside all known hulls. CHC attains $\mathcal{O}(\log^d T)$ regret in $T$ and is minimax optimal for $d=1$. Otherwise, the geometry cannot be reliably learned in general. We show that for queries drawn from a subgaussian mixture and $T \le e^d$, a Center-based Classifier (CC) achieves regret proportional to $N\log{N}$ where $N$ is the number of labels. To bridge these regimes, we introduce the Generalized Hull-based Classifier (GHC), a practical extension of CHC that enables more aggressive guessing via a tunable parameter. Our approach is validated on real-world question-answering datasets using state-of-the-art text embedding models.

翻译：训练或微调基于大语言模型（LLM）的系统往往需要高昂的人工反馈成本，然而关于如何在保证强误差保证的前提下最小化此类干预的研究尚不充分。本文在主动学习框架下针对基于LLM的分类系统展开研究：代理通过两种方式对独立同分布于未知分布的$d$维查询嵌入进行序列标注——调用昂贵的人工专家或零反馈的自发猜测，目标是最小化相对于具备免费专家调用的理想模型的遗憾值。当时间范围$T$至少关于嵌入维度$d$呈指数增长时，类别区域的几何结构可被学习。在该场景下，我们提出基于保守凸包分类器（CHC）：该方法维护经专家标注的查询凸包，当新查询落于所有已知凸包之外时调用专家。CHC在$T$范围内实现$\mathcal{O}(\log^d T)$的遗憾值，且当$d=1$时达到极小极大最优。反之，一般情况下几何结构无法被可靠学习。我们证明对于服从亚高斯混合分布的查询且$T \le e^d$时，基于中心分类器（CC）的遗憾值与标签数$N$满足$N\log{N}$比例关系。为衔接上述两个场景，我们提出广义凸包分类器（GHC）——CHC的实用扩展版本，通过可调参数实现更激进的猜测策略。基于最先进文本嵌入模型在真实问答数据集上的实验验证了本方法的有效性。