We introduce and study an online problem arising in question answering systems. In this problem, an agent must sequentially classify user-submitted queries represented by $d$-dimensional embeddings drawn i.i.d. from an unknown distribution. The agent may consult a costly human expert for the correct label, or guess on her own without receiving feedback. The goal is to minimize regret against an oracle with free expert access. When the time horizon $T$ is at least exponential in the embedding dimension $d$, one can learn the geometry of the class regions: in this regime, we propose the Conservative Hull-based Classifier (CHC), which maintains convex hulls of expert-labeled queries and calls the expert as soon as a query lands outside all known hulls. CHC attains $\mathcal{O}(\log^d T)$ regret in $T$ and is minimax optimal for $d=1$. Otherwise, the geometry cannot be reliably learned without additional distributional assumptions. We show that when the queries are drawn from a subgaussian mixture, for $T \le e^d$, a Center-based Classifier (CC) achieves regret proportional to $N\log{N}$ where $N$ is the number of labels. To bridge these regimes, we introduce the Generalized Hull-based Classifier (GHC), a practical extension of CHC that allows for more aggressive guessing via a tunable threshold parameter. Our approach is validated with experiments, notably on real-world question-answering datasets using embeddings derived from state-of-the-art large language models.

翻译：本文提出并研究了一个源自问答系统的在线问题。在该问题中，智能体必须对用户提交的查询进行顺序分类，这些查询由从未知分布中独立同分布抽取的$d$维嵌入向量表示。智能体可以选择咨询成本高昂的人类专家以获取正确标签，也可以自行猜测且不接收反馈。目标在于最小化相对于可免费访问专家的预言机的遗憾。当时间范围$T$至少为嵌入维度$d$的指数级时，可以学习类别区域的几何结构：在此机制下，我们提出了基于保守凸包的分类器（CHC），该分类器维护专家标记查询的凸包，并在查询落在所有已知凸包之外时调用专家。CHC在$T$内实现$\mathcal{O}(\log^d T)$的遗憾，且在$d=1$时达到极小极大最优。否则，若无额外的分布假设，几何结构无法被可靠地学习。我们证明，当查询从亚高斯混合分布中抽取时，对于$T \le e^d$，基于中心的分类器（CC）实现的遗憾与$N\log{N}$成正比，其中$N$为标签数量。为衔接这两种机制，我们引入了广义基于凸包的分类器（GHC），这是CHC的一种实用扩展，通过可调阈值参数允许更积极的猜测。我们的方法通过实验得到验证，特别是在使用源自最先进大语言模型的嵌入向量的真实世界问答数据集上。