Large language models (LLMs) have achieved remarkable performance but face critical challenges: hallucinations and high inference costs. Leveraging multiple experts offers a solution: deferring uncertain inputs to more capable experts improves reliability, while routing simpler queries to smaller, distilled models enhances efficiency. This motivates the problem of learning with multiple-expert deferral. This thesis presents a comprehensive study of this problem and the related problem of learning with abstention, supported by strong consistency guarantees. First, for learning with abstention (a special case of deferral), we analyze score-based and predictor-rejector formulations in multi-class classification. We introduce new families of surrogate losses and prove strong non-asymptotic, hypothesis set-specific consistency guarantees, resolving two existing open questions. We analyze both single-stage and practical two-stage settings, with experiments on CIFAR-10, CIFAR-100, and SVHN demonstrating the superior performance of our algorithms. Second, we address general multi-expert deferral in classification. We design new surrogate losses for both single-stage and two-stage scenarios and prove they benefit from strong $H$-consistency bounds. For the two-stage scenario, we show that our surrogate losses are realizable $H$-consistent for constant cost functions, leading to effective new algorithms. Finally, we introduce a novel framework for regression with deferral to address continuous label spaces. Our versatile framework accommodates multiple experts and various cost structures, supporting both single-stage and two-stage methods. It subsumes recent work on regression with abstention. We propose new surrogate losses with proven $H$-consistency and demonstrate the empirical effectiveness of the resulting algorithms.

翻译：大型语言模型（LLMs）已取得显著性能，但仍面临关键挑战：幻觉问题与高昂推理成本。利用多专家系统提供了一种解决方案：将不确定的输入递延给能力更强的专家可提升可靠性，同时将简单查询路由至更小规模的蒸馏模型可提高效率。这催生了多专家递延学习问题。本论文对该问题及相关弃权学习问题进行了系统研究，并提供了强一致性理论保证。首先，针对弃权学习（递延的特殊情形），我们分析了多类别分类中基于评分和预测器-拒绝器的形式化框架。我们提出了新的替代损失函数族，并证明了强非渐近、假设集特定的理论一致性保证，解决了两个现有开放性问题。我们分析了单阶段和实用的两阶段设置，在CIFAR-10、CIFAR-100和SVHN数据集上的实验验证了所提算法的优越性能。其次，我们研究分类中通用的多专家递延问题。针对单阶段和两阶段场景分别设计了新的替代损失函数，并证明其具有强$H$一致性界。对于两阶段场景，我们证明所提替代损失函数在常数成本函数下可实现$H$一致性，从而推导出高效的新算法。最后，我们提出了面向连续标签空间的回归递延新框架。该通用框架兼容多专家系统和多种成本结构，支持单阶段与两阶段方法，并涵盖了近期回归弃权研究。我们提出了具有可证明$H$一致性的新型替代损失函数，并通过实验验证了所衍生算法的实际有效性。