The risk-controlling prediction sets (RCPS) framework is a general tool for transforming the output of any machine learning model to design a predictive rule with rigorous error rate control. The key idea behind this framework is to use labeled hold-out calibration data to tune a hyper-parameter that affects the error rate of the resulting prediction rule. However, the limitation of such a calibration scheme is that with limited hold-out data, the tuned hyper-parameter becomes noisy and leads to a prediction rule with an error rate that is often unnecessarily conservative. To overcome this sample-size barrier, we introduce a semi-supervised calibration procedure that leverages unlabeled data to rigorously tune the hyper-parameter without compromising statistical validity. Our procedure builds upon the prediction-powered inference framework, carefully tailoring it to risk-controlling tasks. We demonstrate the benefits and validity of our proposal through two real-data experiments: few-shot image classification and early time series classification.

翻译：风险控制预测集（RCPS）框架是一种通用工具，可将任何机器学习模型的输出转化为具有严格错误率控制的预测规则。该框架的核心思想是利用带标签的保留校准数据来调整影响最终预测规则错误率的超参数。然而，这种校准方案的局限在于，当保留数据有限时，调整后的超参数会产生噪声，导致预测规则的错误率往往过于保守。为突破这一样本量限制，我们提出一种半监督校准方法，该方法利用未标注数据严格调整超参数，同时不损害统计有效性。我们的方法建立在预测驱动推断框架之上，并针对风险控制任务进行了精心适配。我们通过两个真实数据实验——少样本图像分类与早期时间序列分类——验证了所提方法的优势与有效性。