Automated decision systems increasingly rely on human oversight to ensure accuracy in uncertain cases. This paper presents a practical framework for optimizing such human-in-the-loop classification systems using a double-threshold policy. Instead of relying on a single decision cutoff, the system defines two thresholds (a lower and an upper) to automatically accept or reject confident cases while routing ambiguous ones for human review. We formalize this problem as an optimization task that balances system accuracy against human review workload and demonstrate its behavior through extensive Monte Carlo simulations. Our results quantify how different probability score distributions affect the efficiency of human intervention and identify the regions of diminishing returns where additional review yields minimal benefit. The framework provides a general, reproducible method for improving reliability in any decision pipeline requiring selective human validation, including applications in entity resolution, fraud detection, medical triage, and content moderation.

翻译：自动化决策系统日益依赖人工监督来确保不确定案例的准确性。本文提出了一种实用的框架，通过使用双阈值策略来优化此类人在回路的分类系统。该系统不依赖单一决策截断点，而是定义两个阈值（下限与上限），以自动接受或拒绝置信度高的案例，同时将模糊案例转交人工审核。我们将该问题形式化为一项优化任务，旨在平衡系统准确性与人工审核工作量，并通过大量蒙特卡洛模拟展示了其行为。我们的结果量化了不同概率分数分布如何影响人工干预的效率，并识别出额外审核带来收益递减的区域。该框架提供了一种通用、可复现的方法，用于提升任何需要选择性人工验证的决策流程的可靠性，包括实体解析、欺诈检测、医疗分诊和内容审核等应用领域。