Despite the vast body of literature on Active Learning (AL), there is no comprehensive and open benchmark allowing for efficient and simple comparison of proposed samplers. Additionally, the variability in experimental settings across the literature makes it difficult to choose a sampling strategy, which is critical due to the one-off nature of AL experiments. To address those limitations, we introduce OpenAL, a flexible and open-source framework to easily run and compare sampling AL strategies on a collection of realistic tasks. The proposed benchmark is augmented with interpretability metrics and statistical analysis methods to understand when and why some samplers outperform others. Last but not least, practitioners can easily extend the benchmark by submitting their own AL samplers.

翻译：尽管主动学习领域已有大量文献，但仍缺乏一个全面且开放的基准测试框架，能够高效便捷地比较各类采样策略。此外，现有文献中实验设置的差异性导致采样策略的选择变得困难——由于主动学习实验具有一次性特点，这一选择至关重要。为解决上述局限，我们提出OpenAL——一个灵活的开源框架，可在系列现实任务上轻松运行并比较主动学习采样策略。该基准测试框架额外配备了可解释性指标与统计分析方法，以理解不同采样器性能优劣的时机与原因。尤为重要的是，从业者可通过提交自定义的主动学习采样器轻松扩展该基准测试框架。