This paper presents Crowd-Kit, a general-purpose computational quality control toolkit for crowdsourcing. Crowd-Kit provides efficient and convenient implementations of popular quality control algorithms in Python, including methods for truth inference, deep learning from crowds, and data quality estimation. Our toolkit supports multiple modalities of answers and provides dataset loaders and example notebooks for faster prototyping. We extensively evaluated our toolkit on several datasets of different natures, enabling benchmarking computational quality control methods in a uniform, systematic, and reproducible way using the same codebase. We release our code and data under the Apache License 2.0 at https://github.com/Toloka/crowd-kit.

翻译：本文介绍Crowd-Kit，一个面向众包的通用计算质量控制工具包。Crowd-Kit以Python语言提供了流行质量控制算法的高效便捷实现，涵盖真值推断、众包深度学习及数据质量评估等方法。该工具包支持多种答案模态，并提供数据集加载器和示例笔记，以加速原型开发。我们使用同一代码库，在多个不同性质的数据集上对工具包进行了全面评估，从而实现了计算质量控制方法在统一、系统且可复现条件下的基准测试。我们依据Apache许可证2.0在https://github.com/Toloka/crowd-kit开放了代码与数据。