One of the primary catalysts fueling advances in artificial intelligence (AI) and machine learning (ML) is the availability of massive, curated datasets. A commonly used technique to curate such massive datasets is crowdsourcing, where data are dispatched to multiple annotators. The annotator-produced labels are then fused to serve downstream learning and inference tasks. This annotation process often creates noisy labels due to various reasons, such as the limited expertise, or unreliability of annotators, among others. Therefore, a core objective in crowdsourcing is to develop methods that effectively mitigate the negative impact of such label noise on learning tasks. This feature article introduces advances in learning from noisy crowdsourced labels. The focus is on key crowdsourcing models and their methodological treatments, from classical statistical models to recent deep learning-based approaches, emphasizing analytical insights and algorithmic developments. In particular, this article reviews the connections between signal processing (SP) theory and methods, such as identifiability of tensor and nonnegative matrix factorization, and novel, principled solutions of longstanding challenges in crowdsourcing -- showing how SP perspectives drive the advancements of this field. Furthermore, this article touches upon emerging topics that are critical for developing cutting-edge AI/ML systems, such as crowdsourcing in reinforcement learning with human feedback (RLHF) and direct preference optimization (DPO) that are key techniques for fine-tuning large language models (LLMs).

翻译：推动人工智能（AI）和机器学习（ML）进步的主要催化剂之一是海量、经人工标注的数据集的可用性。构建此类海量数据集的常用技术是众包，即将数据分发给多个标注者。标注者产生的标签随后被融合，以服务于下游的学习和推理任务。由于多种原因，例如标注者专业知识有限或不可靠等，此标注过程常常会产生噪声标签。因此，众包的一个核心目标是开发能够有效减轻此类标签噪声对学习任务负面影响的方法。本专题文章介绍了从噪声众包标签中学习的研究进展。重点在于关键的众包模型及其方法论处理，从经典的统计模型到近期基于深度学习的方法，并强调分析见解和算法发展。特别地，本文回顾了信号处理（SP）理论与方法（例如张量分解和非负矩阵分解的可辨识性）与解决众包中长期挑战的新颖、原理性解决方案之间的联系——展示了SP视角如何推动该领域的进步。此外，本文还涉及对开发前沿AI/ML系统至关重要的新兴主题，例如强化学习与人类反馈（RLHF）中的众包以及直接偏好优化（DPO），这些是微调大语言模型（LLM）的关键技术。