With the rapid accumulation of text data produced by data-driven techniques, the task of extracting "data annotations"--concise, high-quality data summaries from unstructured raw text--has become increasingly important. The recent advances in weak supervision and crowd-sourcing techniques provide promising solutions to efficiently create annotations (labels) for large-scale technical text data. However, such annotations may fail in practice because of the change in annotation requirements, application scenarios, and modeling goals, where label validation and relabeling by domain experts are required. To approach this issue, we present LabelVizier, a human-in-the-loop workflow that incorporates domain knowledge and user-specific requirements to reveal actionable insights into annotation flaws, then produce better-quality labels for large-scale multi-label datasets. We implement our workflow as an interactive notebook to facilitate flexible error profiling, in-depth annotation validation for three error types, and efficient annotation relabeling on different data scales. We evaluated the efficiency and generalizability of our workflow with two use cases and four expert reviews. The results indicate that LabelVizier is applicable in various application scenarios and assist domain experts with different knowledge backgrounds to efficiently improve technical text annotation quality.

翻译：随着数据驱动技术产生的文本数据迅速积累，从非结构化原始文本中提取“数据标注”（即简洁、高质量的数据摘要）的任务变得愈发重要。近期弱监督和众包技术的进展为大规模技术文本数据的高效标注提供了有前景的解决方案。然而，由于标注需求、应用场景和建模目标的变化，此类标注在实践中可能出现失效，需要领域专家进行标注验证和重新标注。针对这一问题，我们提出了LabelVizier——一种融入领域知识和用户特定需求的人机协同工作流，能够揭示标注缺陷中的可操作洞察，进而为大规模多标签数据集生成更高质量的标注。我们将该工作流实现为交互式笔记本，以支持灵活的误差分析、针对三种错误类型的深度标注验证，以及不同数据规模下的高效标注重标。通过两个用例和四次专家评估，我们验证了该工作流的效率与泛化能力。结果表明，LabelVizier适用于多种应用场景，并能帮助具有不同知识背景的领域专家高效提升技术文本标注质量。