Data annotation remains a significant bottleneck in the Humanities and Social Sciences, particularly for complex semantic tasks such as metaphor identification. While Large Language Models (LLMs) show promise, a significant gap remains between the theoretical capability of LLMs and their practical utility for researchers. This paper introduces LinguistAgent, an integrated, user-friendly platform that leverages a reflective multi-model architecture to automate linguistic annotation. The system implements a dual-agent workflow, comprising an Annotator and a Reviewer, to simulate a professional peer-review process. LinguistAgent supports comparative experiments across three paradigms: Prompt Engineering (Zero/Few-shot), Retrieval-Augmented Generation, and Fine-tuning. We demonstrate LinguistAgent's efficacy using the task of metaphor identification as an example, providing real-time token-level evaluation (Precision, Recall, and $F_1$ score) against human gold standards. The application and codes are released on https://github.com/Bingru-Li/LinguistAgent.
翻译:数据标注在人文与社会科学领域仍是一个显著的瓶颈,尤其对于隐喻识别等复杂语义任务。尽管大型语言模型展现出潜力,但其理论能力与研究者的实际应用需求之间仍存在显著差距。本文介绍 LinguistAgent,一个集成化、用户友好的平台,它利用反思性多模型架构实现语言标注的自动化。该系统采用双智能体工作流,包含标注器与审核器,以模拟专业同行评审流程。LinguistAgent 支持三种范式的对比实验:提示工程(零样本/少样本)、检索增强生成与微调。我们以隐喻识别任务为例,通过实时词元级评估(精确率、召回率与 $F_1$ 分数)对比人工标注金标准,验证了 LinguistAgent 的有效性。应用程序与代码已发布于 https://github.com/Bingru-Li/LinguistAgent。