In this work, we present VNLP: the first dedicated, complete, open-source, well-documented, lightweight, production-ready, state-of-the-art Natural Language Processing (NLP) package for the Turkish language. It contains a wide variety of tools, ranging from the simplest tasks, such as sentence splitting and text normalization, to the more advanced ones, such as text and token classification models. Its token classification models are based on "Context Model", a novel architecture that is both an encoder and an auto-regressive model. NLP tasks solved by VNLP models include but are not limited to Sentiment Analysis, Named Entity Recognition, Morphological Analysis \& Disambiguation and Part-of-Speech Tagging. Moreover, it comes with pre-trained word embeddings and corresponding SentencePiece Unigram tokenizers. VNLP has an open-source GitHub repository, ReadtheDocs documentation, PyPi package for convenient installation, Python and command-line API and a demo page to test all the functionality. Consequently, our main contribution is a complete, compact, easy-to-install and easy-to-use NLP package for Turkish.
翻译:本文介绍VNLP:首个专为土耳其语设计、功能完备、开源、文档详尽、轻量级、可投产且达到当前最优水平的自然语言处理(NLP)工具包。该工具包包含从句子分割、文本规范化等简单任务,到文本分类与Token分类模型等高级功能的广泛工具。其Token分类模型基于“上下文模型”(Context Model)——一种兼具编码器与自回归模型特性的新型架构。VNLP模型已解决的情感分析、命名实体识别、形态学分析与消歧、词性标注等NLP任务尚不限于此。此外,该工具包提供预训练词向量及配套的SentencePiece Unigram分词器。VNLP拥有开源GitHub仓库、ReadtheDocs文档、便于安装的PyPi包、Python与命令行API及测试全功能的演示页面。综上,本工作的核心贡献是为土耳其语打造了一个完整、精简、易于安装和使用的NLP工具包。