This tool demonstration presents a research toolkit for a language model of Java source code. The target audience includes researchers studying problems at the granularity level of subroutines, statements, or variables in Java. In contrast to many existing language models, we prioritize features for researchers including an open and easily-searchable training set, a held out test set with different levels of deduplication from the training set, infrastructure for deduplicating new examples, and an implementation platform suitable for execution on equipment accessible to a relatively modest budget. Our model is a GPT2-like architecture with 350m parameters. Our training set includes 52m Java methods (9b tokens) and 13m StackOverflow threads (10.5b tokens). To improve accessibility of research to more members of the community, we limit local resource requirements to GPUs with 16GB video memory. We provide a test set of held out Java methods that include descriptive comments, including the entire Java projects for those methods. We also provide deduplication tools using precomputed hash tables at various similarity thresholds to help researchers ensure that their own test examples are not in the training set. We make all our tools and data open source and available via Huggingface and Github.

翻译：本工具演示介绍了一个面向Java源代码语言模型的研究工具包。目标受众包括研究Java中子程序、语句或变量粒度级别问题的研究人员。与许多现有语言模型不同，我们优先考虑面向研究人员的特性：包括开放且易于搜索的训练集、与训练集具有不同去重级别的保留测试集、用于对新示例进行去重的基础设施，以及适用于预算相对有限的设备上执行的实现平台。我们的模型采用类似GPT2的架构，包含3.5亿个参数。训练集包含5200万个Java方法（90亿个token）和1300万个StackOverflow线程（105亿个token）。为提升社区更多成员参与研究的可及性，我们将本地资源需求限制为具有16GB显存的GPU。我们提供了包含描述性注释的保留Java方法测试集，并包含这些方法对应的完整Java项目。同时，我们还提供了使用预计算哈希表在不同相似度阈值下进行去重的工具，帮助研究人员确保其测试用例不在训练集中。我们已将全部工具和数据开源，并在Huggingface和Github上发布。