The success of large language models has shifted the evaluation paradigms in natural language processing (NLP). The community's interest has drifted towards comparing NLP models across many tasks, domains, and datasets, often at an extreme scale. This imposes new engineering challenges: efforts in constructing datasets and models have been fragmented, and their formats and interfaces are incompatible. As a result, it often takes extensive (re)implementation efforts to make fair and controlled comparisons at scale. Catwalk aims to address these issues. Catwalk provides a unified interface to a broad range of existing NLP datasets and models, ranging from both canonical supervised training and fine-tuning, to more modern paradigms like in-context learning. Its carefully-designed abstractions allow for easy extensions to many others. Catwalk substantially lowers the barriers to conducting controlled experiments at scale. For example, we finetuned and evaluated over 64 models on over 86 datasets with a single command, without writing any code. Maintained by the AllenNLP team at the Allen Institute for Artificial Intelligence (AI2), Catwalk is an ongoing open-source effort: https://github.com/allenai/catwalk.

翻译：大型语言模型的出现改变了自然语言处理（NLP）领域的评估范式。社区兴趣已转向在大量任务、领域和数据集间进行跨模型比较，且常以极端规模展开。这带来了新的工程挑战：数据集与模型的构建工作长期碎片化，其格式与接口互不兼容。因此，进行大规模公平可控的比较往往需要大量（重新）实现工作。Catwalk 旨在解决这些问题。它提供统一接口，覆盖从经典监督训练与微调，到上下文学习等现代范式的广泛现有NLP数据集与模型。其精心设计的抽象机制使得模型与数据集易于扩展。Catwalk 大幅降低了大规模受控实验的门槛——例如，我们通过单条命令对86个数据集上的64个模型进行了微调与评估，无需编写任何代码。作为艾伦人工智能研究所（AI2）AllenNLP团队维护的开源项目，Catwalk 仍在持续演进中。项目地址：https://github.com/allenai/catwalk。