Compositional generalisation (CG), in NLP and in machine learning more generally, has been assessed mostly using artificial datasets. It is important to develop benchmarks to assess CG also in real-world natural language tasks in order to understand the abilities and limitations of systems deployed in the wild. To this end, our GenBench Collaborative Benchmarking Task submission utilises the distribution-based compositionality assessment (DBCA) framework to split the Europarl translation corpus into a training and a test set in such a way that the test set requires compositional generalisation capacity. Specifically, the training and test sets have divergent distributions of dependency relations, testing NMT systems' capability of translating dependencies that they have not been trained on. This is a fully-automated procedure to create natural language compositionality benchmarks, making it simple and inexpensive to apply it further to other datasets and languages. The code and data for the experiments is available at https://github.com/aalto-speech/dbca.

翻译：组合泛化（CG）在自然语言处理及更广泛的机器学习领域中，主要依赖人工数据集进行评估。为理解实际部署系统的能力与局限，亟需开发面向真实自然语言任务的CG评估基准。为此，我们的GenBench协作基准测试任务采用基于分布的组合性评估（DBCA）框架，将Europarl翻译语料库划分为训练集和测试集，使测试集必须依赖组合泛化能力才能完成翻译。具体而言，训练集与测试集在依存关系上具有不同分布，以此测试神经机器翻译系统翻译未经训练的依存结构的能力。该流程完全自动化，可便捷、低成本地应用于其他数据集和语言，用于构建自然语言组合性基准。实验代码与数据见https://github.com/aalto-speech/dbca。