The Parallel Meaning Bank (PMB) serves as a corpus for semantic processing with a focus on semantic parsing and text generation. Currently, we witness an excellent performance of neural parsers and generators on the PMB. This might suggest that such semantic processing tasks have by and large been solved. We argue that this is not the case and that performance scores from the past on the PMB are inflated by non-optimal data splits and test sets that are too easy. In response, we introduce several changes. First, instead of the prior random split, we propose a more systematic splitting approach to improve the reliability of the standard test data. Second, except for the standard test set, we also propose two challenge sets: one with longer texts including discourse structure, and one that addresses compositional generalization. We evaluate five neural models for semantic parsing and meaning-to-text generation. Our results show that model performance declines (in some cases dramatically) on the challenge sets, revealing the limitations of neural models when confronting such challenges.

翻译：平行意义库（PMB）作为语义处理语料库，聚焦于语义解析和文本生成任务。当前，神经解析器和生成器在PMB上表现优异，这可能意味着此类语义处理任务已基本解决。但我们认为事实并非如此，过往PMB上的性能得分因非最优数据划分和过于简单的测试集而被高估。为此我们提出多项改进：首先，摒弃原有的随机划分方式，提出更系统的数据分割方法以提升标准测试数据的可靠性；其次，除标准测试集外，我们还构建了两个挑战集——一个包含篇章结构的长文本集合，另一个聚焦组合泛化能力。我们评估了五种语义解析和意义到文本生成的神经模型。结果显示，模型在挑战集上的性能出现下降（部分案例中幅度显著），揭示了神经模型面对此类挑战时的局限性。