Recent advances in the area of long document matching have primarily focused on using transformer-based models for long document encoding and matching. There are two primary challenges associated with these models. Firstly, the performance gain provided by transformer-based models comes at a steep cost - both in terms of the required training time and the resource (memory and energy) consumption. The second major limitation is their inability to handle more than a pre-defined input token length at a time. In this work, we empirically demonstrate the effectiveness of simple neural models (such as feed-forward networks, and CNNs) and simple embeddings (like GloVe, and Paragraph Vector) over transformer-based models on the task of document matching. We show that simple models outperform the more complex BERT-based models while taking significantly less training time, energy, and memory. The simple models are also more robust to variations in document length and text perturbations.

翻译：近期长文档匹配领域的研究主要聚焦于基于Transformer模型的长文档编码与匹配。这类模型面临两大核心挑战：其一，Transformer模型带来的性能提升是以高昂代价换取的——包括训练时间激增与资源（内存及能耗）消耗；其二，它们无法同时处理超出预设输入长度的文本。本研究通过实证证明，在文档匹配任务中，简单神经模型（如前馈网络与CNN）及简易嵌入方法（如GloVe与段落向量）相比Transformer模型更具实效。实验表明，简单模型不仅能在显著缩短训练时间、降低能耗和内存占用的前提下超越复杂的BERT模型表现，而且对文档长度变化和文本扰动具有更强的鲁棒性。