Textual noise, such as typos or abbreviations, is a well-known issue that penalizes vanilla Transformers for most downstream tasks. We show that this is also the case for sentence similarity, a fundamental task in multiple domains, e.g. matching, retrieval or paraphrasing. Sentence similarity can be approached using cross-encoders, where the two sentences are concatenated in the input allowing the model to exploit the inter-relations between them. Previous works addressing the noise issue mainly rely on data augmentation strategies, showing improved robustness when dealing with corrupted samples that are similar to the ones used for training. However, all these methods still suffer from the token distribution shift induced by typos. In this work, we propose to tackle textual noise by equipping cross-encoders with a novel LExical-aware Attention module (LEA) that incorporates lexical similarities between words in both sentences. By using raw text similarities, our approach avoids the tokenization shift problem obtaining improved robustness. We demonstrate that the attention bias introduced by LEA helps cross-encoders to tackle complex scenarios with textual noise, specially in domains with short-text descriptions and limited context. Experiments using three popular Transformer encoders in five e-commerce datasets for product matching show that LEA consistently boosts performance under the presence of noise, while remaining competitive on the original (clean) splits. We also evaluate our approach in two datasets for textual entailment and paraphrasing showing that LEA is robust to typos in domains with longer sentences and more natural context. Additionally, we thoroughly analyze several design choices in our approach, providing insights about the impact of the decisions made and fostering future research in cross-encoders dealing with typos.

翻译：文本噪声（如拼写错误或缩写）是导致标准Transformer模型在多数下游任务中性能下降的公认问题。我们证明，在多个领域（如匹配、检索或释义）的基础任务——句子相似性中，该问题同样存在。句子相似性可通过交叉编码器实现，其输入中将两句子拼接，使模型能利用两者间的相互关系。现有针对噪声问题的研究主要依赖数据增强策略，在处理与训练样本相似的损坏样本时表现出更强的鲁棒性。然而，这些方法仍受限于拼写错误引发的词元分布偏移。本研究提出通过为交叉编码器配备新型词汇感知注意力模块（LEA）来应对文本噪声，该模块能融合两个句子中词汇间的语义相似性。通过利用原始文本相似性，我们的方法规避了词元化偏移问题，从而提升鲁棒性。实验表明，LEA引入的注意力偏误可帮助交叉编码器应对含文本噪声的复杂场景，尤其在短文本描述和有限上下文领域表现突出。基于五种电商产品匹配数据集，使用三种主流Transformer编码器的实验显示：LEA在噪声环境下持续提升性能，同时在原始（纯净）数据集上保持竞争力。我们还在两个文本蕴含和释义数据集上验证了该方法，证明LEA在长句及更自然上下文的领域中同样对拼写错误具有鲁棒性。此外，我们深入分析了方法中的多项设计选择，为决策影响提供见解，并推动未来关于处理拼写错误的交叉编码器研究。