While traditional line-level filtering techniques, such as line-level deduplication and trailing-punctuation filters, are commonly used, these basic methods can sometimes discard valuable content, negatively affecting downstream performance. In this paper, we introduce two methods-pattern-aware line-level deduplication (PLD) and pattern-aware trailing punctuation filtering (PTF)-by enhancing the conventional filtering techniques. Our approach not only considers line-level signals but also takes into account their sequential distribution across documents, enabling us to retain structurally important content that might otherwise be removed. We evaluate these proposed methods by training small language models (1 B parameters) in both English and Korean. The results demonstrate that our methods consistently improve performance on multiple-choice benchmarks and significantly enhance generative question-answering accuracy on both SQuAD v1 and KorQuAD v1.

翻译：尽管传统的行级过滤技术（如行级去重和尾随标点过滤）被广泛使用，但这些基础方法有时会丢弃有价值的内容，从而对下游任务性能产生负面影响。本文通过增强传统过滤技术，提出了两种方法——模式感知行级去重（PLD）和模式感知尾随标点过滤（PTF）。我们的方法不仅考虑行级特征，还兼顾其在文档中的序列分布模式，从而能够保留原本可能被删除的结构重要性内容。我们通过在英语和韩语上训练小规模语言模型（10亿参数）来评估所提方法。实验结果表明，我们的方法在多选题基准测试中持续提升性能，并在SQuAD v1和KorQuAD v1数据集上显著提高了生成式问答的准确率。