The sentence is a fundamental unit in many NLP applications. Sentence segmentation is widely used as the first preprocessing task, where an input text is split into consecutive sentences considering the end of the sentence (EOS) as their boundaries. This task formulation relies on a strong assumption that the input text consists only of sentences, or what we call the sentential units (SUs). However, real-world texts often contain non-sentential units (NSUs) such as metadata, sentence fragments, nonlinguistic markers, etc. which are unreasonable or undesirable to be treated as a part of an SU. To tackle this issue, we formulate a novel task of sentence identification, where the goal is to identify SUs while excluding NSUs in a given text. To conduct sentence identification, we propose a simple yet effective method which combines the beginning of the sentence (BOS) and EOS labels to determine the most probable SUs and NSUs based on dynamic programming. To evaluate this task, we design an automatic, language-independent procedure to convert the Universal Dependencies corpora into sentence identification benchmarks. Finally, our experiments on the sentence identification task demonstrate that our proposed method generally outperforms sentence segmentation baselines which only utilize EOS labels.

翻译：句分法被广泛用作第一个预处理任务,其中输入的文字被分为连续的句子,将句尾(EOS)作为界限。这一任务的措辞依据的是一个强有力的假设,即输入的文字仅包括句子,或我们所称的感应单位。然而,现实世界文本通常包含非文单位,如元数据、句号碎片、非语言标记等,不合理或不可取的、作为SU一部分处理的程序。为了解决这一问题,我们制定了一个新的句号识别任务,目标是在指定句号时识别SU,同时在给定的文本中排除NSUs。为了进行句识别,我们提出了一个简单而有效的方法,将句子开头(BOS)和EOS标签结合起来,以确定最可能的SUs和NSUs等。为了评估这项任务,我们设计了一个自动的、语言独立的程序,将通用的附属Corpora句识别基准转换成。为了解决这一问题,我们制订了一个新的句号识别基准,我们关于定义的实验只是使用我们提议的句子识别基准。