The sentence is a fundamental unit in many NLP applications. Sentence segmentation is widely used as the first preprocessing task, where an input text is split into consecutive sentences considering the end of the sentence (EOS) as their boundaries. This task formulation relies on a strong assumption that the input text consists only of sentences, or what we call the sentential units (SUs). However, real-world texts often contain non-sentential units (NSUs) such as metadata, sentence fragments, nonlinguistic markers, etc. which are unreasonable or undesirable to be treated as a part of an SU. To tackle this issue, we formulate a novel task of sentence identification, where the goal is to identify SUs while excluding NSUs in a given text. To conduct sentence identification, we propose a simple yet effective method which combines the beginning of the sentence (BOS) and EOS labels to determine the most probable SUs and NSUs based on dynamic programming. To evaluate this task, we design an automatic, language-independent procedure to convert the Universal Dependencies corpora into sentence identification benchmarks. Finally, our experiments on the sentence identification task demonstrate that our proposed method generally outperforms sentence segmentation baselines which only utilize EOS labels.

翻译：句子是许多自然语言处理应用中的基本单元。句子分割被广泛用作预处理的第一步，其中输入文本根据句子结束（EOS）作为边界被分割为连续的句子。这种任务公式化依赖于一个强假设，即输入文本仅由句子或我们称之为句子单元（SUs）组成。然而，现实世界的文本通常包含非句子单元（NSUs），如元数据、句子片段、非语言标记等，这些内容被视为SU的一部分是不合理或不可取的。为了解决这个问题，我们提出了一项新颖的句子识别任务，其目标是在给定文本中识别SUs，同时排除NSUs。为了进行句子识别，我们提出了一种简单而有效的方法，该方法结合了句子开始（BOS）和EOS标签，基于动态规划确定最可能的SUs和NSUs。为了评估这一任务，我们设计了一种自动化的、与语言无关的程序，将通用依赖关系语料库转换为句子识别基准。最后，我们在句子识别任务上的实验表明，我们提出的方法通常优于仅利用EOS标签的句子分割基线方法。