Text segmentation is a fundamental task in natural language processing, where documents are split into contiguous sections. However, prior research in this area has been constrained by limited datasets, which are either small in scale, synthesized, or only contain well-structured documents. In this paper, we address these limitations by introducing a novel benchmark YTSeg focusing on spoken content that is inherently more unstructured and both topically and structurally diverse. As part of this work, we introduce an efficient hierarchical segmentation model MiniSeg, that outperforms state-of-the-art baselines. Lastly, we expand the notion of text segmentation to a more practical "smart chaptering" task that involves the segmentation of unstructured content, the generation of meaningful segment titles, and a potential real-time application of the models.

翻译：文本分割是自然语言处理中的基础任务，其目标是将文档划分为连续的段落。然而，该领域的先前研究受限于数据集规模有限、多为合成数据或仅包含结构良好的文档。本文通过提出YTSeg这一新型基准来突破上述局限，该基准聚焦于口语内容，其本质更具非结构性且主题与结构呈现多样性。作为本研究的一部分，我们提出了高效分层分割模型MiniSeg，其性能超越当前最优基线。最后，我们将文本分割概念拓展为更具实用性的"智能章节划分"任务，该任务涉及非结构化内容分割、有意义的章节标题生成，以及模型的潜在实时应用场景。