The Segment Anything Model (SAM), a profound vision foundation model pretrained on a large-scale dataset, breaks the boundaries of general segmentation and sparks various downstream applications. This paper introduces Hi-SAM, a unified model leveraging SAM for hierarchical text segmentation. Hi-SAM excels in segmentation across four hierarchies, including pixel-level text, word, text-line, and paragraph, while realizing layout analysis as well. Specifically, we first turn SAM into a high-quality pixel-level text segmentation (TS) model through a parameter-efficient fine-tuning approach. We use this TS model to iteratively generate the pixel-level text labels in a semi-automatical manner, unifying labels across the four text hierarchies in the HierText dataset. Subsequently, with these complete labels, we launch the end-to-end trainable Hi-SAM based on the TS architecture with a customized hierarchical mask decoder. During inference, Hi-SAM offers both automatic mask generation (AMG) mode and promptable segmentation (PS) mode. In the AMG mode, Hi-SAM segments pixel-level text foreground masks initially, then samples foreground points for hierarchical text mask generation and achieves layout analysis in passing. As for the PS mode, Hi-SAM provides word, text-line, and paragraph masks with a single point click. Experimental results show the state-of-the-art performance of our TS model: 84.86% fgIOU on Total-Text and 88.96% fgIOU on TextSeg for pixel-level text segmentation. Moreover, compared to the previous specialist for joint hierarchical detection and layout analysis on HierText, Hi-SAM achieves significant improvements: 4.73% PQ and 5.39% F1 on the text-line level, 5.49% PQ and 7.39% F1 on the paragraph level layout analysis, requiring $20\times$ fewer training epochs. The code is available at https://github.com/ymy-k/Hi-SAM.

翻译：Segment Anything Model（SAM）是一种基于大规模数据集预训练的深度视觉基础模型，它突破了通用分割的边界并催生了多种下游应用。本文提出Hi-SAM，这是一种利用SAM实现分层文本分割的统一模型。Hi-SAM在像素级文本、单词、文本行和段落四个层级均能实现卓越的分割性能，同时完成版面分析任务。具体而言，我们首先通过参数高效的微调方法将SAM转化为高质量的像素级文本分割（TS）模型。利用该TS模型以半自动方式迭代生成像素级文本标注，从而统一HierText数据集中四个文本层级的标注标准。随后，基于完整的标注数据，我们在TS架构基础上引入定制化的分层掩码解码器，构建可端到端训练的Hi-SAM模型。在推理阶段，Hi-SAM提供自动掩码生成（AMG）模式和可提示分割（PS）模式。在AMG模式下，Hi-SAM首先生成像素级文本前景掩码，随后对前景点进行采样以生成分层文本掩码，并同步完成版面分析。在PS模式下，Hi-SAM仅需单点点击即可提供单词、文本行及段落级别的掩码。实验结果表明，我们的TS模型在像素级文本分割任务上达到最先进性能：在Total-Text数据集上获得84.86%的fgIOU，在TextSeg数据集上获得88.96%的fgIOU。此外，与先前在HierText数据集上进行联合分层检测与版面分析的专用模型相比，Hi-SAM在显著减少20倍训练轮数的前提下，取得显著提升：文本行级别版面分析指标提升4.73% PQ和5.39% F1，段落级别提升5.49% PQ和7.39% F1。代码已开源：https://github.com/ymy-k/Hi-SAM。