Nuclei panoptic segmentation supports cancer diagnostics by integrating both semantic and instance segmentation of different cell types to analyze overall tissue structure and individual nuclei in histopathology images. Major challenges include detecting small objects, handling ambiguous boundaries, and addressing class imbalance. To address these issues, we propose PanopMamba, a novel hybrid encoder-decoder architecture that integrates Mamba and Transformer with additional feature-enhanced fusion via state space modeling. We design a multiscale Mamba backbone and a State Space Model (SSM)-based fusion network to enable efficient long-range perception in pyramid features, thereby extending the pure encoder-decoder framework while facilitating information sharing across multiscale features of nuclei. The proposed SSM-based feature-enhanced fusion integrates pyramid feature networks and dynamic feature enhancement across different spatial scales, enhancing the feature representation of densely overlapping nuclei in both semantic and spatial dimensions. To the best of our knowledge, this is the first Mamba-based approach for panoptic segmentation. Additionally, we introduce alternative evaluation metrics, including image-level Panoptic Quality ($i$PQ), boundary-weighted PQ ($w$PQ), and frequency-weighted PQ ($fw$PQ), which are specifically designed to address the unique challenges of nuclei segmentation and thereby mitigate the potential bias inherent in vanilla PQ. Experimental evaluations on two multiclass nuclei segmentation benchmark datasets, MoNuSAC2020 and NuInsSeg, demonstrate the superiority of PanopMamba for nuclei panoptic segmentation over state-of-the-art methods. Consequently, the robustness of PanopMamba is validated across various metrics, while the distinctiveness of PQ variants is also demonstrated. Code is available at https://github.com/mkang315/PanopMamba.

翻译：细胞核全景分割通过整合不同细胞类型的语义分割与实例分割，以分析组织病理学图像中的整体组织结构和单个细胞核，从而支持癌症诊断。主要挑战包括检测小目标、处理模糊边界以及应对类别不平衡。为解决这些问题，我们提出了PanopMamba，一种新颖的混合编码器-解码器架构，该架构集成了Mamba与Transformer，并通过状态空间建模实现额外的特征增强融合。我们设计了一个多尺度Mamba主干网络和一个基于状态空间模型（SSM）的融合网络，以实现金字塔特征中的高效长程感知，从而扩展了纯编码器-解码器框架，同时促进了细胞核多尺度特征间的信息共享。所提出的基于SSM的特征增强融合整合了金字塔特征网络和跨不同空间尺度的动态特征增强，在语义和空间维度上提升了密集重叠细胞核的特征表示。据我们所知，这是首个基于Mamba的全景分割方法。此外，我们引入了替代性评估指标，包括图像级全景质量（$i$PQ）、边界加权全景质量（$w$PQ）和频率加权全景质量（$fw$PQ），这些指标专门设计用于应对细胞核分割的独特挑战，从而缓解原始全景质量（vanilla PQ）固有的潜在偏差。在两个多类别细胞核分割基准数据集MoNuSAC2020和NuInsSeg上的实验评估表明，PanopMamba在细胞核全景分割任务上优于现有最先进方法。因此，PanopMamba的鲁棒性在各种指标下得到了验证，同时全景质量变体的独特性也得到了证明。代码可在 https://github.com/mkang315/PanopMamba 获取。