We introduce the notion of point affiliation into feature upsampling. By abstracting a feature map into non-overlapped semantic clusters formed by points of identical semantic meaning, feature upsampling can be viewed as point affiliation -- designating a semantic cluster for each upsampled point. In the framework of kernel-based dynamic upsampling, we show that an upsampled point can resort to its low-res decoder neighbors and high-res encoder point to reason the affiliation, conditioned on the mutual similarity between them. We therefore present a generic formulation for generating similarity-aware upsampling kernels and prove that such kernels encourage not only semantic smoothness but also boundary sharpness. This formulation constitutes a novel, lightweight, and universal upsampling solution, Similarity-Aware Point Affiliation (SAPA). We show its working mechanism via our preliminary designs with window-shape kernel. After probing the limitations of the designs on object detection, we reveal additional insights for upsampling, leading to SAPA with the dynamic kernel shape. Extensive experiments demonstrate that SAPA outperforms prior upsamplers and invites consistent performance improvements on a number of dense prediction tasks, including semantic segmentation, object detection, instance segmentation, panoptic segmentation, image matting, and depth estimation. Code is made available at: https://github.com/tiny-smart/sapa

翻译：我们引入点隶属概念于特征上采样中。通过将特征图抽象为由语义相同点形成的非重叠语义簇，特征上采样可被视为点隶属过程——为每个上采样点指定所属语义簇。在基于核的动态上采样框架中，我们证明上采样点可依据其与低分辨率解码器邻域点及高分辨率编码器点之间的互相似性来推断隶属关系。据此，我们提出一种生成相似性感知上采样核的通用公式，并证明此类核不仅能促进语义平滑性，还能增强边界锐利度。该公式构成一种新颖、轻量且通用的上采样方案——相似性感知点隶属（SAPA）。通过采用窗口形状核的初步设计，我们阐释了其工作机制。在探究该设计在目标检测中的局限性后，我们进一步揭示了上采样的深层见解，从而引出了具有动态核形状的SAPA。大量实验表明，SAPA优于先前上采样方法，并在语义分割、目标检测、实例分割、全景分割、图像抠图及深度估计等多项密集预测任务中持续带来性能提升。代码已开源至：https://github.com/tiny-smart/sapa