Both CNN-based and Transformer-based object detection with bounding box representation have been extensively studied in computer vision and medical image analysis, but circular object detection in medical images is still underexplored. Inspired by the recent anchor free CNN-based circular object detection method (CircleNet) for ball-shape glomeruli detection in renal pathology, in this paper, we present CircleFormer, a Transformer-based circular medical object detection with dynamic anchor circles. Specifically, queries with circle representation in Transformer decoder iteratively refine the circular object detection results, and a circle cross attention module is introduced to compute the similarity between circular queries and image features. A generalized circle IoU (gCIoU) is proposed to serve as a new regression loss of circular object detection as well. Moreover, our approach is easy to generalize to the segmentation task by adding a simple segmentation branch to CircleFormer. We evaluate our method in circular nuclei detection and segmentation on the public MoNuSeg dataset, and the experimental results show that our method achieves promising performance compared with the state-of-the-art approaches. The effectiveness of each component is validated via ablation studies as well. Our code is released at: \url{https://github.com/zhanghx-iim-ahu/CircleFormer}.

翻译：基于边界框表示的CNN和Transformer目标检测方法在计算机视觉和医学图像分析领域已得到广泛研究，但医学图像中的圆形目标检测仍探索不足。受近期针对肾病理学中球状肾小球检测的无锚框CNN圆形目标检测方法(CircleNet)启发，本文提出CircleFormer——一种基于动态锚框圆形的Transformer圆形医学目标检测方法。具体而言，Transformer解码器中采用圆形表示的查询迭代优化圆形目标检测结果，并引入圆形交叉注意力模块计算圆形查询与图像特征之间的相似度。我们提出广义圆形交并比(gCIoU)作为圆形目标检测的新回归损失函数。此外，通过添加简单分割分支，该方法可轻松泛化至分割任务。在公开MoNuSeg数据集上的圆形细胞核检测与分割实验表明，与现有最优方法相比，本方法取得了优越性能。通过消融研究验证了各组件的有效性。代码开源地址：\url{https://github.com/zhanghx-iim-ahu/CircleFormer}。