Self-supervised pretraining has transformed computer vision by enabling data-efficient fine-tuning, yet high-resolution training typically requires server-scale infrastructure, limiting in-domain foundation model development for many research laboratories. Masked Autoencoders (MAE) reduce computation by encoding only visible tokens, but combining MAE with hierarchical downsampling architectures remains structurally challenging due to dense grid priors and mask-aware design compromises. We introduce AFFMAE, a masking-friendly hierarchical pretraining framework built on adaptive, off-grid token merging. By discarding masked tokens and performing dynamic merging exclusively over visible tokens, AFFMAE removes dense-grid assumptions while preserving hierarchical scalability. We developed numerically stable mixed-precision Flash-style cluster attention kernels, and mitigate sparse-stage representation collapse via deep supervision. On high-resolution electron microscopy segmentation, AFFMAE matches ViT-MAE performance at equal parameter count while reducing FLOPs by up to 7x, halving memory usage, and achieving faster training on a single RTX 5090. Code available at https://github.com/najafian-lab/affmae.

翻译：自监督预训练通过实现数据高效微调，彻底改变了计算机视觉领域，然而高分辨率训练通常需要服务器级基础设施，这限制了许多研究实验室开发领域内基础模型。掩码自编码器（MAE）通过仅编码可见标记来减少计算量，但由于密集网格先验和掩码感知设计折衷，将MAE与分层下采样架构结合仍存在结构挑战。本文提出AFFMAE，一种基于自适应非网格标记融合的掩码友好型分层预训练框架。通过丢弃掩码标记并仅对可见标记执行动态融合，AFFMAE在保持分层可扩展性的同时消除了密集网格假设。我们开发了数值稳定的混合精度Flash风格聚类注意力内核，并通过深度监督缓解稀疏阶段表征坍缩问题。在高分辨率电子显微镜分割任务中，AFFMAE在参数量相同的情况下达到与ViT-MAE相当的性能，同时将FLOPs降低高达7倍，内存使用减半，并在单张RTX 5090显卡上实现更快的训练。代码发布于https://github.com/najafian-lab/affmae。