Despite recent progress, vision-language encoders struggle with two core limitations: (1) weak alignment between language and dense vision features, which hurts tasks like open-vocabulary semantic segmentation; and (2) high token counts for fine-grained visual representations, which limits scalability to long videos. This work addresses both limitations. We propose T-REN (Text-aligned Region Encoder Network), an efficient encoder that maps visual data to a compact set of text-aligned region-level representations (or region tokens). T-REN achieves this through a lightweight network added on top of a frozen vision backbone, trained to pool patch-level representations within each semantic region into region tokens and align them with region-level text annotations. With only 3.7% additional parameters compared to the vision-language backbone, this design yields substantially stronger dense cross-modal understanding while reducing the token count by orders of magnitude. Specifically, T-REN delivers +5.9 mIoU on ADE20K open-vocabulary segmentation, +18.4% recall on COCO object-level text-image retrieval, +15.6% recall on Ego4D video object localization, and +17.6% mIoU on VSPW video scene parsing, all while reducing token counts by more than 24x for images and 187x for videos compared to the patch-based vision-language backbone. The code and model are available at https://github.com/savya08/T-REN.

翻译：尽管近期取得了进展，视觉-语言编码器仍面临两个核心局限：（1）语言与稠密视觉特征之间的弱对齐，这影响了开放词汇语义分割等任务；（2）细粒度视觉表示的高标记数量，限制了其在长视频上的可扩展性。本研究同时解决这两个局限。我们提出T-REN（文本对齐区域编码器网络），一种高效编码器，可将视觉数据映射为紧凑的文本对齐区域级表示（即区域标记）。T-REN通过在冻结的视觉骨干网络上添加轻量级网络实现，该网络将每个语义区域内基于图块的表示汇集成区域标记，并与区域级文本标注对齐。与视觉-语言骨干网络相比，仅增加3.7%参数，该设计显著增强了稠密跨模态理解能力，同时将标记数量减少数个数量级。具体而言，T-REN在ADE20K开放词汇分割任务上提升+5.9 mIoU，在COCO物体级文本-图像检索任务上提升+18.4%召回率，在Ego4D视频物体定位任务上提升+15.6%召回率，在VSPW视频场景解析任务上提升+17.6% mIoU，同时与基于图块的视觉-语言骨干网络相比，将图像标记数量减少24倍以上、视频标记数量减少187倍。代码与模型已开源：https://github.com/savya08/T-REN。