Recent research has shown that aligning fine-grained text descriptions with localized image patches can significantly improve the zero-shot performance of pre-trained vision-language models (e.g., CLIP). However, we find that both fine-grained text descriptions and localized image patches often contain redundant information, making text-visual alignment less effective. In this paper, we tackle this issue from two perspectives: \emph{View Refinement} and \emph{Description refinement}, termed as \textit{\textbf{Bi}-refinement for \textbf{F}ine-grained \textbf{T}ext-visual \textbf{A}lignment} (BiFTA). \emph{View refinement} removes redundant image patches with high \emph{Intersection over Union} (IoU) ratios, resulting in more distinctive visual samples. \emph{Description refinement} removes redundant text descriptions with high pairwise cosine similarity, ensuring greater diversity in the remaining descriptions. BiFTA achieves superior zero-shot performance on 6 benchmark datasets for both ViT-based and ResNet-based CLIP, justifying the necessity to remove redundant information in visual-text alignment.

翻译：近期研究表明，将细粒度文本描述与局部图像块对齐可显著提升预训练视觉语言模型（如CLIP）的零样本性能。然而，我们发现细粒度文本描述与局部图像块常包含冗余信息，导致图文对齐效果受限。本文从双重视角解决该问题：\emph{视图细化}与\emph{描述细化}，统称为\textit{\textbf{细粒度图文对齐双视角细化方法}}（BiFTA）。\emph{视图细化}通过剔除具有高\emph{交并比}（IoU）的冗余图像块，生成更具区分度的视觉样本；\emph{描述细化}通过移除具有高成对余弦相似度的冗余文本描述，确保剩余描述的更高多样性。BiFTA在6个基准数据集上为基于ViT与ResNet的CLIP模型均实现了卓越的零样本性能，印证了在视觉-文本对齐中消除冗余信息的必要性。