Contrastive Language-Image Pre-training (CLIP) plays an essential role in extracting valuable content information from images across diverse tasks. It aligns textual and visual modalities to comprehend the entire image, including all the details, even those irrelevant to specific tasks. However, for a finer understanding and controlled editing of images, it becomes crucial to focus on specific regions of interest, which can be indicated as points, masks, or boxes by humans or perception models. To fulfill the requirements, we introduce Alpha-CLIP, an enhanced version of CLIP with an auxiliary alpha channel to suggest attentive regions and fine-tuned with constructed millions of RGBA region-text pairs. Alpha-CLIP not only preserves the visual recognition ability of CLIP but also enables precise control over the emphasis of image contents. It demonstrates effectiveness in various tasks, including but not limited to open-world recognition, multimodal large language models, and conditional 2D / 3D generation. It has a strong potential to serve as a versatile tool for image-related tasks.

翻译：对比语言-图像预训练（CLIP）在各类任务中发挥着从图像中提取核心内容信息的关键作用。它通过对齐文本与视觉模态，实现对整幅图像的全面理解，包含所有细节，甚至包括那些与特定任务无关的部分。然而，为了实现更精细的图像理解与受控编辑，聚焦特定感兴趣区域变得至关重要——这些区域可由人类或感知模型通过点、掩膜或边界框指定。为满足这一需求，我们提出Alpha-CLIP，这是CLIP的增强版本：它引入辅助Alpha通道以指示注意力区域，并通过构建的百万级RGBA区域-文本对进行微调。Alpha-CLIP不仅保留了CLIP的视觉识别能力，还能精准控制图像内容的关注重点。在开放世界识别、多模态大语言模型及条件2D/3D生成等多种任务中，它均展现出卓越效能，并有望成为图像相关任务的通用工具。