Modern vision language pipelines are driven by RGB vision encoders trained on massive image text corpora. While these pipelines have enabled impressive zero-shot capabilities and strong transfer across tasks, they still inherit two structural inefficiencies from the pixel domain: (i) transmitting dense RGB images from edge devices to the cloud is energy-intensive and costly, and (ii) patch-based tokenization explodes sequence length, stressing attention budgets and context limits. We explore 2D Gaussian Splatting (2DGS) as an alternative visual substrate for alignment: a compact, spatially adaptive representation that parameterizes images by a set of colored anisotropic Gaussians. We develop a scalable 2DGS pipeline with structured initialization, luminance-aware pruning, and batched CUDA kernels, achieving over 90x faster fitting and about 97% GPU utilization compared to prior implementations. We further adapt contrastive language-image pre-training (CLIP) to 2DGS by reusing a frozen RGB-based transformer backbone with a lightweight splat-aware input stem and a perceiver resampler, training only 9.7% to 13.8% of the total parameters. On a 12.8M dataset from DataComp, GS encoders yield competitive zero-shot performance on 38 datasets from the CLIP benchmark while compressing inputs 3x to 23.5x relative to pixels. Our results establish 2DGS as a viable multimodal substrate, pinpoint architectural bottlenecks, and open a path toward representations that are both semantically powerful and transmission-efficient for edge-cloud learning.

翻译：现代视觉语言处理流程依赖于在大量图文语料上训练的RGB视觉编码器。尽管此类流程已实现出色的零样本能力及跨任务的强大迁移性能，但其仍继承了像素域的两个结构性低效问题：(i) 将稠密RGB图像从边缘设备传输至云端能耗高、成本大；(ii) 基于图像块的标记化处理导致序列长度激增，给注意力计算与上下文容量带来压力。本研究探索将二维高斯泼溅（2DGS）作为替代性视觉对齐基底：这是一种通过一组彩色各向异性高斯函数参数化图像的紧凑型空间自适应表征。我们开发了具有结构化初始化、亮度感知剪枝和批量化CUDA内核的可扩展2DGS流程，相比现有实现实现了超过90倍的拟合加速和约97%的GPU利用率。通过复用冻结的RGB基Transformer主干网络，并搭配轻量级泼溅感知输入模块与感知重采样器，我们将对比语言-图像预训练（CLIP）适配至2DGS表征，仅需训练总参数量的9.7%至13.8%。在DataComp的1280万数据集上，高斯泼溅编码器在CLIP基准测试的38个数据集上展现出具有竞争力的零样本性能，同时实现相对于像素3倍至23.5倍的输入压缩。我们的研究成果确立了2DGS作为可行多模态基底的潜力，揭示了架构瓶颈，并为开发兼具语义表达能力与边缘-云端传输效率的表征体系开辟了新路径。