We present GLEE in this work, an object-level foundation model for locating and identifying objects in images and videos. Through a unified framework, GLEE accomplishes detection, segmentation, tracking, grounding, and identification of arbitrary objects in the open world scenario for various object perception tasks. Adopting a cohesive learning strategy, GLEE acquires knowledge from diverse data sources with varying supervision levels to formulate general object representations, excelling in zero-shot transfer to new data and tasks. Specifically, we employ an image encoder, text encoder, and visual prompter to handle multi-modal inputs, enabling to simultaneously solve various object-centric downstream tasks while maintaining state-of-the-art performance. Demonstrated through extensive training on over five million images from diverse benchmarks, GLEE exhibits remarkable versatility and improved generalization performance, efficiently tackling downstream tasks without the need for task-specific adaptation. By integrating large volumes of automatically labeled data, we further enhance its zero-shot generalization capabilities. Additionally, GLEE is capable of being integrated into Large Language Models, serving as a foundational model to provide universal object-level information for multi-modal tasks. We hope that the versatility and universality of our method will mark a significant step in the development of efficient visual foundation models for AGI systems. The model and code will be released at https://glee-vision.github.io .

翻译：本文提出GLEE，一个面向图像与视频中对象定位与识别的对象级基础模型。通过统一框架，GLEE能够完成开放世界场景中各类对象感知任务的检测、分割、跟踪、定位与识别。采用协同学习策略，GLEE从不同监督级别的多源数据中学习通用对象表征，在零样本迁移至新数据与任务时表现卓越。具体而言，我们采用图像编码器、文本编码器与视觉提示器处理多模态输入，可同时解决多种以对象为中心的下游任务，同时保持最先进性能。通过在超500万张来自不同基准的图像上进行大规模训练，GLEE展现出显著的多功能性与泛化能力，无需任务特定适配即可高效处理下游任务。通过整合大规模自动标注数据，我们进一步增强了其零样本泛化能力。此外，GLEE可集成至大型语言模型，作为基础模型为多模态任务提供通用对象级信息。我们期望本方法的通用性与普适性将为AGI系统中高效视觉基础模型的发展迈出重要一步。模型与代码将开源至https://glee-vision.github.io 。