Controllable image captioning is an emerging multimodal topic that aims to describe the image with natural language following human purpose, $\textit{e.g.}$, looking at the specified regions or telling in a particular text style. State-of-the-art methods are trained on annotated pairs of input controls and output captions. However, the scarcity of such well-annotated multimodal data largely limits their usability and scalability for interactive AI systems. Leveraging unimodal instruction-following foundation models is a promising alternative that benefits from broader sources of data. In this paper, we present Caption AnyThing (CAT), a foundation model augmented image captioning framework supporting a wide range of multimodel controls: 1) visual controls, including points, boxes, and trajectories; 2) language controls, such as sentiment, length, language, and factuality. Powered by Segment Anything Model (SAM) and ChatGPT, we unify the visual and language prompts into a modularized framework, enabling the flexible combination between different controls. Extensive case studies demonstrate the user intention alignment capabilities of our framework, shedding light on effective user interaction modeling in vision-language applications. Our code is publicly available at https://github.com/ttengwang/Caption-Anything.

翻译：摘要：可控图像描述是一项新兴的多模态课题，旨在遵循人类意图（例如关注指定区域或采用特定文本风格）用自然语言描述图像。现有最先进方法依赖输入控制与输出描述的标注配对数据进行训练。然而，此类高标注质量的多模态数据稀缺性极大限制了其在交互式AI系统中的实用性与可扩展性。利用面向单一模态指令遵循的基石模型是一种具有前景的替代方案，可受益于更广泛的数据源。本文提出CAPTION ANYTHING（CAT）——一种增强型基石模型图像描述框架，支持多种多模态控制：1）视觉控制（包括点、框和轨迹）；2）语言控制（如情感倾向、长度、语言及事实性）。依托Segment Anything Model（SAM）与ChatGPT，我们将视觉与语言提示统一至模块化框架中，实现不同控制间的灵活组合。大量案例研究表明，我们的框架具备与用户意图对齐的能力，为视觉-语言应用中的有效用户交互建模提供了新思路。代码已开源至https://github.com/ttengwang/Caption-Anything。