Stylized visual captioning aims to generate image or video descriptions with specific styles, making them more attractive and emotionally appropriate. One major challenge with this task is the lack of paired stylized captions for visual content, so most existing works focus on unsupervised methods that do not rely on parallel datasets. However, these approaches still require training with sufficient examples that have style labels, and the generated captions are limited to predefined styles. To address these limitations, we explore the problem of Few-Shot Stylized Visual Captioning, which aims to generate captions in any desired style, using only a few examples as guidance during inference, without requiring further training. We propose a framework called FS-StyleCap for this task, which utilizes a conditional encoder-decoder language model and a visual projection module. Our two-step training scheme proceeds as follows: first, we train a style extractor to generate style representations on an unlabeled text-only corpus. Then, we freeze the extractor and enable our decoder to generate stylized descriptions based on the extracted style vector and projected visual content vectors. During inference, our model can generate desired stylized captions by deriving the style representation from user-supplied examples. Our automatic evaluation results for few-shot sentimental visual captioning outperform state-of-the-art approaches and are comparable to models that are fully trained on labeled style corpora. Human evaluations further confirm our model s ability to handle multiple styles.

翻译：风格化视觉描述旨在生成具有特定风格的图像或视频描述，使其更具吸引力和情感适宜性。该任务的主要挑战在于缺乏视觉内容对应的成对风格化描述，因此现有研究大多聚焦于无需平行数据集的非监督方法。然而，这些方法仍需使用具有风格标签的充足样本进行训练，且生成的描述仅限于预定义风格。为解决这些局限，我们探索了少样本风格化视觉描述问题，旨在仅通过推理阶段提供的少量示例作为引导，无需额外训练即可生成任意期望风格的描述。我们提出名为FS-StyleCap的框架，该框架采用条件式编码器-解码器语言模型与视觉投影模块。我们的两步训练方案如下：首先，在无标签纯文本语料库上训练风格提取器以生成风格表征；随后，冻结该提取器并使得解码器能够基于提取的风格向量与投影后的视觉内容向量生成风格化描述。在推理阶段，模型可通过用户提供的示例推导风格表征，从而生成所需的风格化描述。针对少样本情感化视觉描述任务的自动化评估结果显示，我们的方法不仅超越了现有最优技术，其性能更与在完整风格标签语料上训练的模型相当。人工评估进一步验证了模型处理多种风格的能力。