Recent advancements in text-only large language models (LLMs) have highlighted the benefit of in-context learning for adapting to new tasks with a few demonstrations. However, extending in-context learning to large vision-language models (VLMs) using a huge amount of naturalistic vision-language data has shown limited success, particularly for egocentric videos, due to high data collection costs. We propose a novel training method $\mathbb{E}$fficient $\mathbb{I}$n-context $\mathbb{L}$earning on $\mathbb{E}$gocentric $\mathbb{V}$ideos ($\mathbb{EILEV}$), which elicits in-context learning in VLMs for egocentric videos without requiring massive, naturalistic egocentric video datasets. $\mathbb{EILEV}$ involves architectural and training data adaptations to allow the model to process contexts interleaved with video clips and narrations, sampling of in-context examples with clusters of similar verbs and nouns, use of data with skewed marginal distributions with a long tail of infrequent verbs and nouns, as well as homonyms and synonyms. Our evaluations show that $\mathbb{EILEV}$-trained models outperform larger VLMs trained on a huge amount of naturalistic data in in-context learning. Furthermore, they can generalize to not only out-of-distribution, but also novel, rare egocentric videos and texts via in-context learning, demonstrating potential for applications requiring cost-effective training, and rapid post-deployment adaptability. Our code and demo are available at \url{https://github.com/yukw777/EILEV}.

翻译：近期文本大语言模型（LLM）的进展凸显了利用少量示例进行上下文学习以适应新任务的优势。然而，将上下文学习扩展到基于大规模自然视觉语言数据训练的大视觉语言模型（VLM）时，尤其是在第一人称视频领域，由于数据采集成本高昂，其成效有限。我们提出一种新颖的训练方法$\mathbb{E}$fficient $\mathbb{I}$n-context $\mathbb{L}$earning on $\mathbb{E}$gocentric $\mathbb{V}$ideos ($\mathbb{EILEV}$)，无需大规模自然第一人称视频数据集即可激发VLM在第一人称视频中的上下文学习能力。$\mathbb{EILEV}$通过架构与训练数据适应性调整，使模型能够处理视频片段与叙述交错构成的上下文，利用相似动词与名词聚类采样上下文示例，采用具有长尾分布（包含低频动词、名词及同音异义词、同义词）的偏态边际分布数据。评估表明，经$\mathbb{EILEV}$训练的模型在上下文学习任务中优于基于海量自然数据训练的大型VLM。此外，该类模型不仅能泛化至分布外场景，还可通过上下文学习适应新型、罕见的第一人称视频及文本，展现了其在低训练成本与快速部署后适应性需求的场景中的应用潜力。我们的代码与演示详见\url{https://github.com/yukw777/EILEV}。