Learning to infer labels in an open world, i.e., in an environment where the target ``labels'' are unknown, is an important characteristic for achieving autonomy. Foundation models pre-trained on enormous amounts of data have shown remarkable generalization skills through prompting, particularly in zero-shot inference. However, their performance is restricted to the correctness of the target label's search space. In an open world where these labels are unknown, the search space can be exceptionally large. It can require reasoning over several combinations of elementary concepts to arrive at an inference, which severely restricts the performance of such models. To tackle this challenging problem, we propose a neuro-symbolic framework called ALGO - novel Action Learning with Grounded Object recognition that can use symbolic knowledge stored in large-scale knowledge bases to infer activities (verb-noun combinations) in egocentric videos with limited supervision using two steps. First, we propose a novel neuro-symbolic prompting approach that uses object-centric vision-language foundation models as a noisy oracle to ground objects in the video through evidence-based reasoning. Second, driven by prior commonsense knowledge, we discover plausible activities through an energy-based symbolic pattern theory framework and learn to ground knowledge-based action (verb) concepts in the video. Extensive experiments on two publicly available datasets (GTEA Gaze and GTEA Gaze Plus) demonstrate its performance on open-world activity inference and its generalization to unseen actions in an unknown search space. We show that ALGO can be extended to zero-shot settings and demonstrate its competitive performance to multimodal foundation models.

翻译：在开放世界（即目标“标签”未知的环境）中学习推断标签是实现自主性的重要特性。基于海量数据预训练的基座模型通过提示学习展现出显著的泛化能力，尤其在零样本推断中表现突出。然而，其性能受限于目标标签搜索空间的正确性。在标签未知的开放世界中，搜索空间可能异常庞大，需要推理多个基本概念的组合才能得出推断结论，这严重限制了此类模型的性能。为解决这一难题，我们提出名为ALGO（基于锚定对象识别的新型行动学习）的神经符号框架，该框架可通过两步利用大规模知识库中存储的符号知识，在有限监督下推断第一人称视频中的活动（动词-名词组合）。首先，我们提出一种新型神经符号提示方法，将面向对象的视觉语言基座模型作为含噪预言机，通过基于证据的推理在视频中锚定对象。其次，在先验常识知识驱动下，我们通过基于能量的符号模式理论框架发现可行活动，并学习将基于知识的动作（动词）概念锚定到视频中。在两个公开数据集（GTEA Gaze和GTEA Gaze Plus）上的大量实验证明了该方法在开放世界活动推断及未知搜索空间中未见动作泛化方面的性能。我们展示ALGO可扩展至零样本设置，并证明其与多模态基座模型相比具有竞争力的性能。