Action understanding matters and attracts attention. It can be formed as the mapping from the action physical space to the semantic space. Typically, researchers built action datasets according to idiosyncratic choices to define classes and push the envelope of benchmarks respectively. Thus, datasets are incompatible with each other like "Isolated Islands" due to semantic gaps and various class granularities, e.g., do housework in dataset A and wash plate in dataset B. We argue that a more principled semantic space is an urgent need to concentrate the community efforts and enable us to use all datasets together to pursue generalizable action learning. To this end, we design a Poincare action semantic space given verb taxonomy hierarchy and covering massive actions. By aligning the classes of previous datasets to our semantic space, we gather (image/video/skeleton/MoCap) datasets into a unified database in a unified label system, i.e., bridging "isolated islands" into a "Pangea". Accordingly, we propose a bidirectional mapping model between physical and semantic space to fully use Pangea. In extensive experiments, our system shows significant superiority, especially in transfer learning. Code and data will be made publicly available.

翻译：动作理解至关重要且备受关注。它可被形式化为从动作物理空间到语义空间的映射。通常，研究者根据各自偏好构建动作数据集来定义类别并推动基准性能提升。然而，由于语义鸿沟和不同类别粒度（如数据集A中的做家务与数据集B中的洗盘子），这些数据集因语义差异和类别粒度差异而互不兼容，如同"孤立岛屿"。我们认为，亟需构建更具原则性的语义空间来凝聚学界力量，使我们能够利用所有数据集共同追求可泛化的动作学习。为此，我们基于动词分类层级体系设计了一个涵盖海量动作的庞加莱动作语义空间。通过将以往数据集的类别对齐到我们的语义空间，我们将（图像/视频/骨架/动作捕捉）数据集整合到统一标签体系下的统一数据库中，即把"孤立岛屿"连接成"泛大陆"。据此，我们提出一种物理空间与语义空间之间的双向映射模型以充分利用泛大陆。大量实验表明，我们的系统展现出显著优势，尤其在迁移学习方面。代码与数据将公开发布。