We propose masked particle modeling (MPM) as a self-supervised method for learning generic, transferable, and reusable representations on unordered sets of inputs for use in high energy physics (HEP) scientific data. This work provides a novel scheme to perform masked modeling based pre-training to learn permutation invariant functions on sets. More generally, this work provides a step towards building large foundation models for HEP that can be generically pre-trained with self-supervised learning and later fine-tuned for a variety of down-stream tasks. In MPM, particles in a set are masked and the training objective is to recover their identity, as defined by a discretized token representation of a pre-trained vector quantized variational autoencoder. We study the efficacy of the method in samples of high energy jets at collider physics experiments, including studies on the impact of discretization, permutation invariance, and ordering. We also study the fine-tuning capability of the model, showing that it can be adapted to tasks such as supervised and weakly supervised jet classification, and that the model can transfer efficiently with small fine-tuning data sets to new classes and new data domains.

翻译：我们提出掩蔽粒子建模（MPM），作为一种自监督方法，用于学习无序输入集合上的通用、可迁移、可复用的表示，以应用于高能物理（HEP）科学数据。本工作提出了一种基于掩蔽建模的预训练新方案，以学习集合上的置换不变函数。更一般地，本工作为构建针对HEP的大型基础模型迈出了一步，这类模型可通过自监督学习进行通用预训练，随后针对多种下游任务进行微调。在MPM中，集合中的粒子被掩蔽，训练目标是恢复其身份，该身份由预训练向量量化变分自编码器的离散化令牌表示定义。我们研究了该方法在粒子对撞机物理实验的高能喷注样本中的有效性，包括离散化、置换不变性和排序的影响研究。我们还研究了模型的微调能力，表明其可适用于有监督和弱监督的喷注重分类等任务，并且该模型能以少量微调数据集高效迁移至新类别和新数据领域。