Large-scale international scientific collaborations, such as ATLAS, Belle II, CMS, and DUNE, generate vast volumes of data. These experiments necessitate substantial computational power for varied tasks, including structured data processing, Monte Carlo simulations, and end-user analysis. Centralized workflow and data management systems are employed to handle these demands, but current decision-making processes for data placement and payload allocation are often heuristic and disjointed. This optimization challenge potentially could be addressed using contemporary machine learning methods, such as reinforcement learning, which, in turn, require access to extensive data and an interactive environment. Instead, we propose a generative surrogate modeling approach to address the lack of training data and concerns about privacy preservation. We have collected and processed real-world job submission records, totaling more than two million jobs through 150 days, and applied four generative models for tabular data -- TVAE, CTAGGAN+, SMOTE, and TabDDPM -- to these datasets, thoroughly evaluating their performance. Along with measuring the discrepancy among feature-wise distributions separately, we also evaluate pair-wise feature correlations, distance to closest record, and responses to pre-trained models. Our experiments indicate that SMOTE and TabDDPM can generate similar tabular data, almost indistinguishable from the ground truth. Yet, as a non-learning method, SMOTE ranks the lowest in privacy preservation. As a result, we conclude that the probabilistic-diffusion-model-based TabDDPM is the most suitable generative model for managing job record data.

翻译：大型国际科学合作项目，如ATLAS、Belle II、CMS和DUNE，会产生海量数据。这些实验需要强大的计算能力来执行各种任务，包括结构化数据处理、蒙特卡洛模拟和终端用户分析。虽然采用集中式工作流和数据管理系统来应对这些需求，但当前数据放置和负载分配的决策过程通常基于启发式方法且相互脱节。这一优化问题本可采用强化学习等现代机器学习方法解决，但这些方法本身需要大量数据和交互环境。为此，我们提出一种生成式代理建模方法，以应对训练数据缺乏和隐私保护问题。我们收集并处理了真实世界的作业提交记录（150天内总计超过200万个作业），并对这些数据集应用了四种表格数据生成模型——TVAE、CTAGGAN+、SMOTE和TabDDPM，对其性能进行了全面评估。除了分别测量特征分布间的差异外，我们还评估了特征间的配对相关性、与最近记录的距离以及对预训练模型的响应。实验表明，SMOTE和TabDDPM能生成与原始数据几乎无法区分的相似表格数据。然而，作为非学习方法，SMOTE在隐私保护方面表现最差。因此，我们得出结论：基于概率扩散模型的TabDDPM是最适合管理作业记录数据的生成模型。