Today's LLM (pre-) training and research workflows typically allocate a significant amount of compute to large-scale ablation studies. Despite the substantial compute costs of these ablations, existing open-source frameworks provide limited tooling for these experiments, often forcing researchers to write their own wrappers and scripts. We propose Modalities, an end-to-end PyTorch-native framework that integrates data-driven LLM research with large-scale model training from two angles. Firstly, by integrating state-of-the-art parallelization strategies, it enables both efficient pretraining and systematic ablations at trillion-token and billion-parameter scale. Secondly, Modalities adopts modular design with declarative, self-contained configuration, enabling reproducibility and extensibility levels that are difficult to achieve out-of-the-box with existing LLM training frameworks.

翻译：当前LLM（预）训练与研究流程通常将大量算力分配至大规模消融实验。尽管这些消融实验的计算成本高昂，现有开源框架为此类实验提供的工具支持有限，往往迫使研究人员自行编写封装脚本。我们提出Modalities——一个端到端的PyTorch原生框架，该框架从两个维度将数据驱动的LLM研究与大规模模型训练相结合。首先，通过集成前沿并行化策略，它能够在万亿token与十亿参数规模上同时实现高效预训练与系统性消融实验。其次，Modalities采用模块化设计配合声明式自包含配置，其可复现性与可扩展性水平是现有LLM训练框架难以开箱即达的。