Modern Large Foundation Model (LFM) training has transformed the data pipeline from a static ingestion layer into a dynamic component that must co-evolve with the training process. Existing systems are ill-equipped: colocated dataloaders offer no failure isolation, while message queue-based disaggregated dataloaders operate on a record/offset abstraction that cannot express the batch-level semantics required by distributed training. We present BatchWeave, an object-store-native training data plane for distributed LFM training. BatchWeave uses versioned manifests and conditional object writes to coordinate batch publication, recovery, and lifecycle management. First, it introduces the Transactional Global Batch (TGB), which builds on versioned-manifest ACID storage semantics and extends them with training-specific consistency, including atomic all-rank batch visibility, a globally ordered step sequence, checkpoint-aligned lifecycle management, and end-to-end exactly-once recovery. Second, it realizes recovery and retention directly in the storage layer, by durably persisting producer state through the commit protocol and tying reclamation to distributed checkpoint state. Third, its Decentralized Adaptive Commit (DAC) algorithm sustains stable ingestion throughput as the manifest grows, without any inter-producer communication. Evaluations on large-scale multimodal pre-training and SFT workloads using 64 GPUs show that BatchWeave outperforms colocated dataloader throughput while providing full failure isolation, outperforms Apache Kafka in ingestion throughput, and achieves lower consumer read latency than Kafka.

翻译：现代大规模基础模型（LFM）训练已将数据流水线从静态数据摄取层转变为需与训练过程协同演进的动态组件。现有系统难以胜任：共置数据加载器缺乏故障隔离能力，而基于消息队列的分离式数据加载器采用记录/偏移量抽象，无法表达分布式训练所需的批次级语义。本文提出BatchWeave——一种面向分布式LFM训练的对象存储原生数据平面。BatchWeave利用版本化清单与条件对象写入协调批次发布、恢复与生命周期管理。首先，它提出事务性全局批次（TGB），构建于版本化清单ACID存储语义之上，并扩展了训练特有的一致性机制，包括原子级全秩批次可见性、全局有序步进序列、检查点对齐生命周期管理以及端到端精确一次恢复。其次，通过提交协议持久化生产者状态并将回收机制关联至分布式检查点状态，在存储层直接实现恢复与保留。第三，其去中心化自适应提交（DAC）算法确保随清单增长维持稳定摄取吞吐量，且无需生产者间通信。基于64 GPU的大规模多模态预训练与SFT工作负载评估表明，BatchWeave在提供完全故障隔离的同时，吞吐量优于共置数据加载器；其摄取吞吐量优于Apache Kafka，且消费者读取延迟低于Kafka。