Decentralized training introduces critical security risks when executed across untrusted, geographically distributed nodes. While existing Byzantine-tolerant literature addresses data parallel (DP) training through robust aggregation methods, pipeline parallelism (PP) presents fundamentally distinct challenges. In PP, model layers are distributed across workers where the activations and their gradients flow between stages rather than being aggregated, making traditional DP approaches inapplicable. We propose SENTINEL, a verification mechanism for PP training without computation duplication. SENTINEL employs lightweight momentum-based monitoring using exponential moving averages (EMAs) to detect corrupted inter-stage communication. Unlike existing Byzantine-tolerant approaches for DP that aggregate parameter gradients across replicas, our approach verifies sequential activation/gradient transmission between layers. We provide theoretical convergence guarantees for this new setting that recovers classical convergence rates when relaxed to standard training. Experiments demonstrate successful training of up to 4B-parameter LLMs across untrusted distributed environments with up to 176 workers while maintaining model convergence and performance.

翻译：去中心化训练在跨不可信、地理分布节点执行时引入了关键的安全风险。现有的拜占庭容忍文献通过鲁棒的聚合方法处理数据并行训练，而流水线并行则带来了根本不同的挑战。在流水线并行中，模型层分布在各个工作节点上，激活及其梯度在阶段之间流动而非被聚合，这使得传统的数据并行方法不再适用。我们提出SENTINEL，一种无需重复计算的流水线并行训练验证机制。SENTINEL采用基于动量的轻量级监控，利用指数移动平均来检测阶段间通信的损坏。与现有针对数据并行的拜占庭容忍方法（跨副本聚合参数梯度）不同，我们的方法验证层间的顺序激活/梯度传输。我们为这一新设置提供了理论收敛保证，在放宽至标准训练时可恢复经典收敛速率。实验表明，在不可信的分布式环境中，使用多达176个工作节点成功训练了高达40亿参数的大语言模型，同时保持了模型的收敛性和性能。