Mixture-of-Experts (MoE) architectures are widely used in modern large language models and multimodal models. However, inference efficiency is often limited by highly dynamic and skewed expert workloads across different modalities. During the prefill stage with large batch sizes, vision tokens frequently dominate the input sequences. Under expert parallelism (EP), this leads to severe load imbalance, where a subset of devices becomes overloaded, reducing overall system throughput. We propose ReaLB, a real-time load balancing method for multimodal MoE (MMoE) inference that introduces zero scheduling overhead. ReaLB dynamically adjusts the computation precision of MoE experts at runtime on a per-EP-rank basis. For ranks dominated by vision-heavy experts, ReaLB assigns lower-precision computation to improve execution efficiency by exploiting FP4 Tensor Cores. ReaLB does not require redundant experts or additional memory allocation. Instead, it performs layer-wise expert precision transformation on the fly and hides the associated overhead within the dispatch phase before MoE computation. Experiments on representative MMoE models show that ReaLB achieves 1.29x layer-level speedup while limiting accuracy loss to within 1.2%.

翻译：混合专家（MoE）架构被广泛应用于现代大型语言模型和多模态模型。然而，不同模态间高度动态且偏斜的专家工作负载常常限制推理效率。在具有大批量大小的预填充阶段，视觉标记经常主导输入序列。在专家并行（EP）下，这会导致严重的负载不均衡，其中部分设备过载，从而降低整体系统吞吐量。我们提出ReaLB，一种面向多模态MoE（MMoE）推理的实时负载均衡方法，该方法引入零调度开销。ReaLB在运行时逐EP秩动态调整MoE专家的计算精度。对于视觉密集型专家主导的秩，ReaLB分配较低精度计算，通过利用FP4张量核心来提升执行效率。ReaLB无需冗余专家或额外内存分配。相反，它在MoE计算前的调度阶段内，即时执行逐层专家精度变换，并隐藏相关开销。在代表性MMoE模型上的实验表明，ReaLB实现了1.29倍的层级别加速，同时将精度损失限制在1.2%以内。