Mixture-of-Experts (MoE) architectures are widely used in modern large language models and multimodal models. However, inference efficiency is often limited by highly dynamic and skewed expert workloads across different modalities. During the prefill stage with large batch sizes, vision tokens frequently dominate the input sequences. Under expert parallelism (EP), this leads to severe load imbalance, where a subset of devices becomes overloaded, reducing overall system throughput. We propose ReaLB, a real-time load balancing method for multimodal MoE (MMoE) inference that introduces zero scheduling overhead. ReaLB dynamically adjusts the computation precision of MoE experts at runtime on a per-EP-rank basis. For ranks dominated by vision-heavy experts, ReaLB assigns lower-precision computation to improve execution efficiency by exploiting FP4 Tensor Cores. ReaLB does not require redundant experts or additional memory allocation. Instead, it performs layer-wise expert precision transformation on the fly and hides the associated overhead within the dispatch phase before MoE computation. Experiments on representative MMoE models show that ReaLB achieves 1.29x layer-level speedup while limiting accuracy loss to within 1.2%.

翻译：混合专家（Mixture-of-Experts，MoE）架构已广泛应用于现代大语言模型和多模态模型。然而，跨模态的动态且极度偏斜的专家工作负载常常制约推理效率。在预填充阶段，由于批次规模较大，视觉token常占据输入序列的主导地位。在专家并行（Expert Parallelism，EP）机制下，这将导致严重的负载不均衡，部分设备因过载而降低系统整体吞吐量。本文提出ReaLB，一种针对多模态MoE（MMoE）推理的实时负载均衡方法，该方法引入零调度开销。ReaLB在运行时以每个EP秩为基础动态调整MoE专家的计算精度。对于视觉密集型专家主导的秩，ReaLB通过利用FP4张量核心分配低精度计算以提升执行效率。ReaLB无需冗余专家或额外内存分配，而是在线执行逐层专家精度转换，并将相关开销隐藏于MoE计算前的调度阶段。在代表性MMoE模型上的实验表明，ReaLB可实现1.29倍的层级加速，同时将精度损失限制在1.2%以内。