In LLM inference, the same prompt may yield different outputs across different runs. At the system level, this non-determinism arises from floating-point non-associativity combined with dynamic batching and GPU kernels whose reduction orders vary with batch size. A straightforward way to eliminate non-determinism is to disable dynamic batching during inference, but doing so severely degrades throughput. Another approach is to make kernels batch-invariant; however, this tightly couples determinism to kernel design, requiring new implementations. This coupling also imposes fixed runtime overheads, regardless of how much of the workload actually requires determinism. Inspired by ideas from speculative decoding, we present LLM-42, a scheduling-based approach to enable determinism in LLM inference. Our key observation is that if a sequence is in a consistent state, the next emitted token is likely to be consistent even with dynamic batching. Moreover, most GPU kernels use shape-consistent reductions. Leveraging these insights, LLM-42 decodes tokens using a non-deterministic fast path and enforces determinism via a lightweight verify-rollback loop. The verifier replays candidate tokens under a fixed-shape reduction schedule, commits those that are guaranteed to be consistent across runs, and rolls back those violating determinism. LLM-42 mostly re-uses existing kernels unchanged and incurs overhead only in proportion to the traffic that requires determinism.

翻译：在LLM推理中，相同的提示词在不同次运行中可能产生不同的输出。在系统层面，这种非确定性源于浮点运算的非结合性与动态批处理以及GPU核函数的结合，其中归约顺序随批处理大小而变化。消除非确定性的一种直接方法是在推理过程中禁用动态批处理，但这样做会严重降低吞吐量。另一种方法是使核函数具有批处理不变性；然而，这会将确定性与核函数设计紧密耦合，需要新的实现。这种耦合还会带来固定的运行时开销，而不管实际需要确定性的工作负载有多少。受推测解码思想的启发，我们提出了LLM-42，一种基于调度的方法，用于在LLM推理中实现确定性。我们的关键观察是，如果一个序列处于一致状态，即使采用动态批处理，下一个生成的token也很可能保持一致。此外，大多数GPU核函数使用形状一致的归约操作。利用这些见解，LLM-42使用非确定性的快速路径解码token，并通过一个轻量级的验证-回滚循环来强制执行确定性。验证器在固定形状的归约调度下重放候选token，提交那些保证在多次运行中一致的token，并回滚那些违反确定性的token。LLM-42主要复用现有核函数而无需修改，并且仅按需要确定性的流量比例产生开销。