We introduce Nemotron 3 Ultra, a 550 billion total and 55 billion active parameter Mixture-of-Experts Hybrid Mamba-Attention language model. We pre-trained Nemotron 3 Ultra on 20 trillion text tokens, then extended the context length to 1M tokens, and post-trained using Supervised Fine Tuning (SFT), Reinforcement Learning (RL), and Multi-teacher On-Policy Distillation (MOPD). Nemotron 3 Ultra is our most capable model yet, employing multiple key technologies - LatentMoE, Multi Token Prediction (MTP), NVFP4 pre-training, multi-environment RLVR, MOPD, and reasoning budget control. Nemotron 3 Ultra achieves up to ~6x higher inference throughput as compared to state-of-the-art publicly available LLMs while attaining on-par accuracy. The state-of-the-art accuracy, high inference throughput, and 1M token context length make Nemotron 3 Ultra ideal for long-running autonomous agentic tasks. We open-source the base, post-trained, and quantized checkpoints, along with the training data and recipe on HuggingFace.

翻译：我们提出Nemotron 3 Ultra，一种总参数量为5500亿、激活参数量为550亿的混合专家（Mixture-of-Experts）Mamba-注意力机制语言模型。我们在20万亿文本令牌上预训练Nemotron 3 Ultra，随后将上下文长度扩展至100万令牌，并通过监督微调（SFT）、强化学习（RL）和多教师同策略蒸馏（MOPD）进行后训练。Nemotron 3 Ultra是我们能力最强的模型，融合了多项核心技术——LatentMoE、多令牌预测（MTP）、NVFP4预训练、多环境RLVR、MOPD以及推理预算控制。与当前最先进的开源大语言模型相比，Nemotron 3 Ultra在达到同等准确率的同时，推理吞吐量提升高达约6倍。其顶尖的准确率、高推理吞吐量以及100万令牌的上下文长度，使其成为长期自主智能任务的理想选择。我们在HuggingFace上开源了基础模型、后训练模型和量化检查点，以及训练数据和配方。