NVIDIA Develops Nemotron 3.5 Lightning NVFP4 Using Quantization-Aware Distillation

NVIDIA Develops Nemotron 3.5 Lightning NVFP4 Using Quantization-Aware Distillation NVIDIA has detailed the development of Nemotron 3.5 Lightning NVFP4, compressing the model from a 66 GB full-precision checkpoint down to roughly 21 to 22 GB. By pairing post-training quantization with quantization-aware distillation, the approach achieves up to 4x faster throughput while recovering accuracy loss across reasoning and agentic evaluations. Compresses the full-precision BF16 base model from ~66 GB to a 21.19 GB W4A16 NVFP4 checkpoint, delivering up to 4x faster throughput. Uses a two-stage Quantization-Aware Distillation (QAD) workflow where a frozen BF16 teacher model guides a quantized student via KL divergence loss. Categorizes QAD into dynamic scale updates for max-calibrated PTQ recipes and frozen scale updates for search-intensive MSE-based recipes. Significantly recovers benchmark accuracy on aggressively quantized models and outperforms standard PTQ on agentic and coding evaluations in the final checkpoint. Integrates the entire quantization and distillation pipeline into NVIDIA Model Optimizer and NeMo Framework's Megatron-Bridge. Two-Stage Quantization-Aware Distillation Post-training quantization (PTQ) alone commonly targets 99% median accuracy recovery, but aggressive quantization under tighter memory constraints can cause performance degradation. To address this, NVIDIA implemented Quantization-Aware Distillation (QAD) for Nemotron 3.5 Lightning. In Stage 1, a PTQ pass quantizes the weights of the full-precision BF16 model (NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16) to W4A16 to initialize the student model. In Stage 2, the student model processes tokens through simulated quantization while matching the logits of the frozen BF16 teacher model via KL divergence loss. Both models share the same footprint of 21.19 GB for W4A16, ensuring performance differences stem entirely from the training method rather than increased memory allocation. Calibration Recipes and Scale Management The PTQ configuration dictates how scales are handled during distillation. Max-calibrated configurations update weight and activation scales dynamically on the fly as the model trains. Conversely, MSE-based calibrations—such as the static MSE, four_over_six, and four_over_six with NVFP4 KV cache—determine scales through an error-minimization search that is too computationally intensive to execute at every step. These feed frozen-scale QAD, where scales remain locked to their PTQ-calibrated values while only the weights receive gradient updates. Calibration sequence lengths were tested up to 128k, with 32K providing optimal PTQ results on the four_over_six recipe. Furthermore, quantizing Mamba linear layers to W4A16 rather than FP8 yielded higher throughput without triggering severe accuracy drops. Ablations and Benchmark Recovery Evaluation on early, aggressively quantized checkpoints demonstrated substantial recovery from QAD. On pre-release Checkpoint A, PTQ dropped median accuracy recovery to 96.33% with notable regressions in AIME 2025, SciCode Subtask, and HLE; 200 iterations of QAD restored median recovery to 99.72%, beating or matching BF16 baselines across 10 of 11 benchmarks. On Checkpoint B, QAD elevated recovery from 95.84% to 98.53%, notably improving the AA v4.1 Index by 3.45 points. For the final release checkpoint, a more conservative quantization recipe was chosen to preserve baseline accuracy, leaving PTQ already close to BF16 (99.24% median recovery versus 98.97% for QAD). However, QAD still outperformed PTQ on key agentic and coding benchmarks, including a 3.79-point increase on Terminal-Bench v2.1 and a 1.07-point gain on SWE-Bench Multilingual. Distributed Training and Tooling Integration Long-context capability required post-training supervised fine-tuning sequence lengths of approximately 522K tokens, scaled up after initial 256K sequence length ablations. Distillation ran at a constant learning rate of 5e-6 without warmup, with dropout disabled and gradient clipping set to 1.0. NVIDIA packaged the full pipeline into the NVIDIA Model Optimizer launcher file megatron_lm_qad.yaml, which interfaces with Megatron-LM. The process is also implemented in Megatron-Bridge through mbridge_qad.yaml, handling data tokenization, student quantization, distillation against the frozen teacher, and exporting to Hugging Face across multi-node, multi-GPU configurations using tensor, pipeline, context, and expert parallelism.

NVIDIA 利用 Model Optimizer 与 QAD 技术开发 Nemotron 3.5 Lightning NVFP4 模型 NVIDIA 介绍了通过模型优化器（Model Optimizer）结合量化感知蒸馏（QAD）开发 Nemotron 3.5 Lightning NVFP4 的全流程。该方案将 66 GB 的 BF16 全精度基座压缩至约 21–22 GB，并带来最高达 4 倍的吞吐量提升，同时借助蒸馏技术恢复了激进量化造成的精度损失。 Nemotron 3.5 Lightning NVFP4 通过将大部分权重压缩至 4 比特，将模型体积从 66 GB 缩减至约 21.19–22 GB，并实现最高 4 倍吞吐提升。 QAD 以冻结的 BF16 模型为教师模型，通过 KL 散度损失对 PTQ 生成的 W4A16 学生模型进行蒸馏训练，有效弥补了激进量化的精度回退。 在激进量化的中间检查点上，QAD 将中位数精度恢复率从 PTQ 的 96.33% 提升至 99.72%，并在智能体（Agent）及编程评测中显著优于 PTQ。 方案支持动态缩放与冻结缩放两种模式，分别对应 Max 校准与 MSE/four_over_six 校准策略。 完整流程已整合至 NVIDIA Model Optimizer 中，并支持通过 Megatron-LM 或 NeMo Framework 的 Megatron-Bridge 进行端到端量化、蒸馏与导出。 量化配方与 QAD 核心机制 针对 Nemotron 3.5 Lightning，团队探索了包括 max、mamba_fp8_max、MSE 以及 four_over_six 等五种 PTQ 配方，其中将 Mamba 线性层设为 W4A16 配合 four_over_six 校准提供了最佳的精度与性能折中。QAD 流程分为两步：第一阶段利用 PTQ 将 BF16 模型量化为 W4A16 学生模型；第二阶段将学生模型的模拟量化前向传播输出与冻结教师模型的 logits 进行 KL 散度对齐。PTQ 阶段支持动态缩放（基于 Max 校准，每步重新计算缩放系数）和冻结缩放（基于 MSE 搜索，在蒸馏期间锁定标定缩放值，仅更新权重）。 长上下文消融与实验设置 为保持长上下文能力，消融实验表明需要达到 522K 的序列长度。实验初期在 256K 序列长度下进行快速迭代，最终 QAD 运行扩展至 522K/524K。蒸馏训练在恒定学习率 5e-6、禁用 dropout、梯度裁剪 1.0 的设置下进行，并推荐基于开源的 Nemotron-Post-Training 数据集复现。 精度恢复表现与评测对比 在早期激进量化的 SFT 检查点 A 上，PTQ 的中位数精度恢复率为 96.33%，在 AIME 2025、SciCode 等测试中下降明显；经过 200 次迭代的 QAD 后，恢复率回升至 99.72%，11 项基准中有 10 项获得改善。在 RL 检查点 B 上，QAD 也将恢复率从 95.84% 提升至 98.53%。最终发布的模型采用了相对保守的量化配方，PTQ 恢复率已达 99.24%，但 QAD 仍在 Terminal-Bench v2.1（提升 3.79 分）、SWE-Bench 多语言版（提升 1.07 分）及 HLE 等智能体与编程任务中表现出明显优势。 工程实现与端到端运行 该 QAD 配方已作为单一启动文件 megatron_lm_qad.yaml 整合入 NVIDIA Model Optimizer，底层调用 Megatron-LM 的 finetune.sh 脚本。此外，流程也支持通过 NeMo 框架的 Megatron-Bridge 运行（mbridge_qad.yaml），覆盖数据分词、PTQ 量化、蒸馏以及 Hugging Face 权重导出的完整管道，可在 2 节点 16 卡或 8 节点 32 卡等多 GPU 环境下并行执行。

Teams customize their models to hit their targets for latency, speed, memory, and compute. With the open NVIDIA Nemotron family of models, developers can find the right-sized model for their needs. The new Nemotron 3.5 Lightning NVFP4 checkpoint, for example, preserves accuracy while unlocking up to 4x faster throughput. It’s compressed down to 22 GB from the 66 GB full precision checkpoint by quantizing many of its weights to 4 bits. To compress models to NVFP4 , post-training quantization (PTQ) is a common method that covers most needs. But if you want to reach high throughput with tighter memory, more aggressive quantization is needed. Quantization-aware distillation (QAD) is an optimal choice in this case. Training Nemotron 3.5 Lightning to adapt to quantization noise with QAD produced an NVFP4 checkpoint that uses less memory and delivers higher throughput while preserving accuracy. This post demonstrates how QAD improves the Nemotron 3.5 Lightning model using NVIDIA Model Optimizer . We walk through the entire training pipeline, from the initial PTQ phase to the final distillation and evaluation. We show that QAD recovers accuracy degradation from aggressive quantization. Even with more conservative configurations, QAD consistently outperforms PTQ on agentic benchmarks, ensuring high quality while reducing memory usage . What is quantization-aware distillation? QAD uses the original full-precision model (teacher) to teach the quantized model (student). First, create a quantized model by running PTQ on the full-precision model. Then distill the frozen BF16 model into the quantized model using a KL divergence loss comparing the teacher’s and student’s logits. Figure 1 shows the two-stage QAD process used to build the Nemotron 3.5 Lightning NVFP4 checkpoint. The full-precision BF16 model serves as the frozen teacher and is also the starting point for Stage 1, a PTQ pass that quantizes weights to W4A16 to produce the quantized student. In Stage 2, the student is trained with QAD, running its forward pass through simulated quantization while a distillation loss aligns it with the teacher, yielding the final NVFP4 checkpoint with accuracy recovered close to baseline. Figure 1. The two-stage QAD process used to build the Nemotron 3.5 Lightning NVFP4 checkpoint The quantization-aware distillation process Follow the steps below to execute the QAD process. Step 1: Post-training quantization The first stage of QAD is running PTQ to produce a quantized checkpoint (student). Since we plan to run QAD, more aggressive quantization can be performed. For Nemotron 3.5 Lightning, we found that quantizing the Mamba linear layers to a more aggressive W4A16, rather than FP8, unlocked higher throughput without a large drop in accuracy. Normally, a median accuracy recovery of over 99% is targeted when performing PTQ only. When combined with QAD a target 95-99% median accuracy recovery can be targeted because QAD will recover more accuracy. This confirms that the quantization has been pushed far enough to bank the size and latency gains, while leaving clear room for QAD to close the gap in the next stage. We expect a small but meaningful drop on the evaluation benchmarks with W4A16 but plan to use QAD to recover the drop. This confirms that the quantization has been pushed far enough to bank the size and latency gains, while leaving clear room for QAD to close the gap in the next stage. Step 2: Quantization-aware distillation During QAD, every forward pass of the student model runs through simulated quantization so the model can account for the quantization noise it will encounter at inference. At the same time, it is trained to match the teacher through a distillation loss. Through the teacher signal, the student is learning to reproduce the full behavior of the model it came from rather than just predicting the next token. Training against both signals together allows QAD to maintain high quality with aggressive quantization. To learn more about the QAD process, see the end-to-end examples of QAD on NVIDIA Model Optimizer . How to develop Nemotron 3.5 Lightning NVFP4 with QAD using Model Optimizer The following sections explain the process we used to develop the Nemotron 3.5 Lightning NVFP4 checkpoint using QAD with NVIDIA Model Optimizer. Step 1: Obtain a PTQ checkpoint The base model, NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 , is the teacher. To create the student, run PTQ on the same base to quantize it to W4A16-NVFP4. Since both come from the same model, the student has a well-matched teacher to learn from. We tried several PTQ recipes for the student, which differ in how the weights are calibrated and how aggressively the Mamba projections and KV cache are quantized. This choice carries into training: max-calibrated recipes feed dynamic scale QAD, while MSE-based recipes feed frozen-scale QAD (see Step 2 of this section). A few settings are shared across every recipe. All of them quantize the lm_head to W4A16, a choice we call a faithful lm_head, while attention projection layers stay in BF16. Calibration uses 1,000 samples and runs on a single NVIDIA DGX B300 . The last recipe, four_over_six plus NVFP4 KV, is the most aggressive. It pushes only K and V to NVFP4 (W4A4) and leaves the \(QK^{\mathsf{T}}\) and \(\mathrm{attn} \cdot V\) batched matrix multiplications in BF16, with Q left unquantized. It’s advantageous to quantize more aggressively at the PTQ stage than if PTQ were the final step, because QAD recovers accuracy afterward. This enables you to select settings a PTQ-only recipe would avoid, such as taking the Mamba linear layers to W4A16 instead of the safer FP8. In fact, we want the PTQ checkpoint to show a small but meaningful drop on the evaluation benchmarks, bringing median accuracy recovery between 95 to 99%. This drop confirms we pushed hard enough to bank the size and latency gains, while leaving clear room for QAD to close the gap in the next stage. We evaluated many PTQ recipes, focusing on the five recipes below. For each recipe, we performed PTQ calibration with sequence length ranging from 8k to 128k. Prior experiments suggest longer sequence length produces better PTQ results. We found that a 32K sequence length provided the best PTQ results on four_over_six and performed all QAD experiments on the 32K calibration four_over_six checkpoint. After evaluating different PTQ recipes, we found the four_over_six with W4A16 Mamba linears provided the best tradeoff of accuracy degradation to boost in inference performance. For full accuracy details, see the QAD Checkpoint evaluations section. Table 1 shows the five PTQ recipes used to build the Nemotron 3.5 Lightning student checkpoints. Each recipe is defined by its weight format for the MoE, shared, and lm_head layers, its calibration method, its Mamba in/out projection format, and its KV cache format. All five use W4A16 NVFP4 weights and range from max-calibrated dynamic recipes to MSE-based static recipes, with the most aggressive variant pushing the KV cache to NVFP4. Light Green rows use dynamic scaling and light gray rows use static scaling in the training step. Recipe MoE / shared / lm_head weights Calibration Mamba in/out_proj KV cache max W4A16 dynamic NVFP4 max W4A16 NVFP4 FP8 mamba_fp8_max W4A16 dynamic NVFP4 max FP8 (W+A) FP8 MSE W4A16 static NVFP4 MSE (mean squared error) W4A16 NVFP4 FP8 four_over_six W4A16 static NVFP4 4/6 (MSE over M=6 versus M=4, arXiv:2512.02010 ) W4A16 NVFP4 FP8 four_over_six + NVFP4 KV W4A16 static NVFP4 4/6 W4A16 NVFP4 NVFP4 Table 1. The five PTQ recipes used to build the Nemotron 3.5 Lightning student checkpoints You can reproduce this recipe on your own model using NVIDIA Model Optimizer . The Hugging Face PTQ example walks through quantizing a Hugging Face model to NVFP4 using the different PTQ recipes previously explained. import modelopt.torch.quantization as mtq # define forward loop with dataset def forward_loop(model): for batch in calib_dataloader: model(batch) # Quantize base model to NVFP4 to create the PTQ student checkpoint # Example uses W4A16_NVFP4_CFG for quantization model = mtq.quantize(model, mtq.W4A16_NVFP4_CFG, forward_loop=forward_loop) The same Hugging Face PTQ example also covers the calibration data, supported formats, and export options. Once you have a PTQ checkpoint, you can move on to the QAD training configuration and scale strategy previously described. Step 2: QAD training This section explains the QAD training, including configuration and quantization scale handling. Setting the training configuration With the student checkpoint in hand, the next step is deciding how to train it. Two choices mattered most in our ablations: the sequence length for training and the data for distillation. Sequence length: Sequence length turned out to be critical for certain benchmarks, especially the longer-context ones, where training too short leaves accuracy on the table. Post-training supervised fine-tuning (SFT) used roughly 522K tokens, and our ablations showed that 522K sequence length was necessary for preserving long-context performance. To balance compute resources with sequence length, we performed initial ablations with 256K sequence length, and then scaled up to 522K for the final QAD run. Datasets: For the data mix, we ablated a few internal mixes before landing on our final recipe. For anyone looking to reproduce this work, we recommend starting from the open datasets NVIDIA has released, Nemotron-Post-Training v1 and Nemotron-Post-Training v2, which cover a similar distribution to what we used. Distillation recipe: The student starts from the PTQ checkpoint: the BF16 model is quantized first to NVFP4 using the Nemotron-3.5-Lightning -30B-A3B/lightning quantization recipe . Then distill from there rather than training the quantized weights from scratch. On each step, the same batch runs through both the BF16 teacher and the NVFP4 student, and the student is trained to match the teacher through a KL divergence distillation loss on the logits, with the teacher kept frozen as the reference signal. Distill at a constant learning rate of 5e-6 with no warmup, dropout disabled, and gradient clipping at 1.0, across two nodes times 8 GPUs (TP=2, EP=4). This is the same QAD workflow that ships with the NVIDIA Model Optimizer . Quantization scale handling during QAD With the checkpoint and training configuration set, the last choice is how to handle the quantization scales during QAD. We tried two strategies, and the application depends on how the PTQ checkpoint from Step 1 was calibrated. Figure 2 shows dynamic versus frozen scale, in which both lanes share the same quantized forward pass consisting of activations, simulated quantization, GEMM, loss with gradients returning to the FP weights. The top lane recomputes its quantization scale each step from the current tensors; the bottom lane keeps the PTQ-calibrated scale locked and updates only the weights. W(t) is the BF16 weights at training step t, s(t) is the scale recomputed from those weights at that same step, and s* is that scale captured once and held for the whole run. Figure 2. Dynamic versus frozen quantization scale Dynamic scale QAD: Starts from a max calibrated PTQ checkpoint. This is either the max or mamba_fp8_max quantization config from Table 1. Both carry dynamic weight and activation scales, so the scales are recomputed on the fly during training and both the weights and the scales adapt as the model learns. Frozen scale QAD: Starts from an MSE based PTQ checkpoint, such as MSE , four_over_six , or four_over_six + NVFP4 KV from Table 1. MSE and four over six arrive at their scales through a search to minimize quantization error, which is far too expensive to repeat at every step. Instead of recomputing the scales,  take the scales found during PTQ and freeze them during training. Only the weights are updated, while the scales stay fixed at their calibrated values. The scale strategy is chosen before training time. It follows directly from the PTQ recipe selected in Step 1. Max-calibrated checkpoints lead to dynamic scale QAD, and MSE-based checkpoints lead to frozen scale QAD. It can be a good idea to try multiple PTQ recipes with a mix of dynamic and frozen scales, and evaluate them to assess which one will benefit from QAD. QAD checkpoint evaluations The following experiments on intermediate Lightning checkpoints demonstrate how QAD allows for more aggressive quantization than PTQ alone. Both the PTQ and QAD checkpoints use the same aggressive W4A16 quantization format recipe and the same 21.19 GB footprint, against 65.85 GB for the BF16 baseline, so any difference between them comes from the method rather than from extra memory. QAD runs on intermediate checkpoints A and B were performed on 256K sequence length for rapid experimentation, whereas the final checkpoint QAD used 524K sequence length. Checkpoint A Checkpoint A (Table 2) is an SFT intermediate checkpoint. Checkpoint B (Table 3) is an intermediate RL checkpoint. We evaluated these checkpoints and found that QAD can recover some accuracy loss from aggressive quantization on only PTQ. Benchmark BF16 65.85 GB Aggressive PTQ 21.19 GB QAD 21.19 GB QAD gain vs PTQ MMLU-Pro 81.27 80.10 81.04 +0.94 GPQA-D 77.08 75.76 77.34 +1.58 AIME 2025 86.72 83.02 86.15 +3.13 AIME 2026 87.81 86.46 87.24 +0.78 SciCode Subtask 35.21 32.36 35.72 +3.37 SciCode Problem 13.28 10.63 12.19 +1.57 AA-LCR 54.00 52.38 53.37 +0.99 AA-Omni Acc. 14.05 13.62 14.57 +0.95 IFBench 74.00 73.00 72.96 -0.04 HLE 12.14 9.78 12.33 +2.55 LM Arena Proxy 10.05 8.81 10.05 +1.23 Median score recovery 100.00% 96.33% 99.72% +3.39 Table 2. Checkpoint A: Aggressive quantization with PTQ introduces accuracy loss. QAD recovers most of it by distilling the full-precision model into the quantized one. PTQ reaches 96.33% median score recovery, while QAD reaches 99.72% and improves 10 of the 11 benchmarks We tried QAD first on an early pre-release Checkpoint A, quantized more aggressively than what eventually shipped: W4A16 pushed all the way through the Mamba linear projections. Post-training quantization alone landed at 96.33% median accuracy recovery, with the largest regressions falling on the reasoning and coding benchmarks, where AIME 2025 dropped 3.70 points, SciCode Subtask 2.85, and HLE 2.36. Two hundred iterations of QAD from that same checkpoint brought recovery to 99.72%, returning AIME 2025 to within 0.57 points of BF16 and placing both SciCode Subtask and HLE slightly above it. Checkpoint B Table 3 shows checkpoint B, scored on the updated evaluation suite, with the same result. PTQ reaches 95.84% median score recovery and QAD reaches 98.53%, again at an identical 21.19 GB. The AA v4.1 Index is the clearest single measure here, rising 3.45 points from 20.03 to 23.48 against a BF16 baseline of 24.81. QAD improves 5 of the 9 benchmarks, so the median rather than any individual score carries the result. Benchmark BF16 65.85 GB Aggressive PTQ 21.19 GB QAD 21.19 GB QAD gain vs PTQ AA v4.1 Index 24.81 20.03 23.48 +3.45 GPQA-D 77.37 75.19 77.46 +2.27 HLE 10.84 10.89 11.35 +0.46 AA-LCR 51.00 47.38 50.25 +2.88 AA-Omni Acc. 17.58 16.78 16.60 -0.18 AA-Omni Non-Halluci. 62.24 69.02 64.67 -4.35 SciCode Subtask 41.12 39.57 38.98 -0.59 Tau3 Banking 8.04 5.77 8.25 +2.48 GDPval Norm. Elo 18.60 16.46 12.90 -3.56 Median score recovery 100.00% 95.84% 98.53% +2.69 Table 3. Checkpoint B, scored on the updated evaluation suite, shows the same result as Checkpoint A We repeated the experiment on a later Checkpoint B and observed the same pattern: 95.84% to 98.53% median recovery. We also measured the AA Index, which rose 3.4 points from 20.03 to 23.48 against a BF16 baseline of 24.81. The launch NVFP4 recipe is deliberately more conservative, so PTQ already sits close to baseline and QAD has considerably less to recover, with the remaining gains narrowing to a subset of the agentic benchmarks. Overall, QAD recovers the accuracy loss that aggressive quantization introduces. PTQ fixes the weights to a low-precision grid and accepts whatever error that produces, while QAD allows the model to continue training against that error and adapt to it, so most of the degradation is recovered rather than absorbed. That difference determines how far a recipe can be pushed. Configurations that PTQ leaves several points below baseline become viable under QAD, which means the memory and throughput budgets that would otherwise force a more conservative precision are now within reach. Final NVFP4 checkpoint Table 4 shows accuracy evaluations for the final NVFP4 checkpoint BF16 versus PTQ versus QAD. The final checkpoint was quantized conservatively to optimize accuracy, so PTQ already sits close to BF16 and there is little loss left for QAD to recover. Median score recovery is 99.24% for PTQ and 98.97% for QAD. The gains show up on the agentic and coding benchmarks instead, where QAD beats PTQ by 3.79 points on Terminal-Bench v2.1, 1.07 on SWE-Bench Multilingual, 0.65 on HLE and smaller margins on BrowseComp, SWE-Bench Verified, and PinchBench. Benchmark BF16 PTQ QAD AGGREGATE Score recovery (median) 100.00 99.24 98.97 AA v4.1 Index 24.51 24.05 23.94 AGENTIC & CODING Terminal-Bench v2.1 24.44 22.05 25.84 SWE-Bench Multilingual 37.13 37.00 38.07 BrowseComp 39.50 39.50 40.00 SWE-Bench Verified 52.20 52.20 52.60 PinchBench 83.86 84.78 85.15 τ²-bench Telecom 59.65 60.96 59.87 τ³-bench Banking 9.90 9.48 7.84 KNOWLEDGE, REASONING & INSTRUCTION FOLLOWING LM Arena proxy 12.06 11.10 11.91 HLE (w/o tools) 10.80 10.70 11.35 AA-Omniscience, accuracy 17.47 16.35 16.80 AA-Omniscience, non-halluc. 68.88 69.20 69.03 SciCode (subtask) 32.51 31.21 30.73 GPQA Diamond (w/o tools) 76.45 75.38 74.43 GDPval (norm. Elo) 17.19 19.09 17.02 IFBench (loose) 72.00 74.04 71.21 LONG CONTEXT AA-LCR 52.38 50.00 48.50 Table 4. Accuracy evaluations for the final NVFP4 checkpoint BF16 versus PTQ versus QAD For this final checkpoint, we optimized for accuracy, meaning more conservative quantization using Nemotron-3.5-Lightning-30B-A3B/lightning_w4a16_nvfp4_4o6 . PTQ therefore lands close to the BF16 baseline and QAD has much less loss to recover, with median score recovery at 99.24% and 98.97%, respectively. The gains from QAD are smaller than under aggressive PTQ, but they show up where it matters: QAD recovers accuracy on several key agentic benchmarks, including Terminal-Bench v2.1, SWE-Bench Multilingual, BrowseComp, PinchBench, HLE, and AA-Omniscience accuracy. How to run the QAD recipe with Model Optimizer The full QAD recipe ships in NVIDIA Model Optimizer as a single launcher megatron_lm_qad.yaml that runs the whole pipeline end-to-end. The same file works for both Nemotron 3 Nano and Nemotron 3.5 Lightning. Point it at the model you want and launch. The final NVFP4 checkpoint is published on Hugging Face. # from tools/launcher source .env-slurm uv run launch.py --yaml examples/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16/megatron_lm_qad.yaml --yes Step 1: Create the PTQ student The first step here is to use PTQ to quantize the BF16 teacher into the NVFP4 student. Nemotron 3.5 Lightning does not use RoPE, so --max-position-embeddings has no effect on positional encoding here. Here we keep it equal to its maximum context length (1M) for consistency. For models that do use RoPE, leave it at the config.json value. task_1: # quantize the BF16 teacher into the NVFP4 student script: common/megatron_lm/quantize/quantize.sh args: - --seq-length 32768 --max-position-embeddings 1048576 - --calib-size 32 environment: - QUANT_CFG: MAMBA_MOE_NVFP4_CONSERVATIVE_CFG - TP: "1" - EP: "4" Step 2: Distill the student from the frozen teacher QAD is enabled by setting the teacher through --export-kd-teacher-load with --modelopt-enabled added. Under the hood, the launcher invokes the Megatron-LM training entry point, finetune.sh , which runs the actual distillation loop against the frozen teacher. To reproduce our setup, point the launcher at your PTQ student checkpoint and teacher, adjust the sequence length and data mix to match your target, and launch. We distill at a constant 5e-6 learning rate with dropout off and gradient clipping at 1.0, on a single Nemotron-Post-Training v2 chat shard: task_2: # distill the NVFP4 student against the BF16 teacher script: common/megatron_lm/train/sft.sh args: - --seq-length 32768 --max-position-embeddings 1048576 - --micro-batch-size 1 --global-batch-size 16 - --train-samples 6400 # -> 400 iterations - --modelopt-enabled - --export-kd-teacher-load /cicd/megatron-lm-bf16/.../BF16-MCore - --lr 5.0e-6 --lr-decay-style constant --lr-warmup-samples 0 - --clip-grad 1.0 --weight-decay 0.0 - --attention-dropout 0.0 --hidden-dropout 0.0 environment: - DATASET: nvidia/Nemotron-Post-Training-Dataset-v2 - TP: "1" - EP: "4" A final export task converts the trained student into a deployable Hugging Face NVFP4 checkpoint. To reproduce on your own model, swap the model paths and choose your QUANT_CFG (which sets the matching scale strategy from Step 3); to train at a higher sequence length, set both –seq-length and –max-position-embeddings to a higher value, such as 524,288 (524k). For the complete launcher, see megatron_lm_qad.yaml . How to run the QAD recipe with Megatron-Bridge The same steps also run through Megatron-Bridge , a PyTorch-native library within the NeMo Framework that provides pretraining, SFT, and LoRA for popular language, vision-language, audio, and multimodal model support. It serves as a bridge between Hugging Face and Megatron Core providing bidirectional checkpoint conversion between the two formats. Projects can leverage Megatron Core parallelism capabilities or export models for various inference engines, with built-in verification. The mbridge_qad.yaml launcher runs the entire process end-to-end as four tasks: tokenize the training data, quantize the BF16 model into the NVFP4 student, distill that student against the frozen teacher, and export a deployable Hugging Face checkpoint. Here we use pre-tokenized data, the Nemotron-Post-Training-Dataset-v2 chat split is tokenized once up front with megatron_preprocess_data and passed to distillation through --data_paths . # from tools/launcher source .env-slurm uv run launch.py --yaml examples/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16/mbridge_qad.yaml --yes This runs on eight nodes with four GPUs (TP=1, PP=1, CP=4, EP=16), giving DP=8, with microbatch size=1 and global batch size=64; 200 iterations cover roughly 419M training tokens. The pipeline comprises the three steps detailed in this post, wrapped in four tasks: Import the BF16 model as the Megatron-Core teacher Quantize a separate student Distill the student against the frozen teacher Export the result This runs on two nodes with eight GPUs (TP=2, PP=1, EP=4), giving DP=8, with microbatch size=1 and global batch size=16; train samples=6,400 yields 400 iterations. Learn more QAD makes it possible to quantize aggressively and still hold accuracy close to the original model. On a compact, sparsely activated model such as Nemotron 3.5 Lightning, this is exactly what unlocks a strong NVFP4 checkpoint while preserving accuracy. The full QAD recipe for Nemotron 3.5 Lightning is available in NVIDIA Model Optimizer . We hope you try it on your own models and build on what we have shared here. To learn more, check out the following resources: Nemotron 3.5 Lightning Model Optimizer 4o6 yaml recipe Nemotron 3.5 Lightning Megatron LM QAD Nemotron 3.5 Lightning Megatron Bridge QAD yaml NVIDIA Model Optimizer Github Repo NVIDIA Model Optimizer QAD Documentation NVIDIA Model Optimizer LLM PTQ Documentation Acknowledgments We would like to thank the Nemotron team for building and open sourcing Nemotron 3.5 Lightning and the post-training datasets that made this work possible, and the NVIDIA Model Optimizer team for the quantization and distillation tooling that powers this recipe. We are grateful to Asma Kuriparambil Thekkumpate, Carlo del Mundo, Chenjie Luo, Daniel Lo, Daria Levy, Frank Sun, Hung-Yueh Chiang, James Shen, Jinhang Choi, Sweta Priyadarshi, Konstantinos Krommydas, Meng Xin, Rohan Joshi, Wei-Ming Chen, Victor Cui, Trenton Starkey, and Yaniv Galronfor their contributions to this work, including the recipe development, ablations, evaluations, and reviews that shaped the results presented here.