Exploring Performance Advances in Mixture of Experts AI Models on NVIDIA Blackwell

AI usage keeps expanding, and so does the demand for tokens (the units generated by language models). When usage grows, the winning platform is often the one that can generate more tokens per second without exploding cost and power. That is where Mixture of Experts (MoE) models and NVIDIA’s Blackwell platform intersect.

Note: This article is informational only and not purchasing or engineering advice. Performance depends on model, sequence length, batching, and software versions. Platform capabilities can change over time.

TL;DR

• Token throughput is the bottleneck for scaled AI services: more tokens per second usually means lower cost per answer.
• MoE models activate only a subset of parameters per token, improving efficiency while keeping model capacity high.
• Blackwell + inference software focuses on faster expert routing, better all-to-all communication, and low-precision execution to lift MoE throughput.

Skim Guide

• MoE basics: why experts help and what new bottleneck appears
• Why tokens/sec matters: cost, latency, and concurrency
• Blackwell focus: routing, communication, low precision, and software stack
• Key numbers: the claims worth tracking in benchmarks
• Action checklist: how to measure and deploy MoE sanely

MoE in one glance

• Many specialists: a model is split into multiple experts, and a router selects which experts to use per token.
• Fewer active weights: only part of the model runs each step, which can increase efficiency.
• New bottleneck: MoE often shifts pain from compute to communication (all-to-all traffic) and routing balance.

Why token throughput is the KPI

• User experience: perceived speed depends on per-user tokens/sec, not just peak cluster throughput.
• Cost per answer: higher throughput on the same hardware typically reduces cost per token.
• Concurrency: better tokens/sec per GPU means serving more users without buying proportionally more GPUs.

What Blackwell targets for MoE inference

• Low-precision acceleration: paths like NVFP4 aim to raise throughput while meeting accuracy targets.
• Faster expert communication: improving all-to-all efficiency matters as much as faster math for MoE.
• Lower overhead: kernel and launch optimizations reduce wasted time in real serving workloads.
• Software as the multiplier: inference libraries and serving features are a big part of observed gains.

Two useful reads

• NVIDIA Technical Blog: MoE inference performance gains on Blackwell
• NVIDIA Blog: MoE models and Blackwell NVL72 performance/cost claims

Numbers worth remembering

• Up to 2.8× per-GPU throughput lift is reported by NVIDIA from recent inference software-stack updates in a DeepSeek-R1 MoE scenario.
• 10× faster and about 1/10 token cost is a headline claim NVIDIA highlights for certain MoE deployments on GB200 NVL72 compared with HGX H200.

Action checklist for teams

• Measure both speed and feel: track tokens/sec and per-user responsiveness, not only batch throughput.
• Profile communication: for MoE, inspect all-to-all utilization and routing balance; it is often the limiter.
• Standardize serving configs: batching, sequence length, and KV-cache settings can change results dramatically.
• Validate quality under low precision: use an evaluation suite before switching precision modes in production.
• Plan mixed workloads: scheduling and isolation prevent MoE traffic spikes from starving other services.

Where this matters most

• Reasoning-heavy assistants: long answers must arrive quickly.
• Agent workflows: chained model calls multiply token demand.
• Enterprise automation: predictable latency matters as much as raw throughput.

FAQ: Tap a question to expand.

▶ What is a mixture of experts model in one sentence?

It routes each token to a small subset of specialized sub-networks (experts) so you get high capability without activating the full model every time.

▶ Why can MoE be faster and still be hard to run?

MoE reduces compute per token, but it increases communication complexity because expert routing can require heavy all-to-all data movement across GPUs.

▶ Does higher throughput always mean better user experience?

No. A system can have high aggregate tokens/sec but still feel slow if per-user responsiveness is poor. Both throughput and per-user interactivity matter.

Keep exploring

• Advanced techniques in large-scale AI inference and scaling
• Efficient long-context AI: managing throughput, memory, and latency
• Key advances in AI models, agents, and infrastructure