Nvidia posts Day-0 inference results for Alibaba's 2.4-trillion-parameter Qwen3.8 model on GB300 NVL72

Nvidia has published Day-0 inference results for Alibaba's open-weight Qwen3.8-2.4T-A95B, showing the 2.4-trillion-parameter model running across its GB300 NVL72 rack-scale system^[1]. The initial FP8 implementation, without additional model tuning, exceeds 4,000 tokens per second per GPU and 350 tokens per second per user^[1].

What Qwen3.8-2.4T-A95B is

Qwen3.8-2.4T-A95B carries 2.4 trillion total parameters but activates only 95 billion for each token, using a fine-grained mixture-of-experts (MoE) architecture^[1]. The model has 92 layers arranged in a 3:1 interleaved pattern: 69 of those layers run linear attention (Gated DeltaNet) and 23 run full attention (GQA). In the full-attention layers every token attends to every other token; in the linear-attention layers the growing KV cache is replaced with a bounded recurrent state, keeping memory requirements under control as context scales^[1].

The MoE structure contains 512 routed experts, with 10 routed experts and one shared expert activated per token^[1]. The model supports a native context of 262,144 tokens and can be extended to approximately one million tokens.

A configurable reasoning depth feature lets developers select low, high, or xhigh reasoning effort on a per-request basis^[1]:

• Low effort - suited to high-throughput document processing and similar tasks
• High effort - for standard coding and reasoning workloads
• xhigh effort - for complex multi-stage agentic pipelines that benefit from deeper inference compute

Why the GB300 NVL72 matters for a model at this scale

The GB300 NVL72 integrates 72 Blackwell Ultra GPUs and 36 Grace CPUs into a liquid-cooled rack, connected by a fifth-generation NVLink fabric that provides 130 TB/s of aggregate all-to-all bandwidth across the 72-GPU domain. That fabric is directly relevant to MoE inference: when expert routing sends activations between accelerators, the available interconnect bandwidth sets a ceiling on throughput. Each GPU carries 288 GB of HBM3e, giving the full rack roughly 20.7 TB of pooled high-bandwidth memory.

The rack-scale design differs from conventional GPU servers connected primarily through external networking. Because all 72 GPUs share one NVLink domain, the system can treat the entire rack as a single multi-GPU unit rather than a cluster of independent nodes^[1].

Deployment paths and what comes next

Nvidia is supporting several software routes rather than tying deployment to a single inference stack^[1]:

• Open-source serving recipes for SGLang, vLLM, and Nvidia Dynamo
• Nvidia NIM as a model-independent inference container
• NeMo AutoModel for post-training, including full supervised fine-tuning and LoRA

The open weights are available through Hugging Face and ModelScope. Alibaba's own Qwen3.8-Max service version, built on the same model, adds vision input, built-in tools, and a one-million-token context by default^[1].

Nvidia describes the current FP8 numbers as an initial result. Further optimisation is planned, including NVFP4 precision, which is expected to deliver additional performance gains. Whether those gains close the gap with proprietary frontier models at comparable scale will be the figure to watch as NVFP4 kernels mature.