AI data centers are splitting inference across CPUs, GPUs, and RDUs to cut token cost

The GPU-only data center is giving way to something more complicated. A Semiconductor Engineering roundtable published on 19 August 2026, drawing on engineers from Arm, Cadence, Expedera, Siemens EDA, and Synopsys, concluded that heterogeneous clusters combining CPUs, GPUs, NPUs, and custom accelerators are now the primary lever for reducing AI inference token cost^[1]. The shift is structural, not incremental, and it is already in commercial production.

Inference disaggregation splits the pipeline by chip type

The core idea is that a single LLM inference request passes through stages with fundamentally different compute profiles. Inference disaggregation assigns the pre-fill stage, the decode stage, and agentic execution each to a separate cluster with its own hardware and software mix, stitched together - most commonly over Ethernet^[1].

The practical consequence is that no single processor architecture is optimal across all three stages. Prefill is compute-bound and highly parallel, which suits GPUs. Decode is memory-bandwidth-bound, which favours architectures with deterministic dataflow. Agentic orchestration - managing tool calls, branching logic, and long-running reasoning loops - maps naturally to CPUs.

That blueprint moved from theory to production at Computex 2026 in Taipei on 3 June. Vista Equity Partners and Cambium Capital launched Vector Core Compute (VC2), an inference cloud combining Intel Xeon 6 CPUs for orchestration, SambaNova SN40 RDUs for decode, and NVIDIA Blackwell GPUs for prefill and prompt caching. Independent measurement by Artificial Analysis found the architecture at least two to three times faster than a GPU-only stack.

Parallelism strategies impose distinct network topology requirements

Moving to heterogeneous clusters does not eliminate the networking problem - it deepens it. Each parallelisation paradigm (tensor parallel, data parallel, context parallel, and pipeline parallel) carries a different network topology requirement^[1]. Tensor-parallel workloads demand all-reduce collectives across high-bandwidth scale-up fabric such as NVLink; pipeline-parallel jobs tolerate lower-bandwidth point-to-point links. Operators must choose their interconnect before they choose their silicon mix.

The memory hierarchy adds another constraint. HBM cost and utilisation become critical variables as model parameter counts grow, because the ratio of static model weights to dynamic context determines how many concurrent requests a cluster can serve at acceptable latency^[1].

Software is now the integration layer

The roundtable was explicit that hardware heterogeneity is only useful if software can exploit it. When no single vendor controls the end-to-end flow, software orchestration becomes the critical path for optimising across a mixed cluster^[1]. Hyperscalers have an advantage here: placement-group abstractions that co-locate virtual machines on high-speed interconnects are already mature in public cloud environments^[1].

The challenge is sharper for operators assembling best-of-breed silicon from multiple vendors. Each cluster segment - prefill, decode, agent execution - may run a different software stack, and the handoffs between them over Ethernet introduce latency that must be budgeted against the throughput gains from specialisation.

The near-term question is whether the software layer can keep pace with hardware fragmentation. SambaNova's next-generation SN50 chip, targeting 10x throughput at 500 tokens per second per user, is expected in the second half of 2026, which will widen the performance gap between decode-optimised silicon and general-purpose GPUs. Whether orchestration frameworks can route workloads across an increasingly diverse chip landscape without manual tuning will determine how quickly disaggregated inference scales beyond early adopters.