AI data centers are splitting inference across CPUs, GPUs, and RDUs to cut token cost
A Semiconductor Engineering roundtable finds that heterogeneous clusters - not faster GPUs alone - are the primary lever for reducing AI inference token cost in 2026.

The GPU-only data center is giving way to something more complicated. A Semiconductor Engineering roundtable published on 19 August 2026, drawing on engineers from Arm, Cadence, Expedera, Siemens EDA, and Synopsys, concluded that heterogeneous clusters combining CPUs, GPUs, NPUs, and custom accelerators are now the primary lever for reducing AI inference token cost[1]. The shift is structural, not incremental, and it is already in commercial production.
Inference disaggregation splits the pipeline by chip type
The core idea is that a single LLM inference request passes through stages with fundamentally different compute profiles. Inference disaggregation assigns the pre-fill stage, the decode stage, and agentic execution each to a separate cluster with its own hardware and software mix, stitched together - most commonly over Ethernet[1].
The practical consequence is that no single processor architecture is optimal across all three stages. Prefill is compute-bound and highly parallel, which suits GPUs. Decode is memory-bandwidth-bound, which favours architectures with deterministic dataflow. Agentic orchestration - managing tool calls, branching logic, and long-running reasoning loops - maps naturally to CPUs.
That blueprint moved from theory to production at Computex 2026 in Taipei on 3 June. Vista Equity Partners and Cambium Capital launched Vector Core Compute (VC2), an inference cloud combining Intel Xeon 6 CPUs for orchestration, SambaNova SN40 RDUs for decode, and NVIDIA Blackwell GPUs for prefill and prompt caching. Independent measurement by Artificial Analysis found the architecture at least two to three times faster than a GPU-only stack.
Parallelism strategies impose distinct network topology requirements
Moving to heterogeneous clusters does not eliminate the networking problem - it deepens it. Each parallelisation paradigm (tensor parallel, data parallel, context parallel, and pipeline parallel) carries a different network topology requirement[1]. Tensor-parallel workloads demand all-reduce collectives across high-bandwidth scale-up fabric such as NVLink; pipeline-parallel jobs tolerate lower-bandwidth point-to-point links. Operators must choose their interconnect before they choose their silicon mix.
The memory hierarchy adds another constraint. HBM cost and utilisation become critical variables as model parameter counts grow, because the ratio of static model weights to dynamic context determines how many concurrent requests a cluster can serve at acceptable latency[1].
Software is now the integration layer
The roundtable was explicit that hardware heterogeneity is only useful if software can exploit it. When no single vendor controls the end-to-end flow, software orchestration becomes the critical path for optimising across a mixed cluster[1]. Hyperscalers have an advantage here: placement-group abstractions that co-locate virtual machines on high-speed interconnects are already mature in public cloud environments[1].
The challenge is sharper for operators assembling best-of-breed silicon from multiple vendors. Each cluster segment - prefill, decode, agent execution - may run a different software stack, and the handoffs between them over Ethernet introduce latency that must be budgeted against the throughput gains from specialisation.
The near-term question is whether the software layer can keep pace with hardware fragmentation. SambaNova's next-generation SN50 chip, targeting 10x throughput at 500 tokens per second per user, is expected in the second half of 2026, which will widen the performance gap between decode-optimised silicon and general-purpose GPUs. Whether orchestration frameworks can route workloads across an increasingly diverse chip landscape without manual tuning will determine how quickly disaggregated inference scales beyond early adopters.
Written by Electronics Insider's automated desk from the sources above and published automatically. How we work.
Related
Design & EDAETH Zürich and NYU show JEDEC test loops miss worst-case HBM2 power by 23.6%
Researchers at ETH Zürich and NYU tested 36 real HBM2 chips and found JEDEC measurement loops understate peak power by 23.6%, with read ops and data-bus toggling as the main drivers.
15 Oct 2026
Design & EDAFlowAccel's Joel Jorgensen argues chip teams must design their workflows before AI can help them hit tape-out
Joel Jorgensen of FlowAccel argues that AI tools alone won't save a tape-out schedule - teams must first design the cross-team workflows that let those tools compound into real gains.
15 Oct 2026
SemiconductorsETH Zürich and NYU find JEDEC test loops underestimate HBM2 power by up to 35.4%
Researchers at ETH Zürich and NYU tested 36 real HBM2 chips and found JEDEC-specified current loops can underestimate peak power by up to 35.4%, with implications for AI accelerator thermal design.
15 Oct 2026