Thirty years ago, general-purpose chips beat specialized ones in the semiconductor industry’s great strategy debate. Today the same debate is back, but with new constraints.
The 2-10x Pareto is easy to read as a chip win if the phase is not on the point. Prefill is the compute-heavy half and decode is the memory-bandwidth-bound half this piece already separates, so an on-die SRAM slice can look like 10x on the tokens that fit and do nothing once context spills to HBM. I would label each point with which phase ran on which device, and with the bytes that did not fit, before taking the frontier as evidence that specialization beat the GPU rather than that the workload was split. Dally's 1000x already shows most of the gain was format and instructions, which a general GPU can absorb the way CPUs absorbed vector units.
1. Hardware handoff challenge - while disaggregating the compute bound prefill to GPUs and bandwidth bound decoder to specialized chips like SRAM accelerators (Groq) look perfect on paper, the real bottleneck here is the KV cache transfer wall. Shuttling those massive context states across an external bus/network from one to another can easily wipe out the decode speed gains of the on chip SRAM
2. Heterogeneity - We have here not just the compute architectural heterogeneity (CPU/GPU etc.) but also the silicon process heterogeneity - GPU/Accelerator on bleeding edge, HBM on another & I/O and other dies on other processes. Couple that with the packaging complexities - CoWoS etc.)
3. Redefining “specialized” chips (if I may, thanks!) - Accelerators like Groq or Cerebras are not the single customer legacy ASICs (with customer owned mask). They are commercial domain specialised chips - just differentiating between ASICs and specialised chips.
The arithmetic in the Goldman line makes your case harder than you make it. 24x tokens against 2x power is a required 12x in tokens per watt by 2030. On Dally's own decomposition of the previous 1000x, process contributed 2.5x across roughly a decade. So almost none of the 12x can come from the node, which means it has to come from numerics, architecture and orchestration, and the room below 4 bits is thin as you say. That is a stronger argument for heterogeneity than the pendulum framing, and it is falsifiable.
One correction to the 1990s postmortem. Volume was the visible cause, but what actually killed Thinking Machines and Multiflow customers was that the hardware replacement cycle was shorter than the software amortisation period. You rewrote for the machine and the machine was obsolete before the rewrite paid back. That matters for the agent point: agents compress kernel authoring, which was never the expensive half. The expensive half was re-authoring against a moving target, plus numerical validation and debugging on hardware with no mature tooling, and that compresses much less.
Which gives the pendulum a measurable leading indicator rather than an illustrative curve: the ratio of hardware useful life to workload half life. Above one, specialisation pays. Below one, generality wins. You note accelerators need multi-year deployment to be economic while the workload keeps moving, which is that ratio heading the wrong way, so the real thesis is that orchestration decouples the two and lets you keep silicon when the workload shifts. Worth saying plainly, because it is the whole bet.
The number that would settle the disaggregation case is not the Pareto frontier at fixed power, it is stranded capacity in a live fleet: percentage of LPU hours idle while GPU queues are non-empty, over a quarter. A part that is excellent on decode and useless on prefill is only economic if the real prefill to decode mix stays inside the band you provisioned for, and that mix moves with context length and reasoning depth. Publish that figure and the 2 to 10x stops being a curve and becomes a result.
The historical comparison makes the case for specialization more persuasive—at what utilization level does coordinating different accelerators erase the performance-per-watt advantage?
The DVD decoder card is the whole story in miniature: good at its one job, extinct within about two years because the general chip learned to do it in software. So the test for a new AI chip isn't "is it faster today" but "who buys it by the millions next year." Volume pays for the next generation of design, and a narrow chip is betting the workload holds still. The GPU got out of that trap because gamers bought it in huge numbers first.
The 2-10x Pareto is easy to read as a chip win if the phase is not on the point. Prefill is the compute-heavy half and decode is the memory-bandwidth-bound half this piece already separates, so an on-die SRAM slice can look like 10x on the tokens that fit and do nothing once context spills to HBM. I would label each point with which phase ran on which device, and with the bytes that did not fit, before taking the frontier as evidence that specialization beat the GPU rather than that the workload was split. Dally's 1000x already shows most of the gain was format and instructions, which a general GPU can absorb the way CPUs absorbed vector units.
Great share, Zain Asgar, thanks.
A few points:
1. Hardware handoff challenge - while disaggregating the compute bound prefill to GPUs and bandwidth bound decoder to specialized chips like SRAM accelerators (Groq) look perfect on paper, the real bottleneck here is the KV cache transfer wall. Shuttling those massive context states across an external bus/network from one to another can easily wipe out the decode speed gains of the on chip SRAM
2. Heterogeneity - We have here not just the compute architectural heterogeneity (CPU/GPU etc.) but also the silicon process heterogeneity - GPU/Accelerator on bleeding edge, HBM on another & I/O and other dies on other processes. Couple that with the packaging complexities - CoWoS etc.)
3. Redefining “specialized” chips (if I may, thanks!) - Accelerators like Groq or Cerebras are not the single customer legacy ASICs (with customer owned mask). They are commercial domain specialised chips - just differentiating between ASICs and specialised chips.
The arithmetic in the Goldman line makes your case harder than you make it. 24x tokens against 2x power is a required 12x in tokens per watt by 2030. On Dally's own decomposition of the previous 1000x, process contributed 2.5x across roughly a decade. So almost none of the 12x can come from the node, which means it has to come from numerics, architecture and orchestration, and the room below 4 bits is thin as you say. That is a stronger argument for heterogeneity than the pendulum framing, and it is falsifiable.
One correction to the 1990s postmortem. Volume was the visible cause, but what actually killed Thinking Machines and Multiflow customers was that the hardware replacement cycle was shorter than the software amortisation period. You rewrote for the machine and the machine was obsolete before the rewrite paid back. That matters for the agent point: agents compress kernel authoring, which was never the expensive half. The expensive half was re-authoring against a moving target, plus numerical validation and debugging on hardware with no mature tooling, and that compresses much less.
Which gives the pendulum a measurable leading indicator rather than an illustrative curve: the ratio of hardware useful life to workload half life. Above one, specialisation pays. Below one, generality wins. You note accelerators need multi-year deployment to be economic while the workload keeps moving, which is that ratio heading the wrong way, so the real thesis is that orchestration decouples the two and lets you keep silicon when the workload shifts. Worth saying plainly, because it is the whole bet.
The number that would settle the disaggregation case is not the Pareto frontier at fixed power, it is stranded capacity in a live fleet: percentage of LPU hours idle while GPU queues are non-empty, over a quarter. A part that is excellent on decode and useless on prefill is only economic if the real prefill to decode mix stays inside the band you provisioned for, and that mix moves with context length and reasoning depth. Publish that figure and the 2 to 10x stops being a curve and becomes a result.
So involve Huawei style stack and cluster topology, greased with Tau scaling?
Interesting read! I compiled my thoughts on advancements in embedded computing as well. Please take a look : https://techwithnav.substack.com/p/the-new-compute-stack-lets-decode?r=91qxhl&utm_campaign=post-expanded-share&utm_medium=post%20viewer
The historical comparison makes the case for specialization more persuasive—at what utilization level does coordinating different accelerators erase the performance-per-watt advantage?
The DVD decoder card is the whole story in miniature: good at its one job, extinct within about two years because the general chip learned to do it in software. So the test for a new AI chip isn't "is it faster today" but "who buys it by the millions next year." Volume pays for the next generation of design, and a narrow chip is betting the workload holds still. The GPU got out of that trap because gamers bought it in huge numbers first.
Brilliant summary of the industry, thank you for putting it together.