GPUs are 70% of installed worldwide AI compute capacity by manufacturer (see figure 1), with custom silicon accounting for 30%, primarily Google and Amazon.
Fig. 1: Current installed worldwide AI compute capacity by manufacturer. Source: Epoch.ai
Installed AI compute capacity has grown 17X in three years. In 2023, Nvidia had 90% share and Google 10%. Custom AI compute took 20% share in the last 3 years. Will this continue? (Note, Epoch’s data does not include Meta, Microsoft, or Cerebras AI compute, which are relatively small.)
Let’s look at Nvidia’s formidable product line-up and recent very strong growth projections. Then we’ll look at the progress and advantages of custom AI Compute from hyperscalers and frontier model makers. Finally, we’ll consider what GPU players — Nvidia and AMD — can do to maintain share in the face of growing custom AI compute performance and competition.
Nvidia has impressive demand and a formidable systems offering
Nvidia announced earnings on August 26. In a CNBC interview that day, Jensen Huang said:
- Demand is superstrong and, incredibly, is accelerating;
- Nvidia expects 70% growth in the coming fiscal year (limited by supply);
- AI is now making money for everyone, so everyone wants more compute;
- Half of Nvidia’s business is hyperscalers, and half is everything else, including Neoclouds, Sovereign, etc.
- Nvidia’s GPU is fungible, usable across the entire lifecycle, and capable of running every model;
- Cloud providers say the AI compute they bought five years ago is worth more now;
- Regarding OpenAI’s Jalapeño and other competitive chips: “I’m okay with it,” Huang said in an interview with Mad Money. “There are so many XPUs that are being announced…lots of projects get started, lots of projects get canceled. We’re here to support our partners.”;
- AWS is going to buy 2M GPUs and millions of (Vera) CPUs.
Each new Nvidia GPU generation is much more efficient than its predecessors. Nvidia is delivering more than just the GPU. It provides the NVL72 scale-up networking, the racks, a low-latency inference option (Groq 3 LPX), an agentic-optimized CPU (Vera), BlueField DPU and Spectrum-6 SPX scale-out networking. All the pieces are engineered to work together and provide more performance boosts than what the GPU alone provides. At Hot Chips 2026, Nvidia presented five papers on its various new chips, 1/5 of the total.
Nvidia provided details on its Vera CPU at Hot Chips. Vera is optimized for low-latency compute for AI agents, which is accomplished by using much larger single-threaded cores. The company estimates this delivers 1.8X more performance for agentic workloads versus a “traditional” CPU.
Nvidia’s Vera Rubin is already ramping in full production with racks running at CoreWeave, Google Cloud, Microsoft Azure, Oracle Cloud, and Nebius. Nvidia’s robust revenue growth, and projections of 70% more growth next year, suggest the Vera Rubin ramp could increase the company’s share of installed AI compute base in 2026 and 2027.
At Hot Chips, Nvidia presented on Vera Rubin, showing pareto performance vs. GB300 (see fig. 2, below).
Fig. 2: Vera Rubin NVL72 performance on Deepseek-v4-PRO vs GB300. Source: Nvidia/Hot Chips 2026
One of the scarcest resources for data centers is power, so they want more TPS/MW. And many customers want very fast response on very large models (= more TPS/user). Vera Rubin gives both. Higher interactivity correlates with higher value, so Vera Rubin’s advantage over GB300 is largest – up to 30X – where it is most valuable. This is the result of numerous GPU innovations but also innovations like Vera.
Last year Nvidia licensed Groq’s LPU technology, and it hired Groq’s team to enable the LPU to work with Vera Rubin as a co-processor for much better low-latency response for the most valuable workloads. At Hot Chips, Nvidia detailed the LPU architecture and programming and showed several ways for the LPU to work with the GPU. As the benchmark below shows (using GPT-OSS-2T in this case, not Deepseek), Vera Rubin + LPU delivers much, much lower-latency inference, providing much greater TPS/user.
Fig. 3: LPU with Vera Rubin improves both TPS/MW & TPS/user. Source: Nvidia/Hot Chips 2026
AMD MI455x GPU & Helios
AMD had 2 presentations on its newest GPU and Helios rack. The company has significantly upped performance specs versus earlier GPUs. AMD’s scale-up domain is now 72 GPUs using Broadcom Ethernet switches. Helios is a complete rack-scale system. Like Nvidia, it has a full solution for its customers. AMD has come a very long way in a relatively short time and has grown worldwide installed share from 0% to 6%. Unfortunately, it did not provide a performance chart for any AI workload.
Do model makers have the edge in building the best AI compute?
At Hot Chips, Meta, Microsoft, and Google detailed their follow-on AI accelerators. And OpenAI detailed its first AI inference accelerator, Jalapeño, with very impressive results. The Hyperscalers/frontier model makers host a lot of the workload and clearly want to run more of it on their own optimized silicon.
The model makers have a big advantage. They know their workloads better than the GPU vendors, they know the split between training and inference, and more importantly, they know the bottlenecks and their strategic direction. And they are well positioned to run detailed benchmarks of their current silicon, planned silicon features, and Nvidia silicon using their huge installed compute capacity.
Hyperscalers and model makers have a significant percentage of their total compute workload consumed by their own models, both training and inference serving. In addition, Google and Amazon have been offering, and even selling, their TPU/Trainium chips to customers to shift their cloud customers’ workloads from GPUs to TPU/Trainium.
The scale of AI compute capacity being installed by the hyperscalers/OpenAI/Anthropic has grown by an order of magnitude. So now it is economical to optimize AI compute for specific workloads, starting with dedicated training chips versus those dedicated to inference. A few years ago, the incremental cost of doing 2 chips was maybe $100M+ ($30M+ masks, tools, team), which was prohibitive. But at today’s scales, it is affordable if the optimized chip can deliver 10% to 30% better throughput/watt with 10% to 30% smaller silicon area, because power and wafers are the scarcest resources in the constrained supply environment we are now in, which will likely persist through 2030.
OpenAI’s Jalapeño out of the gate appears to rival Vera Rubin
Jalapeño, OpenAI’s first chip, is very impressive, as were presenters Richard Ho, Ravi Narayanaswami, and Chris Leary (notably, all three of them are Google TPU alumni).
Jalapeño was developed in a short timeframe with a small team, augmented by extensive use of OpenAI’s AI tools. It was optimized for their workloads, but can run any models. It is for inference. They decided to optimize it both for prefill and decode, as well as keeping KV Cache (Key-Value) memory local.
They noted that HBM memory raw capacity should enable much higher token rates. They architected to avoid bottlenecks so as to utilize more HBM capacity. They expect to do better in each generation (they are already working on Gen2).
They use Broadcom Tomahawk 6 switches for a 128 Jalapeño domain and more Tomahawk switches for a 2048 Jalapeño global domain.
Jalapeño has been in testing for a couple months now, and OpenAI shared several benchmarks.
Fig. 4: DeepSeek performance on Jalapeño vs Nvidia GB300. Source: OpenAI/Hot Chips 2026
The Jalapeño benchmarks are all with single-token prediction (STP), given the short time they’ve had to do the benchmarks. They will apply MTP (multi-token prediction) in the future, which will boost performance 3 to 5X.
They benchmarked GPT-OSS, DeepSeek R1, and Kimi K2.5. They compared it to Nvidia GB300 STP and MTP. Jalapeño outperforms GB300 in all cases by a significant amount.
How does Jalapeño compare to Vera Rubin NVL72? Nvidia’s Vera Rubin presentation shows it to be 2 to 30X faster than GB300 (2X at lower interactivity, 30X at very high interactivity) – very roughly shifting the GB300 curve by this amount gets to around the Jalapeño performance, which suggests Jalapeño is roughly competitive with Vera Rubin. This is a guesstimate.
Following Hot Chips, SemiAnalysis published its analysis and estimated Jalapeño appears to go head-to-head with Vera Rubin.
Jalapeño is an incredible achievement. As we get more details, we’ll see if the results hold up. If so, OpenAI is likely to build as many Jalapeños as they can get Broadcom to supply – wafer allocations from TSMC will set an upper limit. But Jalapeño cannot help with model training, which is probably 1/3+ of OpenAI’s total compute needs.
OpenAI will continue to get compute wherever it can, but pushing Gen2 and Gen3 Jalapeño will likely be a high priority. When supply constraints ease, Jalapeño will be a fast-growing part of OpenAI’s total AI compute.
Interestingly, Nvidia invested $30 Billion in OpenAI not long ago.
Anthropic is building its own AI compute chip team
Anthropic recently hired a key member of OpenAI’s hardware team, Clive Chan, for its own AI compute chip. Anthropic has publicly confirmed recently that it’s building an in-house silicon team to design custom chips for Claude, calling it “the latest step in our multi-chip approach.”
Google TPU 8t & 8i
Google has been doing AI compute for 10 years. TPU #1 was in 2015, and it was optimized for inference. TPU #2 was for training. So Google has been optimizing for inference versus training for a long time, usually alternating between training and inference each year.
Now Google is launching TPU 8t for training and TPU 8i for inference at the same time. The scale of the business has grown an order of magnitude, so the cost of doing two chips in parallel is affordable and the benefits of getting the best use of scarce watts and scarce wafers is compelling. The TPU 8 generation provides about double the performance of previous TPUs.
Fig. 5: TPU 8i for inference is 1.4x the size of TPU 8t due to more HBM. Source: Google/Hot Chips 2026
The relative size of TPU 8i and 8t gives a sense of the savings of having optimized training silicon. A single chip that does both inference and training well would probably be another 10% to 20% larger than TPU 8i to add all the back-propagation logic needed for training.
The interconnect topologies for inference and training differ because of different needs:
- Inference uses Boardfly. Four TPUs on each tray are fully connected, then 8 trays are fully connected with copper links into a group, and then 36 groups are fully connected for 1,152 TPU 8i’s working together. The maximum number of hops is 7 to minimize latency, which is critical for inference performance. Though not mentioned in the Hot Chips slides, Google has earlier explained Boardfly uses optical circuit switches (OCS).
- Training uses Google’s traditional 3D torus. It has higher bisection bandwidth, which is critical for training performance, but which takes as many as 16 hops. Up to 9,600 TPU 8ts in a superpod share memory using OCS, which enables creation of any slice size or shape and easy mapping around any failing units. OCS is critical for training. Google has developed the Virgo Network to provide a global non-blocking cluster fabric for up to 134,400 TPUs for the largest frontier models.
Major TPU users include Anthropic, OpenAI, Meta, and Apple.
Morgan Stanley recently estimated Google’s sales of TPUs will be $9 billion in the second half of this year, $84 billion in 2027, and $108 billion in 2028. This is significant. (Nvidia in the most recent quarter had revenue of $96 billion, and it projects $108 billion in the coming quarter. Some 90% of this is data center revenue. Nvidia projects 70% growth next year.)
Google Cloud serves a very wide range of customers and workloads. Many of these customers and workloads will continue to want, or need, Nvidia GPUs. But Google Cloud is working to get them interested in migrating to TPUs.
Other custom AI accelerators at Hot Chips 2026
Meta’s first AI inference accelerator was the MTIA for accelerating its recommendation system models used with Facebook, WhatsApp, and Instagram. Meta said GPUs are not well suited for its workloads, which have terabytes of sparse embedding tables and memory-bound operations. Meta’s second accelerator, the MTIA 300, was designed for training recommendation models. It is now showing slightly higher performance/$ for large batch training than GPUs. MTIA 400 is in development for both GenAI and recommendation systems. Meta’s roadmap has more chips coming every six+ months. Meta knows its workload best, and where the bottlenecks are, so it can design hardware to accelerate the bottlenecks for its own workloads.
Cerebras, meanwhile, announced its CS-4, which offers 2X faster tokens/user than the current generation CS-3, and up to 10X throughput/watt of CS-3. This is important for customers that want the lowest latency model support. The company says CS-5 will double performance again in a year, then CS-6 another doubling the year after that with 3D stacked DRAM!
Fig. 6: Cerebras CS-4 doubles low-latency TPS/user at 10X more capacity. Source: Hot Chips 2026
Microsoft, SambaNova, and D-Matrix also presented their latest AI compute silicon.
What more can GPU makers do to maintain share over custom AI?
Nvidia is already spending more per year on technology development and product introduction than any of the custom AI silicon developers. Strong revenue growth will grow available cash.
Nvidia is using its capital to lock in supply. On its recent earnings call, company executives said their supply commitments more than doubled from $119 billion to $279 billion. Nvidia’s strong position with TSMC and deep pockets for memory purchases probably put it in position for several years to be the dominant supplier because of greatest access to the scarcest resources.
Even with this, Nvidia has supply constraints. As its CFO noted on the earnings call, the projected 70% increase in demand next year might be larger without supply constraints.
The biggest constraints in supply are probably TSMC wafer allocations and HBM supply.
Nvidia could better utilize these scarce resources by doing a dedicated training chip and a dedicated inference chip. Given its ability to crank out complex silicon, it might even do a general-purpose GPU alongside inference-only and training-only.
AMD might consider buying Cerebras (whose market cap is 1/10th of AMD’s) to provide an LPX-like solution for customers wanting very low-latency inferencing. Or perhaps one of the many AI compute startups, like Etched, could provide a similar solution at a lower cost.
Google says OCS is a big part of its advantage. Nvidia and/or AMD could embrace OCS to improve their scale-out and even scale-up networks. (Full disclosure: I’m a board member at an OCS company.)
Copper interconnects are reaching their limit. GPU performance increases will require optical interconnects. The hyperscalers are cautious and are expected to first use NPO to minimize risk. But NPO cannot achieve the performance that CPO can. Nvidia and/or AMD could push to integrate CPO onto their GPUs early to deliver performance that others cannot match because they are using inferior NPO/copper. (Full disclosure: I’m also a board member at an optical engine company.)
Building models gives the model makers an edge in building inference. Nvidia has invested in developing open-weight free models like Nemotron, which takes 3 of the top 4 slots in Open Router’s tracking of most-used free models. Recently, Nvidia did a $7 billion deal with Poolside to license its open-weight model software and hire the bulk of its engineers (>100). Free models don’t make Nvidia money, so Jensen must see value in the insight they provide and the customer engagement they generate.
Conclusion
Nvidia has major moats, but it also has very large customers with huge resources. As supply of wafers and HBM exceeds demand in 2 to 5 years, its challenge (and AMD’s) will be that some of their customers can develop and use their own custom AI compute to run their own workloads.
Anthropic and OpenAI workloads are 100% their own. Google’s and Meta’s workloads are probably a majority their own. Amazon’s and Microsoft’s workloads are mostly their customers’. Nvidia and AMD need to continue to run hard to stay ahead of their customers’ efforts on AI compute. If Jalapeño turns out in production deployment to be as good (or better) than Vera Rubin, that will be a big warning sign for GPUs.