Nvidia's AI Lead Extends Beyond GPUs

Nvidia's advantage in artificial intelligence is moving beyond just selling GPUs. The company's recent earnings and product launches highlight a new focus on the complex systems needed to run megascale AI data centers efficiently.

A new narrative has taken shape since the company's earnings on Wednesday. Investors are starting to realize that Nvidia's advantage goes far beyond GPUs. As AI's compute grows into the gigawatt scale, orchestration has become an increasingly complex task. Nvidia has built much of the state-of-the-art hardware needed to handle it.

For all the talk of compute as a commodity, it's still incredibly difficult to operate a megascale data center at peak efficiency. As deployments get bigger and faster, that challenge is only growing.

Rack by Rack

The company is currently rolling out its Vera Rubin architecture. This system pairs the Rubin GPU with a collection of other specialized units.

These systems are extremely specialized. Instead of churning through tokens, they're making sure everything outside the GPU works as efficiently as possible. If the GPU is the engine, these are the rest of the car.

The Vera CPU in particular is focused on the problem of orchestrating data. "Vera is important because there's only so much memory that you can put in a single server or any sort of compute platform," Jason Hardy, Nvidia's VP of storage technology, said.

As data centers have scaled up computing power, memory capacity has scaled up too. But getting that data to the GPU at the right time isn't straightforward. Companies are looking to drive tokens-per-watt lower and lower. They're realizing how important that kind of traffic direction is.

"We saw upwards of 3x improvement in these operations, where the Vera CPU is allowing for acceleration," Hardy said. "So now we can use our flash to its fullest potential, because we can get all that performance out of it without bottlenecking."

You can see versions of the same problem outside of Nvidia. When OpenAI developed its Jalapeño chip, a major focus was avoiding these challenges entirely. The goal was to minimize the amount of data that needs to be moved around.

"We designed Jalapeño to minimize data movement and communication delays," the company said in a blog post earlier this month. "Its large domain allows the entire workload to remain within one connected system, minimizing data movement and helping the complete request stay fast and efficient from beginning to end."

It's a different approach. It avoids data movement entirely by conducting a workload within one integrated chip. But the overall logic is the same. It increases efficiency with smarter traffic control instead of just more processor cycles. That in turn opens up a whole new layer of infrastructure for companies to compete over.

This new focus on data orchestration isn't automatically a win for Nvidia. The company will have to compete with rival chipmakers and hyperscalers just as it has with GPUs. But the competition has moved to a new layer. Building a rival GPU matters less than being able to make the entire system work efficiently.

And at least in the early stages, Nvidia looks to have a commanding lead.