We all are well aware of NVIDIA and the AI "gold mine" that has recently taken everyone by storm. In the midst of everything stands Team Green's H100 AI GPUs, which are simply the most sought-after piece of hardware for AI at the moment with everyone trying to get hands on one to power their AI needs.

NVIDIA H100 GPU Is The Best Chip For AI At The Moment & Everyone Wants More of Those

This article isn't particularly news but highlights readers on the current situation of the AI industry, and how companies are revolving around the H100 GPUs for their "future".

Before we go into the crux of the article, giving a recap becomes a necessity. So, at the start of 2022, everything was going fine with the usual developments. However, with November's arrival, a revolutionary application emerged named "ChatGPT", which established the foundations of the AI hype. While we cannot categorize "ChatGPT" as the founder of the AI boom, we certainly can say that it acted like a catalyst. With it emerged competitors like Microsoft and Google, getting forced into an AI race to release generative AI applications.

You might say, where does NVIDIA come in here? The backbone of generative AI involves hefty LLM (Large Language Model) training periods, and the NVIDIA AI GPUs come in clutch here. We won't go into tech specs and factual bits since that makes things dull and no fun to read. However, if are into getting to know specifics, we are dropping a table below, highlighting every AI GPU release from NVIDIA, dating back to Tesla models.

NVIDIA HPC / AI GPUs

The question still isn't answered here, why the H100s? Well, we are getting there. NVIDIA's H100 is the company's highest-end offering, providing immense computing capabilities. One might argue that the bump in performance brings in higher costing, but companies tend to order huge volumes, and "performance per watt" is the priority here. Compared to the A100, the Hopper "H100" brings in 3.5 times more 16-bit inference and 2.3 times 16-bit training performance, making it the obvious choice.

2 of 9

So now, we hope the superiority of the H100 GPU is evident here. Now, moving on to our next segment, why is there a shortage? The answer to this involves several aspects, the first being the vast volumes of H100s needed to train a single model. An astonishing fact is that OpenAI's GPT-4 AI model required around 10,000 to 25,000 A100 GPUs (at that time, H100s weren't released).

Modern AI startups such as Inflection AI and CoreWeave have acquired humongous amounts to H100s, with a total worth accounting in billions of dollars. This shows that a single company requires huge volumes, even to train a basic-to-decent AI model, due to which the demand has been tremendous.

If you question NVIDIA's approach, one can say, "NVIDIA could increase production to cope with demand." Saying this is much easier than actually implementing it. Unlike gaming GPUs, NVIDIA AI GPUs require extensive processes, with most of the manufacturing assigned to the Taiwanese semiconductor behemoth TSMC. TSMC is the exclusive supplier of NVIDIA's AI GPU, leading all stages from wafer acquisition to advanced packaging.

H100 GPUs are based on TSMC's 4N process, a revamped version of the 5nm family. NVIDIA is the biggest customer for this process since Apple previously utilized it for its A15 bionic chipset, but A16 Bionic has replaced that. Of all of the relevant steps, the production of HBM memory is the most complicated since it involves sophisticated equipment currently utilized by a few manufacturers.

CoWos Packaging Utilized in NVIDIA's H100

HBM suppliers include SK Hynix, Micron, and Samsung while TSMC has limited its suppliers, and we are unaware of who they are. However, apart from HBM, TSMC also faces problems maintaining CoWoS (Chip-on-Wafer-on-Substrate) capacity, a 2.5D packaging process, and a crucial stage in developing H100s. TSMC can't match the demand from NVIDIA, due to which order backlogs have reached new heights, getting delayed to December.

So when people use the word GPU shortage, they're talking about a shortage of, or a backlog of, some component on the board, not the GPU itself. It's just limited worldwide manufacturing of these things... but we forecast what people want and what the world can build.

-Charlie Doyle, NVIDIA's DGX VP and GM (via Computerbase.de)

We have left out many specifics, but going into detail will deviate from our primary aim, which is to detail an average user about the situation. While for now, we don't believe the shortage could reduce and, in turn, is expected to increase. However, we could see a landscape shift here after AMD's decision to consolidate its position in the AI market.

DigiTimes reports that "TSMC seems to be particularly optimistic about demand for AMD’s upcoming Instinct MI300 series, saying it will be half of Nvidia's total output of CoWoS-packaged chips" It may distribute the workload across companies. Still, judging by Team Green's greedy policies in the past, something like this would require a severe offering from AMD.

Summing up our talk, NVIDIA's H100 GPUs are leading the AI hype to new heights, which is why this frenzy surrounds them. We aimed to wrap up our talk by giving readers a general idea of the whole scenario. Credits to GPU Utilis for the idea behind this article; make sure to look at their report too.

News Source: GPU Utilis