The big picture: AI acceleration is set to become one of the most prosperous hardware businesses in the upcoming months and years, and Nvidia is in a prime position to capture a significant share of this market. The H100 data center GPU is already proving to be a major revenue generator for the Santa Clara-based company.

For every H100 GPU accelerator sold, Nvidia appears to be making a remarkable profit, with reported margins reaching 1,000 percent of production costs. Tae Kim, a senior technology writer at Barron's, recently highlighted that Nvidia spends around $3,320 to manufacture a single H100 unit, which is then sold to end customers for a price ranging from $25,000 to $30,000. These estimates are provided by consulting firm Raymond James and seemingly encompass costs related to the onboard HBM memory chips as well.

If the estimations prove accurate, this could mark the onset of an unprecedented golden era for Nvidia's GPU business. The demand for H100 GPU units is so high that they are essentially sold out until 2024, Kim said. Meanwhile, AI companies are scrambling to secure sufficient GPU accelerators to fuel their generative models and AI-powered services. Foxconn predicts that the AI server market will reach a value of $150 billion by 2027, and these modern AI servers heavily rely on the robust computing capabilities of the latest Nvidia hardware.

The H100 GPU is based on the Hopper microarchitecture, designed as the data center counterpart to the Ada Lovelace architecture that empowers the latest generation of GeForce RTX gaming GPUs. However, Hopper's primary focus is not on gaming, as indicated by performance benchmarks. The H100 accelerator is equipped with a modified GH100 GPU featuring 14,592 CUDA cores, in addition to 80GB of HBM3 RAM with a 5,120-bit memory bus.

Nvidia is certainly riding a high wave in the current AI boom, although the suggested $3,320 estimated cost per H100 GPU proposed by Kim requires further clarification. Developing a new GPU is a costly and time-consuming endeavor, involving numerous hardware engineers and specialized workers. These workers require compensation, and recent estimates indicate that Nvidia's average salary for a hardware engineer is around $202,000 per year.

On the one hand, producing and selling H100 GPUs could be much more costly than the $3,320 expense reported. On the other hand, Kim has been following Nvidia for 30 years and now he's even writing a book as the "definitive history" of the company. Therefore, when it comes to Nvidia's inner working and cost levels, he likely knows what he is talking about.