AMD admits its Instinct MI300X AI accelerator still can't quite beat Nvidia's H100 Hopper

[DragonSlayer101]

In context: The first official performance benchmarks for AMD's Instinct MI300X accelerator designed for data center and AI applications have surfaced. Compared to Nvidia's Hopper, the new chip secured mixed results in MLPerf Inference v4.1, an industry-standard benchmarking tool for AI systems with workloads designed to evaluate AI accelerator training and inference performance.

On Wednesday, AMD released benchmarks comparing the performance of its MI300X with Nvidia's H100 GPU to showcase its Gen AI inference capabilities. For the LLama2-70B model, a system with eight Instinct MI300X processors reached a throughput of 21,028 tokens per second in server mode and 23,514 tokens per second in offline mode when paired with an EPYC Genoa CPU. The numbers are slightly lower than those achieved by eight Nvidia H100 accelerators, which hit 21,605 tokens per second in server mode and 24,525 tokens per second in offline mode when paired with an unspecified Intel Xeon processor.

When tested with an EPYC Turin processor, the MI300X fared a little better, reaching a throughput of 22,021 tokens per second in server mode, slightly higher than the H100's score. However, in offline mode, the MI300X still scored lower than the H100 system, reaching only 24,110 tokens per second.

The MI300X supports higher memory capacity than the H100, potentially allowing it to run a 70 billion parameter model like the LLaMA2-70B on a single GPU, thereby avoiding the network overhead associated with model splitting across multiple GPUs at FP8 precision. For reference, each instance of the Instinct MI300X features 192 GB of HBM3 memory and delivers a peak memory bandwidth of 5.3 TB/s. In comparison, the Nvidia H100 supports up to 80GB of HMB3 memory with up to 3.35 TB/s of GPU bandwidth.

The results largely align with Intel's recent claims that its Blackwell and Hopper chips offer massive performance gains over competing solutions, including the AMD Instinct MI300X. Likewise, Nvidia provided data showing that in LLama2 tests, a system with eight MI300X processors reached only 23,515 tokens per second at 750 watts in offline mode. Meanwhile, the H100 achieved 24,525 tokens per second at 700 watts. The numbers for server mode are similar, with the MI300X hitting 21,028 tokens per second, while the H100 scored 21,606 tokes per second at lower wattage.

Permalink to story:

AMD admits its Instinct MI300X AI accelerator still can't quite beat Nvidia's H100 Hopper

 

[ZackL04]

Blackwell is going to smash

 

[yRaz]

Blackwell is going to smash

Your wallet. Cost per frame will probably remain about the same. I'm sure we'll see $600 5060s.

 

[ZackL04]

Your wallet. Cost per frame will probably remain about the same. I'm sure we'll see $600 5060s.

Maybe, but we'll have to see on the graphics card market. AMD and Nvidia seem to offer similar performance in price ranges anyway..

Nvidia hasn't been the cheaper option for a while, and its higher price tag always comes with benefits, even when it was just as simple as more stable drivers. Not to mention better resale value.

A company looking to build the strongest AI server doesn't care as much about cost of investment vs how much work it can do and how long it will last.

 

[yRaz]

Maybe, but we'll have to see on the graphics card market. AMD and Nvidia seem to offer similar performance in price ranges anyway..

Nvidia hasn't been the cheaper option for a while, and its higher price tag always comes with benefits, even when it was just as simple as more stable drivers. Not to mention better resale value.

A company looking to build the strongest AI server doesn't care as much about cost of investment vs how much work it can do and how long it will last.

I'm fairly certain we've seen peak nVidia. The 50 series might be faster, but I don't think we're going to get another card that is revolutionary as the 4090 was. There was the 1080ti before it and maybe the titan is worth mentioning.

And as far as the AI market is concerned, not only are investors pulling their money out of nvidia stock, there really aren't many companies left to sell more chips to. They saturated the market with H100 chips to the point where they blew threw their MSRP of 36k to upwards of 60k PER CARD but now you can buy them new on Amazon for under 30k. people are selling pallets of them on ebay for around 25k. For data centers, they can buy as many of the need and link them together. With how data centers scale the cards, they won't see much benefit from faster single slot cards.

The AI market is slowing down and the solution to better AI isn't faster chips, it's more data. Without the data to make AIs with, the speed of the card becomes irrelevant.

 

[ZackL04]

I'm fairly certain we've seen peak nVidia. The 50 series might be faster, but I don't think we're going to get another card that is revolutionary as the 4090 was. There was the 1080ti before it and maybe the titan is worth mentioning.

And as far as the AI market is concerned, not only are investors pulling their money out of nvidia stock, there really aren't many companies left to sell more chips to. They saturated the market with H100 chips to the point where they blew threw their MSRP of 36k to upwards of 60k PER CARD but now you can buy them new on Amazon for under 30k. people are selling pallets of them on ebay for around 25k. For data centers, they can buy as many of the need and link them together. With how data centers scale the cards, they won't see much benefit from faster single slot cards.

The AI market is slowing down and the solution to better AI isn't faster chips, it's more data. Without the data to make AIs with, the speed of the card becomes irrelevant.

Then I'd better sell

 

[yRaz]

Then I'd better sell

I keep halving my position each time it reaches a new peak and putting that money into the SCHD ETF.

 

[themastergoose]

Blackwell is going to smash

I'll pass. I don't like my GPU's requiring a re-mortgage to purchase one.

 

[ZackL04]

I keep halving my position each time it reaches a new peak and putting that money into the SCHD ETF.

How is that working for you, looks like of flat since mid 2021?

 

[yRaz]

How is that working for you, looks like of flat since mid 2021?

Im into dividend/income investing. So the chart looks flat unless you reinvest the dividends. Also, I'm happy it looks flat because there is lots of talk of a coming recession in 25-26 as the economy has been slowing down. I'm protecting myself from a crash, essentially.

Everyone has different styles of investing and I'm not a financial professional. I just like the idea of income investing and I also hate what investors did to the real-estate market so I try my best to stay away from real-estate. I will not contribute to the housing crisis. The artifical housing crisis because there are more unlisted empty homes than their are buyers on the market.

If you knew what I knew about the state of the real-estate market, you wouldn't touch it. It is common knowledge that big firms like Blackrock have started to sell their real-estate positions even if it's at a loss.

My 5 year plan is commodities and income investing while I wait out the market correction and get into real-estate sometime in 2028-2030.

 

[ZackL04]

Im into dividend/income investing. So the chart looks flat unless you reinvest the dividends. Also, I'm happy it looks flat because there is lots of talk of a coming recession in 25-26 as the economy has been slowing down. I'm protecting myself from a crash, essentially.

Everyone has different styles of investing and I'm not a financial professional. I just like the idea of income investing and I also hate what investors did to the real-estate market so I try my best to stay away from real-estate. I will not contribute to the housing crisis. The artifical housing crisis because there are more unlisted empty homes than their are buyers on the market.

If you knew what I knew about the state of the real-estate market, you wouldn't touch it. It is common knowledge that big firms like Blackrock have started to sell their real-estate positions even if it's at a loss.

My 5 year plan is commodities and income investing while I wait out the market correction and get into real-estate sometime in 2028-2030.

I could see a real estate correction, or at least there needs to be one. I'm in the new housing business, and since rates have flattened, builders are incentivizing and material costs have leveled (or gone down) they are continuing to get buyers. That should only improve with rate drops next month. So the correction may never happen....

I do think the real problem was allowing whales to purchase all the available real estate, then scalp it all back to everyday people. I lean right, but I also think we need "more government" to keep better tabs on who is buying these homes and buildings to prevent this current issue. Cant just sit back and jack up rates either, people still need to buy and live in a home.

 

[yRaz]

I could see a real estate correction, or at least there needs to be one. I'm in the new housing business, and since rates have flattened, builders are incentivizing and material costs have leveled (or gone down) they are continuing to get buyers. That should only improve with rate drops next month. So the correction may never happen....

I do think the real problem was allowing whales to purchase all the available real estate, then scalp it all back to everyday people. I lean right, but I also think we need "more government" to keep better tabs on who is buying these homes and buildings to prevent this current issue. Cant just sit back and jack up rates either, people still need to buy and live in a home.

People need a stable place to live before there can be a functioning economy. There are new markets where the prices won't crash that much. I feel Arizona and Colorado won't see much of a crash. However, in my home state of Pennsylvania, I see thousands of houses that are empty and aren't in the market. They're essentially just a land bank.

I see 1 of 2 things happening to housing in my home town of Pittsburgh in the next 5 years. Either housing prices will drop 30-50% or wages will increase to match housing costs. Something in the middle is more likely to happen because we have people who are making 100k+ a year being house poor. 4 years ago, I could pay all my bills with less than 25% of my income. I make more money now than I did then and I check my bank account if I want to extra bag of chips or feel like going out to eat.

Anyway, regardless of what my story is, my personal opinion is that the fundamentals of how our economy works are so far off that I'm moving everything to low risk forawhile. I'm taking my profit from years of growth and putting it into something stable for awhile.

 

[toooooot]

I would guess that the process and materials costs are about the same for Nvidia and AMD, given they both use TSMC and other companies that help create chips.
Nvidia is probably making much more from one chip than AMD, while spending the same amount to make it.

 

[ZackL04]

I would guess that the process and materials costs are about the same for Nvidia and AMD, given they both use TSMC and other companies that help create chips.
Nvidia is probably making much more from one chip than AMD, while spending the same amount to make it.

I personally think they have more into their R&D

 

[nnguy2]

Doesn't matter when the AI bubble bursts.

 

[WhiteLeaff]

It's unprecedentedly stupid to underutilize the mi300x's huge buffer using such a lightweight model. You're still buying 2x mi300x for the price of one H100.

 

[MasterAce]

Who can spot an intel in this article?

 

[zamroni111]

Amd needs to fix many of the acceleration software:
zentorch doesn't work
instead, Intel ipex works in 7840u;

xdna npu driver for Linux gets compile error while there is no binary download which means the npu is unusable in Linux.

xdna llm torch plug in needs user to compile, instead of quick install like Intel ipex llm, which surprisingly works well in 7840u.

no rocm for igpu while Intel doesn't prohibit onednn for theirs

 

[Mr Majestyk]

Blackwell is going to smash

Let's see Nvidia offer 7900XTX raster with 4070 Ti RT for $500 and then you can boast, because that's roughly what the 8800XT will offer.