AMD next-gen APUs reportedly sacrifice a larger cache for AI chips

[Daniel Sims]

Why it matters: As chipmakers embark on a widespread transition to locally processed generative AI, certain users are still questioning the need of this technology. NPUs have emerged as a new buzzword as hardware vendors aim to introduce the concept of the "AI PC," yet their arrival prompts speculation about whether the valuable die space they occupy could have been allocated to more beneficial purposes.

According to members of the Anandtech forums, AMD has significantly reduced cache size to accommodate large AI chips on upcoming Strix Point hardware. If the reports prove accurate, it would suggest that AMD and other processor vendors are placing considerable bets on a trend that is still unproven.

User "uzzi38" claimed that AMD initially intended to equip Strix Point APUs with system-level cache, which would have substantially improved CPU and integrated graphics performance. However, this plan was replaced with an emphasis on enhanced Neural Processing Units (NPUs), which are positioned as the central feature driving the new wave of AI-enhanced PCs.

Another forum member, "adroc_thurston," added that Strix Point was originally intended to have 16MB of MALL cache.

Intel, AMD, and Qualcomm are heavily promoting AI as an integral feature of their upcoming generations of CPUs. They plan to leverage Neural Processing Units (NPUs) to locally process generative AI workloads, tasks typically handled by cloud services like ChatGPT.

Intel led the charge toward this trend with the launch of Meteor Lake late last year. It aims to boost NPU performance with subsequent releases such as Arrow Lake, Lunar Lake, and Panther Lake. AMD's Strix Point is also set to enhance its Zen 5 CPUs and RDNA 3.5 graphics chips with increased AI capabilities upon its launch later this year. These chipmakers are aligning with Microsoft's initiative for AI-powered PCs, which includes requirements for a dedicated AI key and NPUs capable of achieving at least 40 TOPs.

However, hardware makers and software developers have yet to fully explore how generative AI can benefit end users. While text and image generation are currently the primary applications, they face controversy over copyright and reliability concerns. Microsoft envisions generative AI revolutionizing user interactions with Windows by automating tasks such as file retrieval or settings adjustments, but these concepts remain untested.

Many participants in the Anandtech thread view generative AI as a potential bubble that could negatively impact AI PCs and multiple generations of SoCs if it bursts. If the technology fails to attract mainstream users, it may leave numerous products equipped with NPUs of limited utility.

Permalink to story:

AMD next-gen APUs reportedly sacrifice a larger cache for AI chips

 

[Watzupken]

Long and short answer, yes. It is a waste of die space. Most people don’t give 2 hoots about NPU or whatever AI TOPS race. Moreover, most AI service provider won’t let you use whatever AI capabilities your PC can afford since they will most likely move towards a paid subscription model to cover the cost of maintenance .

 

[Mr Majestyk]

More fool them. Who gives a crap what MS wants, does AMD dictate MS policy?

I'm sure Adobe fro example will suddenly allow "generative fill" to done locally rather than on their servers for a fee after very limited number of uses. This is just more BS from MS to allow their garbage class and useless copilot to run locally on a OS no one will want to install. Windows 11 is already bad enough, imagine the crap fest Windows 12 will be.

 

[J95qRP7]

More fool them. Who gives a crap what MS wants, does AMD dictate MS policy?

Unfortunately Microsoft does kind of set the policy for AMD and Intel though, so they need to care what Microsoft wants. If Microsoft has some new certifications for PCs that need 40 TOPS of AI performance and AMD doesn’t worry about it, vendors won’t be able to sell their devices that contain AMD hardware as certified. That means AMD needs to make a product/s that Microsoft dictates. I hope AI crashes harder than 3d TVs.

 

[brucek]

Yuck. Larger memory cache is highly likely to be useful in many scenarios. The use case for local vs. server-based "AI" seems specialized and unproven.

In fact, if having older hardware automatically disabled AI in future Windows versions, I'd view that as a great extra feature to go along with the larger cache.

I hope they'll make an enthusiast class chip with this configuration.

 

[Vanderlinde]

AI on local PC's is so hyped, for only one reason. Telemetry! If you have billions of users working with AI and feeding it's model - for both google and MS its a treasure of telemetry data.

 

[takaozo]

AMD always finds a way to kick their own sack after a time when they have some success.
From the same people that brought us "more cores and more cache" now "more AI".
And silly me was thinking that Zen5 will bring more L1, L2 and L3 along with more cores.
If only game developers will use this NPU to allow better interaction with NPC's and open worlds I see a great usage for it. Otherwise as all said using this to get more people's data, will only work for me if a BIOS switch allows me to disable this, like Intel AMT and EMA.

 

[Atmajaya2020]

If only game developers will use this NPU to allow better interaction with NPC's and open worlds I see a great usage for it.

it seems like the time for "if only" to be implemented is still very long..

 

[GodisanAtheist]

I didn't realize the **** posting on Anandtech rose to the level of news story on Techspot.

 

[Dimitriid]

Yeah well, at least in a couple years when the AI bubble crashes down these will be pretty cheap and plentiful, I guess that's something.

The only other silver lining would be that more people just fully rejects both the hardware and software push for AI and just stop complaining and install Linux instead of remaining on Windows

 

[JamesBlond]

Imagine you write an OS that obeys its master server, you click a button say "10hrs of Idle time" us the PC's CPU and GPU to do algorithms even if the owner of the PC does not know about it, Imagine you want to have 10000CPU's and GPU's all doing one calculation to compute something, you complain and say don't use my stuff, they says, its for the greater good and plus you clicked ok on the Agreement when installing the updates and OS.....no, I will stick with Windows 10 Tiny and 12th Gen Intel CPU for the next 10yrs...MS can go die slowly

 

[cawopid]

Unfortunately Microsoft does kind of set the policy for AMD and Intel though, so they need to care what Microsoft wants. If Microsoft has some new certifications for PCs that need 40 TOPS of AI performance and AMD doesn’t worry about it, vendors won’t be able to sell their devices that contain AMD hardware as certified. That means AMD needs to make a product/s that Microsoft dictates. I hope AI crashes harder than 3d TVs.

source please ?

 

[Thanthan]

Imagine you write an OS that obeys its master server, you click a button say "10hrs of Idle time" us the PC's CPU and GPU to do algorithms even if the owner of the PC does not know about it, Imagine you want to have 10000CPU's and GPU's all doing one calculation to compute something, you complain and say don't use my stuff, they says, its for the greater good and plus you clicked ok on the Agreement when installing the updates and OS.....no, I will stick with Windows 10 Tiny and 12th Gen Intel CPU for the next 10yrs...MS can go die slowly

Imagine inventing something entirely fake to justify spamming hate posts against a company on the interwebs… now I dislike Microsoft as much as the next guy, but can we please critique them on things they’ve have actually done, rather than things we imagine they could do?

 

[kiwigraeme]

Nvidia had CUDA cores for a long time.
Google it's Tensor cores

Nvidia is adding things like sdr to hdr
Both AMD, Nvidia have upscaling

Makes Perfect sense Microsoft wants this

If I was building a Media Centre in a few years . these chips would make a great build.
Upscale DVDs. emulators , improve sound and dialogue clarity and spaciousness .

Allow for auto translate to extract voice and generate subs

Really this is the chance for Clippy 2.0 to shine.
I for one want AI on my PC to sort out all my photos, videos, music , media etc laying around on various discs .

I mean file explorer in 2024 is still a joke .

Plus Windows Defender, Chrome security should get better, even if introduces another attack vector.
Plus will allow folks to write some pretty nifty games with low specs , ie make opponents seem more smart , unpredictable . that a set of rules . look up table , and probabilities or whatever current designers use .

Will allow better spell checkers etc etc

 

[mistyrain]

Well, at the end of the day, one tech will be as good as the people make it. Not so long ago people were expecting CPUs will go to 5GHz then 10 then 20. But, then cores came. So, this might be utilized and really be the next big step up.

 

[Puiu]

After reading the comments, I don't think people here understand just how big AI will be in just a few years. And not just the server sided AI, there will be a huge push to integrate on-device AI.

Both AMD and Intel want to be able to push these chips on the market for years to come just like how AMD is still selling a lot of AM4 chips. It's a very reasonable long term strategy.

 

[AlphaLyrae]

For laptop live camera effects like background replacement and such, NPU can be useful in offloading things like that from CPU or GPU to save power (this can extend to photo editing too, with smart object detection and click-hold object cut/copy/paste, not unlike iPhone). But, in the short term, I think most NPU silicon will be underutilized until more software is available and matures, as it has on smartphones. Microsoft wants to push more AI PC features that may require more TOPs, so they're requiring Intel and AMD to treble their current AI NPU silicon performance for future Windows.

A MALL cache would've been great for both CPU and iGPU, although when used together, iGPU should have priority during resource contention of a system-level cache. There's still a chance a MALL cache can be developed, but it might have to be in a more expensive active interposer and AMD may need to split up more of the IO-die into other chiplets to allow for silicon upgrades, like new GPU IP or new SoC platform features. This will take time to get down to the price ranges most APUs are at.

 

[Theinsanegamer]

After reading the comments, I don't think people here understand just how big AI will be in just a few years. And not just the server sided AI, there will be a huge push to integrate on-device AI.

Both AMD and Intel want to be able to push these chips on the market for years to come just like how AMD is still selling a lot of AM4 chips. It's a very reasonable long term strategy.

OR, AI is the next Crypto, a tech with a very narrow function that is being SOLD as "the next big thing" and will crash and burn instead. Any AI that is game changing wont be running on your 15 watt laptop, it will be running in server rooms consuming gigawatts of power. "digital assistants" have existed for a decade, calling them "ai" isnt changing anything, and they dont need special processing hardware. Not to mention the severe privacy issues.

Meanwhile, sacrificing cache directly impacts 99% of consumer workloads. More cache has ALWAYS benefitted us, and programs are not getting less memory intensive. AMD's GPUs would greatly benefit from having 3d cache available, as even the lowly 610m has shown.

 

[Aranarth]

Well, at the end of the day, one tech will be as good as the people make it. Not so long ago people were expecting CPUs will go to 5GHz then 10 then 20. But, then cores came. So, this might be utilized and really be the next big step up.

The reason we switched from single thread machines running faster and faster mhz wise to machines getting wider and wider cores wise is becuase you simply couldn't rev transistors that high without the cpu melting down.

That is why netburst failed and we switch from netburst to core.

 

[Puiu]

OR, AI is the next Crypto, a tech with a very narrow function that is being SOLD as "the next big thing" and will crash and burn instead. Any AI that is game changing wont be running on your 15 watt laptop, it will be running in server rooms consuming gigawatts of power. "digital assistants" have existed for a decade, calling them "ai" isnt changing anything, and they dont need special processing hardware. Not to mention the severe privacy issues.

Meanwhile, sacrificing cache directly impacts 99% of consumer workloads. More cache has ALWAYS benefitted us, and programs are not getting less memory intensive. AMD's GPUs would greatly benefit from having 3d cache available, as even the lowly 610m has shown.

Comparing AI to cripto makes zero sense. One has limited uses, the other is already putting people out of a job and you use it regularly.

Losing a bit in performance to have an dedicated AI module inside your CPU is not a bad deal. Do you really care about a few extra FPS on an APU where you will almost always be GPU limited? Let's not exaggerate.

To use your own made up percentages: 99% of consumers will not notice the reduced performance. And the remaining 1% are like us, people who know what they want, read 100 reviews and make informed decisions to maximise the value.

"Any AI that is game changing wont be running on your 15 watt laptop" - yes it will run, and yes it will be game changing. Having something like ChatGPT running on your device or video/image editors/generators is game changing. Developers can make applications that don't require expensive servers to run the AI code on. Imagine not having to upload your personal photo on a server to edit it with AI features (like you do with phone apps now), it huge. and low powered devices is exactly where you want such dedicated hardware since it is very power efficient.

Even today I had to use AI upscalers on several images that were too blurry to use in an website header. Being able to do that locally, easily and fast would be as you put it "game changing".

It's also a chicken and egg situation. Not having the hardware means that the software can't be made/released. This type of hardware needs to come out as fast as possible.

 

[NotQuiteSapien]

AI use cases are far from narrow. We often focus on the possibilities already available, but have difficulties imagining what local (Edge) AI can bring. Secure and private AI can sort, search, and enhance all my texts and family photos.

Games often feel dead and mechanical, but with the help of AI, a lot can be done to increase immersion:
- NPCs and bots can react intelligently and unpredictably.
- Animals can exhibit varied, believable behavior.
- Hearing the same "I took an arrow to the knee" for the fifth time kills immersion. With Large Language Models (LLMs), this can be a thing of the past, along with NPCs moving around and acting with purpose.
- AI voice and sound effect generation for NPCs.
- Natural and varied movement, making it possible to slip on a banana without needing to program it.

I can run LLM AI models on my local machine with my GPU, but to have that option in games, we need to be able to offload the tasks.
Where do we need extra speed today? The reason some games are CPU-restricted today often stems from the need for NPC/AI calculations (Baldur's Gate...). I will take 500% AI enhancement on the CPU over 5% generic.

 

[wiyosaya]

Unfortunately Microsoft does kind of set the policy for AMD and Intel though, so they need to care what Microsoft wants. If Microsoft has some new certifications for PCs that need 40 TOPS of AI performance and AMD doesn’t worry about it, vendors won’t be able to sell their devices that contain AMD hardware as certified. That means AMD needs to make a product/s that Microsoft dictates. I hope AI crashes harder than 3d TVs.

As I see it, the bigger driver should be the Microsoft should care what their users want. If they continue down this avenue, they may well find that their monopoly is being eroded because they stopped listening to what their users want.

IMO, AI is, at this time, little more than a blip in which people will lose interest. It simply allows people to waste even more time buried in whatever screen they are already buried in.

I am right there with you on your hope of an AI crash.

As I see it, its unfortunate, for the consumer, that AMD is following the cash cow. It does make sense from a profit standpoint, as much as I hate to admit it, for AMD to follow that cash cow and collect its droppings into their bag of tricks.

 

[redgarl]

Sony, Microsoft, Apple, Google, Oracle, Adobe... they all want chips to be able to provide local inference. New AI capabilities will need to be able to use local inference.

On this topic, I will trust more the expertise of the chip makers than any fanboys who have no clue about chip making.

 

[brucek]

On this topic, I will trust more the expertise of the chip makers than any fanboys who have no clue about chip making.

Sony, Samsung, Panasonic, LG, etc. all know a lot more about TVs and monitors than I do, but they all also tried their hardest to sell me 3D TVs, because they wanted the money.

 

[brucek]

"Any AI that is game changing wont be running on your 15 watt laptop" - yes it will run, and yes it will be game changing. Having something like ChatGPT running on your device or video/image editors/generators is game changing.

Even if the "computation" part fit on that 15 Watt laptop, how is it going to make sense for the global knowledge database to be stored there, and kept updated? The edge algorithms that perform the last steps in Google Search might run fine on a laptop, but the knowledge base that you're trying to access ultimately exists only on external resources. I don't see how ChatGPT can be any different. And once you accept you need to go off device, I don't see why it makes sense to put the hardware I'll use a few times a day on the client vs. to have it on the server where the same silicon can serve 100 or 1000 people instead.