Forward-looking: The AI boom has companies scrambling for Nvidia's H100 and A100 GPUs, which are in short supply and cost tens of thousands of dollars. Meanwhile, a modder has discovered and shared a method of performing AI tasks on hardware available for less than one percent of that price.
A modder recently published instructions for coaxing AMD APUs that cost around $100 into running AI tasks usually associated with far more expensive graphics cards. If it catches on, the method could significantly expand the number of people who can at least experiment with AI.
The most prominent players in AI today operate tools like large language models using H100 and A100 graphics cards that Nvidia sells for $25,000 to $30,000 (reportedly a 1,000% profit margin for the GPU manufacturer), and they can't get enough of them. Meanwhile, smaller-scale AI operations on consumer hardware typically involve high-end cards costing at least several hundred dollars.
However, Reddit user chain-77 discovered that a $95 Ryzen 5 4600G APU can do respectable AI work by telling Linux to see it as a 16GB GPU. Although the processor doesn't compare to dedicated cards in traditional graphics rendering, AI relies heavily on memory, where an APU's ability to allocate shared memory freely becomes an advantage.
Devoting half of a system's 32GB of RAM to the integrated GPU gives it more memory than many beefier dedicated chips. Responses on Reddit suggested that assigning more RAM to video memory might be possible with some motherboards, but chain-77 is currently unable to test the theory.
The resulting DIY AI device supports AMD's ROCm platform, enabling it to run tools like Pytorch and TensorFlow. When tested on Stable Diffusion, the 4600G generates a 50-step 512 x 512-pixel image in under two minutes, comparing favorably against some high-end dedicated cards. Fastchat, MiniGPT 4, and Whisper also work, but the APU struggled with LLaMA. The more recent $130 Ryzen 5 5600G, which TechSpot considered an excellent choice for compact PCs upon its 2021 launch, can perform the same trick as its predecessor.
On top of being cheaper, the APUs also draw relatively little power while running AI – about 0.35 KWh per day. The modder released detailed descriptions of his work on Medium and YouTube (above) for those looking to save money while performing AI workloads.
https://www.techspot.com/news/99847-modder-converts-95-amd-apu-16gb-linux-ai.html
