AMD has publicly announced it is withdrawing its support from BAPCo, a non-profit consortium which develops and distributes a benchmarking program called SYSmark, refusing to endorse the latest version of the suite because its results are allegedly unrepresentative of the workloads used in everyday computing. The company also believes there's a bias in favor of Intel.
The suite uses a number of application-based benchmarks to recreate usage patterns in the areas of office productivity, data/financial analysis, system management, media creation, 3D modeling and web development. Among the applications used in SYSmark 2012 are Microsoft Office, Adobe Creative Suite, Adobe Acrobat, WinZip, Autodesk AutoCAD and 3ds Max, and others. But AMD argues that BAPCo is not taking advantage of GPUs for general purpose computing tasks, despite the fact that many applications support it, and instead solely relies on performance of CPUs.
"The heart of our complaint is this: the SYSmark benchmark is not only comprised of unrepresentative workloads (workloads that ignore the importance of heterogeneous computing and, frankly, favor our competitor’s designs), but it actually generates misleading results that can lead to very poor purchasing decisions," AMD's chief marketing officer Nigel Dessau wrote in a blog post.
He went on to explain how a previous complaint in 2010 got them a ban threat from BAPCo and that they nonetheless tried to work within the consortium in order to get the next-generation benchmark, SYSmark 2012, correct. Unfortunately that didn't happen, according to Dessau, who claims that only a small number of the benchmark operations in SYSmark 2012 actually contribute to the final score, and that a large proportion of the score is actually based on system performance during OCR and file compression tasks -- things an average user will rarely ever do.
Interestingly, AMD is not the only defector, as Nvidia and VIA have also decided to leave the consortium. Echoing AMD's sentiment, VIA said it wanted a more open and transparent process for developing fair and objective benchmarks that accurately measure real world performance, while Nvidia declined to comment on their reasons.
It's also not the first time AMD has had issues with BAPCo. In 2002, AMD accused the consortium of removing applications from the benchmark that showed a performance advantage for AMD over Intel.
For its part BAPCo is defending its 2012 benchmark, noting that AMD participated in its two-year development, and lamented that the latter had "once more violated the confidentiality agreement it signed in an attempt to dissuade customers from using SySmark to assess the performance of their systems." Other industry giants still backing BAPCo include Microsoft, Dell, Hewlett-Packard, Lenovo, Samsung, Sony, Toshiba and Intel.