Nearly all Android OEMs found to cheat in benchmarks

By on October 3, 2013, 7:00 AM
google, android, qualcomm, samsung, nvidia, htc, motorola, lg, benchmark, cheating

Yesterday, along with the first wave of Galaxy Note 3 reviews, it was revealed that Samsung once again had optimized their new smartphone for Android benchmarks. Through forcing the device's CPU to run at its maximum clock speed of 2.3 GHz in benchmarks, it scored 20% better than if the CPU was allowed to power gait cores like normal.

AnandTech dug deeper into the issue, testing a number of Android devices to see if OEMs other than Samsung were guilty of optimizing their smartphones for benchmarks. Asus, HTC and LG were caught boosting CPU speeds of their Qualcomm-based devices in a selection of benchmarks, while Samsung was the most aggressive. Out of the seven benchmarks checked, the Note 3 was optimized for six, while Samsung's Intel Atom-based Galaxy Tab 3 10.1 and the Exynos-based Galaxy Note 10.1 (2014) were also found guilty.

Motorola, alongside Google's Nexus devices and Nvidia's Shield, were found not to optimize the CPU for benchmarks. This isn't overly surprising, because the cheating code isn't part of vanilla Android or AOSP, which is essentially what is included on these devices.

The whole process of optimizing CPU clock speeds for running benchmarks is a phenomenal waste of engineering time and money, as in the end it has no real-world effect on performance. It's only making the devices look marginally better in reviews, and now that the media has exposed the trickery of OEMs, it's doubtful whether the effort is worth the negative attention.

Needless to say we'll be looking out for any benchmark cheaters in our upcoming smartphone reviews, making note of where benchmarks may be affected.




User Comments: 12

Got something to say? Post a comment
2 people like this | ikesmasher said:

In other news, I just ate breakfast.

Guest said:

If it's only a marginal difference then what prompted AnandTech to do such an in-depth study to reveal this "Trickery". Fact is, it's faster. Maybe not when we get it, but complaining about it would be like complaining that a fast car has a governor chip in it.

Scshadow said:

Ah... finally the truth revealed. Its been awhile since the first accusation against Samsung and just now someone finally decides, hey lets check everyone else too before running off to point the giant negative publicity finger at Samsung. So everyone or at least almost everyone that isn't running vanilla android is cheating with at least some of the benchmarks? Props to Samsung for being the most thorough in your cheating efforts. Seriously, if other brands are getting away with it, there is hardly any argument for blaming Samsung for following suit. At least, that is, not in this day and age of competitive marketing.

wastedkill said:

Big woop stop acting like cry babies thought benchmarks were meant to test the device thats what its doing would you rather it show it at its "meh" time or at its "ye this is how the device is how powerful the device is" time.

its not a cheat its just how stuff works I dont want to see a 790 doing 30fps on bf3 because its "not at its maximum" I want to test my hardware to the max so I know what it can do instead of what its not showing you it can do.

OneSpeed said:

AnandTech is trying hard to make their benchmark relevant in the marketplace.

Tygerstrike said:

I think the issue is less with the act of cheating and more with the expectation of performance that cheating gives. I deal with cellphones on a daily basis. I can tell you from my own experiences selling them that the customer doesnt give a rats ass about the specific specs. They want a big screen and lots of memory. Thats about all they care about. The only ones impressed with a phones individual specs are the technophiles. They are the ones truly impressed with the numbers. Unfortunatly, the technos are NOT the consumer base. The consumer base is ignorant and generally stupid when it comes to either the inner workings or their application and use in smartphones. Im not trying to be mean mind you. Just honest.

1 person liked this | Nobina Nobina said:

Even before articles like this one started showing up, I think that everyone with some knowledge knew that this was happening. And it's not only the case with smartphones, but with PC components also. Nvidia, Intel and AMD are doing the same for their stuff.

OneSpeed said:

I think the issue is less with the act of cheating and more with the expectation of performance that cheating gives. I deal with cellphones on a daily basis. I can tell you from my own experiences selling them that the customer doesnt give a rats ass about the specific specs. They want a big screen and lots of memory. Thats about all they care about. The only ones impressed with a phones individual specs are the technophiles. They are the ones truly impressed with the numbers. Unfortunatly, the technos are NOT the consumer base. The consumer base is ignorant and generally stupid when it comes to either the inner workings or their application and use in smartphones. Im not trying to be mean mind you. Just honest.

I agree with you to a point, but how would knowing the true specs make the user less ignorant. If a user's expectations are met in regards to performance, wouldn't that be alright? Consumers don't necessarily look for or buy the fastest performing CPU'd phones on the market. They generally look at the eco-system that phone brings with it, Google Play, Apple's Appstore, etc. and the odd thing a user might want their phone to do. For even the technophile, how often are they maxing out the CPU cores that are available to them, and for how much of the day? In the end, they are probably the first to upgrade their phones before their contracts are up. Just saying.

OneSpeed said:

Even before articles like this one started showing up, I think that everyone with some knowledge knew that this was happening. And it's not only the case with smartphones, but with PC components also. Nvidia, Intel and AMD are doing the same for their stuff.

Yes, and what that does is make these benchmarks less to non-relevant. I don't buy a computer/device based on the benchmarks; I buy them based on meeting application requirements.

IAMTHESTIG said:

The US government, like Samsung, is quite large. The US government, also cheats and lies. Coincidence?

Guest said:

Well, the almighty apple do it, so why not....

Load all comments...

Add New Comment

TechSpot Members
Login or sign up for free,
it takes about 30 seconds.
You may also...
Get complete access to the TechSpot community. Join thousands of technology enthusiasts that contribute and share knowledge in our forum. Get a private inbox, upload your own photo gallery and more.