Samsung caught optimizing the Galaxy S4 for benchmarks

Scorpus

Posts: 2,156   +238
Staff member

samsung galaxy s4 galaxy android smartphone benchmark cheating optimization

This wouldn't be the first time a company has been caught optimizing their products for a certain benchmark, but quite possibly a first in the world of smartphones. It appears to be Samsung's turn after the discovery of code in the Galaxy S4 that deliberately optimizes the smartphone for various testing suites. A "BenchmarkBooster" was found to include hard coded profiles for certain applications that, depending on the Galaxy S4 model, pushed the phones to their limits.

AnandTech did some snooping to discover that the Galaxy S4 i9500 - the model that features Samsung's Exynos 5 Octa chipset - has code that pushes the PowerVR SGX544MP3 GPU above normal frequencies in benchmarks. They found that in some applications, such as AnTuTu and GLBenchmark 2.5.1, the GPU ran at 532 MHz rather than the standard 480 MHz that is utilized during gaming: a 10.8% increase in clock speed that resulted in ~14% better scores.

Aside from the benchmarking software, at no other time did the Galaxy S4 push its GPU beyond 480 MHz, even in the most demanding of games, making it clear that Samsung is artificially lifting their scores.

The hidden profiles also forced the CPU to use a higher performance mode. Across a range of benchmarks including Linpack, Quadrant and GLBenchmark 2.5.1, upon launching the applications the CPUs were immediately forced to use the most powerful cores (Cortex-A15s) at their top clock speeds (in the case of the i9500 Exynos model), or activate all four cores and run them at their highest clock speeds (in the case of the i9505 Snapdragon model).

Samsung appears to have disabled downclocking the CPU when benchmarks are run, regardless of load, which presents a slightly different situation to real-world usage. Normally the CPU would have to detect heavy usage and either increase the core frequencies, activate more cores, or in the case of the Exynos model, switch to the A15 cores. There can be a slight delay associated with this that could degrade performance.

At the end of the day these performance optimizations don't mean a great deal to end-users, although it would be nice if Samsung made the higher GPU clock available to applications when they need it. Just be careful when comparing the Galaxy S4 i9500 against other phones in benchmarks, as the BenchmarkBooster cheat may have affected the results.

Permalink to story.

 
That should constitute a money back recall of every effected phone, under a penalty of false advertising. And then a notification of compliance or face a year cellular ban.
 
I won't be surprised if all the competition do this to be fair, and I'm not sure its false advertising as I'm pretty sure all the adverts were not filled with graphs and lots of geek numbers explaining the details, but this is cheeky non the less.
 
How appropriate that this story comes out right about the time major league baseball is handing out suspensions for performance enhancing drugs. Turns out the S4 is also juiced.

How do I order some for my car?

Oh.. and don't miss one of the most important lines of the story..
Galaxy S4 i9500 - the model that features Samsung's Exynos 5 Octa chipset

[FONT=arial]That's the international model. So if you're in the US and own an S4 you don't have a phone that has been giving phony benchmark numbers.... The conclusion of Anandtech's article states. "None of this ultimately impacts us." and "The rest of our suite [meaning, benchmarks they actually use] isn’t impacted by the aggressive CPU governor and GPU frequency optimizations on the Exynos 5 Octa " [/FONT]

[FONT=arial]So check which model S4 you have if you're curious. (US models use the Snapdragon 600 quad core). Basically Samsung was caught cheating with a phone only some countries sell, in benchmarks that aren't heavily used.[/FONT]
 
Samsung save yourselves! Say the code is for a future update where overclocking will become available. Then throw some free apps and accessories for the device and never release the update.
 
How appropriate that this story comes out right about the time major league baseball is handing out suspensions for performance enhancing drugs. Turns out the S4 is also juiced.

How do I order some for my car?

Oh.. and don't miss one of the most important lines of the story..
Galaxy S4 i9500 - the model that features Samsung's Exynos 5 Octa chipset

[FONT=arial]That's the international model. So if you're in the US and own an S4 you don't have a phone that has been giving phony benchmark numbers.... The conclusion of Anandtech's article states. "None of this ultimately impacts us." and "The rest of our suite [meaning, benchmarks they actually use] isn?t impacted by the aggressive CPU governor and GPU frequency optimizations on the Exynos 5 Octa " [/FONT]

[FONT=arial]So check which model S4 you have if you're curious. (US models use the Snapdragon 600 quad core). Basically Samsung was caught cheating with a phone only some countries sell, in benchmarks that aren't heavily used.[/FONT]

But the article clearly states this also affects the Snapdragons as well? It puts all 4 cores into a high frequency mode.
 
That should constitute a money back recall of every effected phone, under a penalty of false advertising. And then a notification of compliance or face a year cellular ban.
That is not quite how false advertising works. While you may be spot on in that they are claiming something that is not technically true, and can be called out on it, you would not be eligible for any kind money back refund. Why? Simple: you would have to prove that the altered benchmarks were the leading reason you bought a Samsung phone over another device. Doing this is a hard endeavour and something that would take a lot more trouble than it is worth. Unless it was the driving motivation for the purchase then that element of advertising that is misleading is irrelevant, because you essentially bought it because of other advertising by Samsung that was not misleading.

It is a very uphill battle that exists for a good reason: because without it, every Tom, **** and Harry would try and get refunds for even the most insignificant of things because, well, people (that is, even more than they do now).

In any case, not that anyone is going to care as folks tend to have a painfully short memory :(?
 
Samsung will probably come out with a statement that they were just testing how far the phone could be pushed under some benchmarks but never intended to leave those profiles on the phone. Sorry 'bout that.

I'm fairly sure other hardware providers also optimise code for benchmarks. I seem to remember it happening a lot on graphics cards.
 
But the article clearly states this also affects the Snapdragons as well? It puts all 4 cores into a high frequency mode.


Don't worry. The rest of us have already established Mike can't read. Learn to read Mike.

My bad, I must have been confused into thinking this article was about the international model based on the lines "Somehow both Anand and I ended up with international versions of Samsung?s Galaxy S 4, equipped with the first generation Exynos 5 Octa (5410) SoC"
and "More recently, a post over at Beyond3D from @AndreiF gave us reason to dust off our international SGS4s." from the article on Anandtech.

Maybe there's an article about a US version SGS4 being boosted, but the source for the above story used an international model to discover the hack.
 
In the end it has not effect in real world useage anyways. No one outside of enthusiats even care about what ALL companies are doing with their phones. So they make it seem faster in benchmarks, so what?
Would be nice if the article actually stated that they were orignally talking about the International version, which most of the article is based on. The US version gets little talked about. They should make that more clear. Again, your average user DOES NOT care and the as the article states makes no difference in real world useage.
 
All benchmarks given from the people who benefit will have some bias in it. This is why the only benchmarks that matter at the ones provided by a neutral party. If this was their first time doing this I'd be surprised. Much like how Apple who designs both the hardware and software, most likely does this sort of dishonest thing on a larger scale. In anycase, I don't believe any sane person believes the benchmarks that come directly from the people who made the device. Most people wait for something like Tom's Hardware to do a full review along with tons of benchmarks using different tools.
 
Good thing I stuck with my Motorola Razz Max Dual core much quicker now with the latest Jelly Bean update. Samsung S-Series has everyone running around. I've own three of their smart phones never like how they change the UI. Anyway shame on them for playing head games with the public.
 
It is a very uphill battle that exists for a good reason: because without it, every Tom, **** and Harry would try and get refunds for even the most insignificant of things because, well, people (that is, even more than they do now).
What do you think a recall is?

And yes, benchmarks are a form of advertising. Why else would they invest in false benchmarks. The fact that Samsung created this lie, is proof enough that they are using the benchmarks as a form of advertising. If not they would have had no reason to lie. Don't try and make this more complicated than it really is, thats the problem with our legal system today.

I let benchmarks and reviews for graphics cards and CPUs do all the advertising for me. In fact benchmarks are the next best advertising to word of mouth. You must not be referring to benchmarks as advertising, because you don't see them plastered in commercials or on billboards. Advertising that doesn't cost revenue is still advertising.
 
All these comments and not one response that accurately identifies the issue. So, here we go:

Mobile phones aren't made for benching. They are made for portability, performance, stability, battery life, balance. Consumer CPU chips, nor GPU's, are also not made for benching.

What do you do when you want to run the best bench, or if you're overclocking like mad with liquid nitrogen? You turn off Intel Speed Step, you turn off Turbo Boost, you turn off every other feature that limits or downclocks the CPU. You also make sure that the CPU can run at max speed no matter what.

Samsung included those lines of code to make sure that the benchmarks accurately represent a BENCHMARK RESULT. If you weren't aware, benchmarks are usually synthetic tests that don't represent real-world very well.

The only crime here is the overclock of the GPU. That can't be excused.
 
Here we go again. Why do we even have benchmarks if they don't show real world results?

I'm not arguing the differences in synthetic vs real world. I'm arguing the differences the over-clock made to both and making a statement it should not be allowed at all. Lets set an example to all manufacturers and make them recall every product, that does not function as advertised.
 
"And yes, benchmarks are a form of advertising. Why else would they invest in false benchmarks. The fact that Samsung created this lie, is proof enough that they are using the benchmarks as a form of advertising. If not they would have had no reason to lie. Don't try and make this more complicated than it really is, thats the problem with our legal system today.
I let benchmarks and reviews for graphics cards and CPUs do all the advertising for me. In fact benchmarks are the next best advertising to word of mouth. You must not be referring to benchmarks as advertising, because you don't see them plastered in commercials or on billboards. Advertising that doesn't cost revenue is still advertising."

You are making a logic mistake. The average consumer doesn't buy a smartphone the same way a computer enthusiast goes about buying a new GPU or CPU. I guarantee 99% of smartphone users don't even know you can benchmark a phone. Samsung probably did this because, quite frankly, who gives a ****? Does it really matter how high of a benchmark score your phone gets? Wouldn't you put more emphasis on it's feel, responsiveness, battery life, screen, or ANYTHING else? If Samsung did this to cater to the .01 percentage of users that like to stroke their e-peen, then that's just good marketing.
 
There was only an overclock to the GPU, which is not acceptable and there's no excuse.

But IMO making sure that the CPU runs at max speed throughout the test is a valid tactic, albeit sneaky I guess (since no one else has been caught doing it).

Frankly, there should really be a second graph showing 'approx performance while operating on battery', but I guess this would be a damn lot slower than the max potential.


Edit: Oh, and mobile phone benches have been flawed since the start. This exploit that Samsung used is pretty well known to Android gurus and can be done (amongst other exploits) very easily on all phones to bump up the benchmark dramatically.
 
I guarantee 99% of smartphone users don't even know you can benchmark a phone.
Which is irrelevant!

False advertising is what it is, regardless how many people know about it. The idea is to prevent false advertising, no matter how insignificant it may be. Now tell me how flawed my simple logic is.

Niether AMD or Intel is programing their CPU's to bench at 5.0GHz and then selling them at a lower frequency. I'd bet everyone would be screaming foul play under those conditions. This case is no different.
 
Niether AMD or Intel is programing their CPU's to bench at 5.0GHz and then selling them at a lower frequency. I'd bet everyone would be screaming foul play under those conditions. This case is no different.

But, the chip in the Samsung isn't being sold at 0.5GHz. Neither is it downclocked. The full potential 1.x GHz (or whatever it is) is still there, in your phone, waiting. It's only Android and the kernel that limits the speed to optimise battery life.
 
Now that it is known that you can increase the clocks and other such things, look for the "xda" types to take advantage of that, and make some interesting mods...win win?
 
Now that it is known that you can increase the clocks and other such things, look for the "xda" types to take advantage of that, and make some interesting mods...win win?

You don't increase clocks! The CPU is still clocked the same at the hardware level!
 
What do you think a recall is?

And yes, benchmarks are a form of advertising. Why else would they invest in false benchmarks. The fact that Samsung created this lie, is proof enough that they are using the benchmarks as a form of advertising. If not they would have had no reason to lie. Don't try and make this more complicated than it really is, thats the problem with our legal system today.

I let benchmarks and reviews for graphics cards and CPUs do all the advertising for me. In fact benchmarks are the next best advertising to word of mouth. You must not be referring to benchmarks as advertising, because you don't see them plastered in commercials or on billboards. Advertising that doesn't cost revenue is still advertising.
But it is irrelevant. The most that would happen would be that Samsung would issue a patch or statement, or both. there will never be anything else because false advertising only applies when you can prove that you bought a product based on that one piece of advertising (the benchmarks).

They may be advertising falsely, but it is only relevant if you can prove that it is the number one reason you bought the device. After all, I still have not been able to fly regardless of how many Red Bulls I drink.

If it means that much to you then do something about it. No, not that, actually DO something about it. Write to Samsung; write to your local or state member; get in contact with the trade practices department of your state government (do not do that unless you have contact Samsung first); do research and see if other manufacturers do this same practise. If you are not going to do anything about it why the flip are you going on and on to us?

Either bring the fight to them or move on...
 
Back