Geekbench bans four generations of Galaxy devices over benchmark manipulation

[vannvicente]

Bottom line: Earlier this week, users discovered that Samsung was throttling thousands of apps on its flagship Galaxy S series phones, including its latest Galaxy S22, while conveniently excluding popular benchmarking tools like Geekbench. In response, the developers behind Geekbench have banned the last four generations of Samsung Galaxy devices from its services for manipulating benchmark results.

Thermal throttling on phones is nothing new. Most modern devices throttle when their processor becomes dangerously hot, which helps prevent overheating and maintains the battery's longevity. However, following the recent launch of the Samsung Galaxy S22 series, we reported that several users on Twitter and YouTube noticed that the built-in Game Optimizing Service (GOS) was pre-emptively slowing down common apps without an option to turn off the performance reduction.

This was corroborated by S22 owners on Korean social networking sites and forums, who noticed that apps were loading slower than expected.

Despite ostensibly being a "game" optimizing service, users have noticed that only a third of the titles on the list are games, with the rest being standard apps. There are roughly 10,000 affected apps, including common streaming services like Netflix and Spotify, social networking tools like TikTok, and even Samsung's own home launcher. Not present on the list are standard benchmarking tools like Geekbench, Antutu, 3DMark, and GFXBench.

The developers at Geekbench have responded by banning the last four generations of Samsung flagship phones from its comparison chart for benchmark manipulation, with further research revealing that GOS has been active since the Galaxy S10.

"Earlier this week, we were made aware of Samsung's Game Optimizing Service (GOS) and how it throttles the performance of games and applications. GOS decides to throttle (or not to throttle) applications using application identifiers and not application behavior," Geekbench tweeted out on Friday. The company added that they viewed the practice as "a form of benchmark manipulation."

After extensive internal testing we have determined the following Samsung Galaxy handsets use GOS:

- Samsung Galaxy S22 (all models)
- Samsung Galaxy S21 (all models)
- Samsung Galaxy S20 (all models)
- Samsung Galaxy S10 (all models)

— Geekbench (@geekbench) March 4, 2022

Geekbench points out that devices commonly use behavior to throttle, while GOS reduces performance based on the app itself. Most benchmarks are meant to be used comparatively -- giving consumers an idea of how different devices compare to each other in terms of performance. However, when a manufacturer pre-emptively thermal throttles common apps, these figures can no longer accurately represent the differences in performance between devices.

According to The Verge, Samsung has promised to publish an update to GOS that would enable users to configure the performance of apps directly. "We value the feedback we receive about our products and after careful consideration, we plan to roll out a software update soon so users can control the performance while running game apps," said Kelly Yeo from Samsung.

Geekbench policy prevents devices from being relisted on their services even after a patch fixes the issue, so the Galaxy devices likely won't return to the benchmark charts anytime soon.

This isn't the first time a phone manufacturer has attempted to manipulate benchmarks through this method. In July 2021, OnePlus was also caught throttling the performance of common apps on the OnePlus 9 and OnePlus 9 Pro. These devices were also subsequently banned from Geekbench.

Permalink to story.

https://www.techspot.com/news/93660-geekbench-bans-four-generations-galaxy-devices-over-benchmark.html

 

[Nanochip]

4 nanometers of fakeness. Decadent.

 

[yRaz]

I don't know how much this really matters for S series phones. My S21+ is so fast that I keep it in low power mode and I experience no performance issues. A full charge on low power mode gets me around 3 days of battery life and I'd much rather have battery than performance.

 

[DukeJukem]

If android could manage background processes better this wouldn't be an issue. I don't see how throttling other apps is considered manipulating a geekbench benchmark. If you're not using an app it should be throttled whilest in the background in order to prioritize foreground processes. I don't see the issue here.

 

[Nanochip]

If android could manage background processes better this wouldn't be an issue. I don't see how throttling other apps is considered manipulating a geekbench benchmark. If you're not using an app it should be throttled whilest in the background in order to prioritize foreground processes. I don't see the issue here.

If the normal user experience is for the phone to throttle apps, then Geekbench and other benchmarks should also have been throttled too, so as to paint the correct picture of how the phone performs. But Samsung didn't do this. It cheated. The fix is for Geekbench to also be throttled (just like the 10,000 other apps), and use the throttled score to compare the normal Galaxy performance to that of other phones.

Therefore, Samsung's behavior is manipulation because the truth is hidden. Samsung hides the fact that 10,000 apps run throttled on the Galaxy except for the flashy benchmark Apps (like Geekbench) used in reviews. Why are the benchmark apps (used by reviewers and article publishers) not included in the throttled list? Using Geekbench and other non-throttled benchmarks to compare scores of Galaxy phones to that of other phones would give users a wrong impression of the Galaxy's actual (I.e., throttled) performance in 10,000 other apps.

The hidden assumption behind comparing Geekbench scores from different phones is that the phones will also run other apps unconstrained (and only constrain apps due to heat/power management). But that assumption doesn't hold here, as shown by the evidence. The Geekbench scores have no connection to actual user experience! The reality is 10,000 apps are throttled all the time!! Samsung selectively chose not to throttle the apps used by reviewers and article publishers to paint the phone in a good light and that is manipulation and should be called out!

 

[Strawman]

If the normal user experience is for the phone to throttle apps, then Geekbench and other benchmarks should also have been throttled too, so as to paint the correct picture of how the phone performs. But Samsung didn't do this. It cheated. The fix is for Geekbench to also be throttled (just like the 10,000 other apps), and use the throttled score to compare the normal Galaxy performance to that of other phones.

Therefore, Samsung's behavior is manipulation because the truth is hidden. Samsung hides the fact that 10,000 apps run throttled on the Galaxy except for the flashy benchmark Apps (like Geekbench) used in reviews. Why are the benchmark apps (used by reviewers and article publishers) not included in the throttled list? Using Geekbench and other non-throttled benchmarks to compare scores of Galaxy phones to that of other phones would give users a wrong impression of the Galaxy's actual (I.e., throttled) performance in 10,000 other apps.

The hidden assumption behind comparing Geekbench scores from different phones is that the phones will also run other apps unconstrained (and only constrain apps due to heat/power management). But that assumption doesn't hold here, as shown by the evidence. The Geekbench scores have no connection to actual user experience! The reality is 10,000 apps are throttled all the time!! Samsung selectively chose not to throttle the apps used by reviewers and article publishers to paint the phone in a good light and that is manipulation and should be called out!

Really now? And please explain to me, if Galaxy throttles X app lets say, 50%, Y app at 70% and Z app at 30%, how much throttling should be applied to Geekbench in order for it to accurately demonstrate performance?

 

[OneSpeed]

I have never relied on benchmarks when I buy that next phone. I've been happy with the "real-time" performance of the Galaxy phones. Who f'ing buys a phone to run a throttling app??? If the phone performs well running your apps, these benchmarks become obsolete.

 

[Lew Zealand]

Really now? And please explain to me, if Galaxy throttles X app lets say, 50%, Y app at 70% and Z app at 30%, how much throttling should be applied to Geekbench in order for it to accurately demonstrate performance?

You can't. Hence the ban.

 

[Uncle Al]

Shame on Samsung ... they know better. Looks Like I made a good move away from their products, we can only guess what else they have been up to.

 

[OneSpeed]

Shame on Samsung ... they know better. Looks Like I made a good move away from their products, we can only guess what else they have been up to.

Why do benchmarks ever matter to you? Do you use a benchmark app on your phone to see how it performs? Can't you rely on real-time usage to determine if a phone is right for you? How do you know the phone you are using is not throttled back like all the others?

 

[Sausagemeat]

I would be surprised if this is limited only to Samsung. Phone manufacturers are not exactly pillars of integrity.

 

[psycros]

Why do benchmarks ever matter to you? Do you use a benchmark app on your phone to see how it performs? Can't you rely on real-time usage to determine if a phone is right for you? How do you know the phone you are using is not throttled back like all the others?

So you don't come to tech sites to actually get news about this kind of thing but just to blindly comment? Wow.

 

[OneSpeed]

So you don't come to tech sites to actually get news about this kind of thing but just to blindly comment? Wow.

I read the article, but did you? How did it come as a blind comment to you? How is it shameful for Samsung to throttle apps? Oh, I didn't see your earlier comment on the article...do you come on to this website to troll?

 

[DukeJukem]

If the normal user experience is for the phone to throttle apps, then Geekbench and other benchmarks should also have been throttled too, so as to paint the correct picture of how the phone performs. But Samsung didn't do this. It cheated. The fix is for Geekbench to also be throttled (just like the 10,000 other apps), and use the throttled score to compare the normal Galaxy performance to that of other phones.

Therefore, Samsung's behavior is manipulation because the truth is hidden. Samsung hides the fact that 10,000 apps run throttled on the Galaxy except for the flashy benchmark Apps (like Geekbench) used in reviews. Why are the benchmark apps (used by reviewers and article publishers) not included in the throttled list? Using Geekbench and other non-throttled benchmarks to compare scores of Galaxy phones to that of other phones would give users a wrong impression of the Galaxy's actual (I.e., throttled) performance in 10,000 other apps.

The hidden assumption behind comparing Geekbench scores from different phones is that the phones will also run other apps unconstrained (and only constrain apps due to heat/power management). But that assumption doesn't hold here, as shown by the evidence. The Geekbench scores have no connection to actual user experience! The reality is 10,000 apps are throttled all the time!! Samsung selectively chose not to throttle the apps used by reviewers and article publishers to paint the phone in a good light and that is manipulation and should be called out!

Idk man. If you're on a pc running cinebench you don't leave an abundance of programs open. You close them all and run the benchmark. Samsung throttling apps (which is less harsh than actually closing them) that aren't in the foreground to get a proper geekbench score isn't manipulation, it's common sense because that's how you properly run a benchmark by closing everything but the benchmarking program itself.

This also implies that if I purposely go into my phones app list and force close everything then run geekbench I'm manipulating the benchmark. You close everything possible before running a benchmark. It's common sense, not manipulation.

 

[Nobina]

Don't care much about benchmarks but Samsung in Europe uses their Exynos SoCs which are way shittier compared to Snapdragon ones. They should drop the price for Exynos models.

 

[nismo91]

Samsung is laughing because no matter what they do people still buy their S-series phones.
1. they advertise fast soc speed with snapdragon and shipped exynos for half of the world instead
2. just like VW they resorted to manipulation to achieve different numbers in benchmarks
3. they designed S series to be difficult-to-repair, starting from S6 and replacement parts are limited
4. they mocked apple but followed them by removing earbuds and charger from the bundle
5. price increment every year even for the same old 128GB configuration
6. for S22 they advertise 45W charging speed but you can't tell the difference from 25W charger.

sounds like a company heading for trillion dollar valuation or am I wrong lol

I just hope the others can stand up to samsung for some time. I don't want to have only 2 choices when buying phones like when I'm shopping for gpu.

 

[Watzupken]

If android could manage background processes better this wouldn't be an issue. I don't see how throttling other apps is considered manipulating a geekbench benchmark. If you're not using an app it should be throttled whilest in the background in order to prioritize foreground processes. I don't see the issue here.

While I feel that the purpose of benchmarks is to see what is the full potential of the hardware, I also believe that users should also be able to experience the same performance base on their preference. The problem here is what you see on benchmark, is not quite what you are going to get in your day to day use case. For example, when you buy a phone and planning to game on it, the system actually throttles performance such that it is not representative of what you think you will be getting. For most light users, you may not feel that the phone is slow only because modern SOCs are actually quite fast. But when you are running CPU, and/or, GPU intensive apps or games, you will surely feel or notice the difference. A lot of reviews are based on running benchmarks to see how well the device runs, and many will make their decision to buy based on positive reviews. So if Samsung is skewing the performance to make their products appear better than in real life, then that is misrepresentation to me.
Anyway, I am never a fan of running benchmarks. It is a waste of power and because most of the tests are too short, you are unlikely to get a meaningful or truthful performance results since most drivers are optimised to bring the best performance only in benchmarks.
It is a rough start of the year for Samsung where they are caught throttling performance and at the same time, getting hacked. I do hope they can sort things out soon.

 

[rmcrys]

If the normal user experience is for the phone to throttle apps, then Geekbench and other benchmarks should also have been throttled too, so as to paint the correct picture of how the phone performs. But Samsung didn't do this. It cheated.

What you are saying makes no sense. This optimization has to do with allocating resources, NOT overclocking. If you have a VPN and anti-virus app that, instead of using a low priority task and the code is optimized, it is very badly optimized and eats battery life, Google/Android makes nothing against it. (Apple does) So Samsung took care of it. Is that "cheating"? No.

Of course benchmark apps have to run unlimited to access full hardware performance. But I have in truth to say that Samsung should add that optimization feature where someone easily sees it and can disable it. I have high end Samsung and I only thought it was for games.

Nevertheless all apps on my S8, S9+, S20 ultra and S22 Ultra run full time well. Kudos for Samsung and Android world.

 

[Theinsanegamer]

What you are saying makes no sense. This optimization has to do with allocating resources, NOT overclocking. If you have a VPN and anti-virus app that, instead of using a low priority task and the code is optimized, it is very badly optimized and eats battery life, Google/Android makes nothing against it. (Apple does) So Samsung took care of it. Is that "cheating"? No.

Of course benchmark apps have to run unlimited to access full hardware performance. But I have in truth to say that Samsung should add that optimization feature where someone easily sees it and can disable it. I have high end Samsung and I only thought it was for games.

Nevertheless all apps on my S8, S9+, S20 ultra and S22 Ultra run full time well. Kudos for Samsung and Android world.

He never said anything about overclocking. Slow down and read his comment again.

If all other apps run slow, then geekbench should be throttled to the same level to avoid manipulation.

 

[HapticFeedback]

Why do benchmarks ever matter to you? Do you use a benchmark app on your phone to see how it performs? Can't you rely on real-time usage to determine if a phone is right for you? How do you know the phone you are using is not throttled back like all the others?

Two things. I agree that real-time usage is a great way to measure how good a phone works for you but that requires using it in the real world. Which means having it. I myself prefer to decide if a phone is worth owning before I buy it so I find benchmarks helpful. Secondly I am not against throttling but I am against dishonesty and excusing it because the other phone makers MIGHT also be doing something shady is a pretty sad defense.

 

[HapticFeedback]

I have never relied on benchmarks when I buy that next phone. I've been happy with the "real-time" performance of the Galaxy phones. Who f'ing buys a phone to run a throttling app??? If the phone performs well running your apps, these benchmarks become obsolete.

First congrats for being able to buy something before you need two find out if it will work for you but as to you questions. No-one buys a phone to run a throttling app that's why Samsung didn't tell anyone that's what it was going. It promised one level of performance and then gave people another and hoped they would not notice. If your ok with bait and switch then I guess that's ok but your personal opinions on benchmarks doesn't make them any less helpful to me or obsolete.

 

[Aceseven]

I didnt know people took cellphone benchmarks seriously.......

 

[Lew Zealand]

Some people still need to play the "mine is not smaller than yours" game wherever they can. And now, it's portable!

 

[rmcrys]

If all other apps run slow, then geekbench should be throttled to the same level to avoid manipulation.

You clearly never used an high end Samsung.