MediaTek caught cheating on benchmarks, says everyone is doing it

[midian182]

A hot potato: Here’s a story you’ve heard before: a company has been caught cheating on mobile benchmark software. Rather than being limited to phones from a particular firm, the culprit in this instance is chipset maker MediaTek, meaning multiple handsets from several vendors are engaging in the practice. Its excuse? Everyone does it.

Anandtech discovered the cheating after benchmarking the MediaTek Dimensity 1000-powered Oppo Reno3 (Chinese version), and the European version of the Oppo Reno3 Pro, which is powered by the older, lower-end Helio P95.

Strangely, the Oppo Reno3 Pro’s Cortex-A75 with its two cores outperformed the Reno3 and its four-core Cortex-A77 in PCMark.

To confirm the cheating, the publication used an altered version of PCMark that can’t be identified as a benchmarking tool by systems. Not too surprisingly, the P95 scored lower: a 30 percent drop in the overall score, with up to a 75 percent difference in specific workloads.

In the Chinese version of the Reno3 Pro, which uses the Snapdragon 765G, no cheating was observed when running the benchmarks.

Examining the P95-powered Reno3 Pro’s firmware revealed a file with a list of popular applications and benchmarks with various power marks and management tweaks applied to them. This ‘sports mode’ boosts the memory controller and scheduler when the apps are running, improving performance.

Several MediaTek-powered devices, not just those from OPPO, were found to exhibit the same behavior, including the Sony Xperia XA1 that launched back in 2016, indicating this has been going on for some time. Check out the full list below.

Image credit: Anandtech

MediaTek responded to the article with a statement that claimed boosting the performance in benchmark tests shows the SoC’s full capabilities, and that the chipsets scale in performance for certain resource-intensive apps. It also added that some brands give consumers the option to enable “sports mode” or “monster mode” in specific regions, while the modes are switched off in other locations.

MediaTek emphasized that manufacturers have the final say on what optimizing they want to include in APKs, and that “benchmarking optimizations” are an industry standard. When Huawei was caught manipulating benchmark scores, it also used the “others do it” excuse.

Sadly, MediaTek and Huawei are right: most manufacturers are guilty of this shady practice. OnePlus and Meizu did the same thing with the OnePlus 3 and OnePlus 5, and Samsung had to pay $13.4 million in damages after source code was discovered in the Galaxy S4 that detected when certain benchmarking software was running, at which point the GPU frequency would jump around 11 percent from 480MHz to 533MHz. The Korean firm settled the subsequent lawsuit, which meant owners got a massive $10 each.

Main image credit: g0d4ather via Shutterstock

Permalink to story.

https://www.techspot.com/news/84761-mediatek-caught-cheating-benchmarks-everyone-does.html

 

[Lionvibez]

Everyone else does it so we should too is not a good excuse MediaTek!!

 

[lipe123]

Everyone else does it so we should too is not a good excuse MediaTek!!

Yeah but if every other person on the starting line next to you is doping and you're not how are you going to appear competitive to consumers?
I don't know how it works for the general public but the first thing I look at when looking for a new phone is how it performs on benchmarks to get an idea of how long it might be a useful device.

 

[Lionvibez]

Yeah but if every other person on the starting line next to you is doping and you're not how are you going to appear competitive to consumers?
I don't know how it works for the general public but the first thing I look at when looking for a new phone is how it performs on benchmarks to get an idea of how long it might be a useful device.

And now that you know the benchmarks scores are "enhanced" and not legit will you still use them for your purchasing decisions.

When I look at a phone purchases performance is only part of the criteria that I look at. And these numbers always have an * next to them because I'm aware of what these guys do.

 

[mojoxer]

I know this seems bad, but isn't this called "game-specific driver optimizations" when Nvidia and AMD do it in their GPU drivers?

 

[daimag]

In open market, the majority gets to decide what is right or wrong. Consumers can only vote thru their purchase.

 

[Lionvibez]

In open market, the majority gets to decide what is right or wrong. Consumers can only vote thru their purchase.

True but most consumers are tech illiterate. So those are not the best people to decide what is right or wrong they just follow marketing and buy. Or buy what there friends use. You are not going to get objective decision making on products from most of them.

 

[brucek]

Of course I want my devices to be smart about when they optimize for performance vs. battery life, and of course I want them to make reasonable choices about what apps should have higher priorities over others.

Assuming the same 'cheating' configs are applied to other applications that justify performance over battery tweaks, the only issue I see here is lack of transparency. And even there, I understand that you are not going to provide your average consumer with a lengthy user manual and a complex system options app covering a ton of options that only a hardware enthusiast would begin to understand.

To me the easy happy balance is to just provide that documentation and control app to hardware reviewers as part of the press kit, at which point they can figure out and explain the testing methodology that makes sense to their readers.

 

[lipe123]

And now that you know the benchmarks scores are "enhanced" and not legit will you still use them for your purchasing decisions.

When I look at a phone purchases performance is only part of the criteria that I look at. And these numbers always have an * next to them because I'm aware of what these guys do.

Of course you can, it's not an artificial number, they are just temp overclocking the hardware. So it's still a legit thing the platform is capable of.

Also if everyone is doing it then just take that into account, don't just look at a single benchmark.

 

[Lionvibez]

To me the easy happy balance is to just provide that documentation and control app to hardware reviewers as part of the press kit, at which point they can figure out and explain the testing methodology that makes sense to their readers.

I think this is a good idea.

 

[ratWorks]

I know this seems bad, but isn't this called "game-specific driver optimizations" when Nvidia and AMD do it in their GPU drivers?

No, it's totally different. Game driver enhancements are about improving peoples experience when playing games, they're not about convincing users that the games are somehow more enjoyable then they actually are. Often they're about preventing crashes, and or enabling some rendering enhancements.

 

[BigBoomBoom]

I know this seems bad, but isn't this called "game-specific driver optimizations" when Nvidia and AMD do it in their GPU drivers?

I don't think any driver from NVIDIA and AMD suddenly overclock their GPU by 11%.

 

[EClyde]

Almost everyone does. Truth

 

[Dustyn]

Everyone else does it so we should too is not a good excuse MediaTek!!

Sounds like the company is run by a bunch of children, "Come on guy!? Everybody else is doing it! PEER PRESSURE!!!"

 

[lipe123]

Sounds like the company is run by a bunch of children, "Come on guy!? Everybody else is doing it! PEER PRESSURE!!!"

No it doesn't. It's how business works.

Take restaurants for example, the pictures of the food on their menus are absolutely not edible. They spray it with varnish and oils and WD40 for all we know to give it that shiney "fresh" look.
So if you were a restaurant would you take pictures of your food as-is without all the extra trickery? Of course not.

 

[Lionvibez]

When it comes to food the quality of your food will speaks for itself.

Some of the best restaurants I went to I never looked at the pictures of the food before I went in.

Word of mouth was good enough.

 

[Dustyn]

So if you were a restaurant would you take pictures of your food as-is without all the extra trickery? Of course not.

I can tell you that there are absolutely good quality restaurants that do NOT do this.
What restaurants are you going to??

 

[captaincranky]

...[ ]...Take restaurants for example, the pictures of the food on their menus are absolutely not edible. They spray it with varnish and oils and WD40 for all we know to give it that shiney "fresh" look ...[ ]....

While obviously chefs want the best representation of their craft in photos of their product, the "doping" of the food with cooking oils is very often a necessity.

For example, suppose the food has to be transported to a studio? It sits around, and loses its "just cooked look".

As a familiar example, the pizza you asked to be delivered, never looks as good as the moment it came out of the oven. Now picture leaving that same pizza sitting on you table for a couple of hours, it wouldn't have that "just cooked look", now would it?

So what you typify as "cheating", is in reality, merely restoring the food's look to that time it came out of the kitchen.

There's a lot of work and manipulation that goes into advertising photos, and granted that they are idealized. But, if you know what you're doing with a camera, lights, settings, backgrounds, and Photoshop, you could make your own leftovers look like they came out of a culinary magazine