OnePlus and Meizu caught red-handed cheating on benchmarks; OnePlus promises to stop

A group of XDA developers have confirmed that OnePlus and Meizu are cheating on benchmarks. OnePlus has even admitted as much and promises to not do it in the future.

In 2013, it was discovered that just about every smartphone manufacturer was cheating on benchmark. Notably, Samsung and HTC were called out for artificially boosting performance in benchmarking apps.

Given the backlash from the community, we assumed that manufacturers valued their own community enough to avoid practices like this. It looks like OnePlus and Meizu are unconcerned.

When testing the OnePlus 3T, XDA noted that the Snapdragon 821 chipset used was forced to run at a higher base clock whenever a benchmark app was detected.

Before we dive into the details of the cheating though, a small explainer on how mobile CPUs work is in order.

How does a mobile CPU work?

Most mobile CPUs today use the so-called big.LITTLE architecture. In this design, a set of high-performance cores (Big) are paired with a set of low-performance ones (Little). The former are battery intensive and the latter are not.

The idea is that tasks requiring low power — like background applications, playing music, etc. — are handled by the little cores. Intensive tasks like gaming are handled by the big cores. This combination of cores helps in power and thermal management.

The CPU on an SoC will also increase or decrease its frequency depending on the application being run. This is called scaling. So, a CPU might hit its maximum speed when opening an app and then settle to idling speed after an app is open. This happens very fast.

READ  Experts argue that concessions should be given to all manufacturers and not just Apple

The chipset uses speed governors to determine how this is handled. Smartphone manufacturers are given the freedom to tweak these governors from within the OS — Android, in this case.

What did OnePlus do?

Coming back to OnePlus, the OnePlus 3T uses a Snapdragon 821 chipset, which uses two big cores and two little cores.

Normally when an app is opened, the cores are supposed to scale up to high speed (over 1.29 GHz) and then fall back to an idle state of 0.31 GHz when there’s no load.

XDA noted that the OnePlus 3T would maintain the little cores at 0.98 GHz and the big cores at 1.29 GHz in certain applications like benchmarks. Assuming that OnePlus was cheating on the benchmarks, they approached Primate Labs, the makers of the popular GeekBench suite of benchmarks, for help.

Primate Labs discovered that the OnePlus 3T’s OS was identifying apps by name and modifying the speed governors accordingly. To overcome this, Primate Labs built a benchmark with a different name. When tested on the OnePlus 3T, it was immediately noted that the chipset was performing as it normally should.

When the tests were run again, XDA noted that the difference in performance was minimal. In fact, the device ran hotter for no notable gain in performance.

XDA also discovered that this ‘cheating’ was a ‘feature’ of Hydrogen OS used by OnePlus. Apparently, the OS identifies apps and games by name and forces the CPU into a higher performance state when it does.

One might think that a targeted boost in performance is acceptable, but we have to remember that this is a hack. OnePlus isn’t optimising the CPU to handle varying loads. A new app, for example, will not benefit from this. If OnePlus stops supporting the OnePlus 3T, you lose any performance “optimisation” for newer apps. The resultant performance is inconsistent for the user.

READ  Apple is building a custom ARM chip to handle low power tasks on future MacBooks: Report

When approached by XDA, OnePlus apparently “promised to stop targeting benchmark apps” and that future builds will cease to do so.

It’s good that OnePlus is at least admitting this. But I don’t think they had a choice. Any other stance would have just generated more backlash from the community.

Meizu: Cheater extraordinaire

On testing other phones from Xiaomi, HTC, Sony and the like, XDA was happy to note that there were no performance inconsistencies. On testing the Meizu Pro 6 however, XDAdiscovered something very disturbing.

The Meizu Pro 6 uses a 10-core Helio X25 chipset with a combination of four low performance cores, four medium performance cores and two high performance cores. Think of it as an extension to the big.LITTLE design, and in principal, uses the same mechanism for scaling performance.

XDA was surprised to learn that the Pro 6 switches off the big cores most of the time. They only come on in certain apps and when the phone is explicitly placed in ‘Performance Mode’.

The performance disparity was so great that scores fell by over 30 percent when Primate Labs’ secret benchmark was used. XDA believes that Meizu is also targeting apps by name.

Meizu is effectively selling you what it claims is a high-performance phone, but, for whatever reason, is throttling it down to the level of a mid-range phone.