OnePlus 5 cheats in benchmarks for a performance boost

At the start of the year it was discovered that the OnePlus 3T was cheating in benchmarks, artificially boosting performance when testing tools were detected. The OnePlus 5 was only officially launched yesterday but the same accusation is already being leveled at the latest version of the flagship killer.

As with the OnePlus 3T, the benchmark trickery of the OnePlus 5 was discovered in tests carried out by XDA Developers. The site says that its review unit "manipulates benchmarks to maximize scores" and issues a stark warning in an article entitled: "Do NOT Trust OnePlus 5 Benchmarks in Reviews -- How our Review Unit is Grossly Cheating at Benchmarks."

Just as before, XDA Developers found that OxygenOS -- the OnePlus take on Android -- was gaming benchmarks. While with the OnePlus 3T things were somewhat subtle, this time around it appears to be more blatant. It also affects a large number of well-known benchmarking tools: AnTuTu, Androbench, Geekbench 4, GFXBench, Quadrant, Nenamark 2, and Vellamo.

The site explains how the cheating that has been detected in the OnePlus 5 differs from the technique used in the OnePlus 3T:

Last time around, OnePlus introduced changes to the behavior of their ROM whenever it detected a benchmark application was opened. Such application names were explicitly listed by their package IDs within the ROM in a manifest that specified the targets. Then, the ROM would alter the frequency in relation to an adjusted CPU load -- our tools showed CPU load would drop to 0 percent regardless of obvious activity within the application, and the CPU would see a near-minimum frequency of 1.29GHz in the big cores and 0.98GHz in the little cores. This minimum frequency reduced the effective frequency range, which in turn reduced the number of step frequencies; in benchmarks, this resulted in slightly lower variance and, as we showed, higher sustained performance as the higher minimum frequency could not be overridden by thermal throttling. In short, cheating behavior was clear and demonstrable by both looking at score variance, and by monitoring CPU frequencies throughout the benchmark, which showed a frequency floor that -- for the most part -- allowed the device to consistently score closer to its full potential.

The OnePlus 5, on the other hand, is an entirely different beast -- it resorts to the kind of obvious, calculated cheating mechanisms we saw in flagships in the early days of Android, an approach that is clearly intended to maximize scores in the most misleading fashion. While there are no governor switches when a user enters a benchmark (at least, we can’t seem to see that’s the case), the minimum frequency of the little cluster jumps to the maximum frequency as seen under performance governors. All little cores are affected and kept at 1.9GHz, and it is through this cheat that OnePlus achieves some of the highest GeekBench 4 scores of a Snapdragon 835 to date -- and likely the highest attainable given its no-compromise configuration with its specific configuration. Scores certainly higher than those obtained by similar devices and Qualcomm’s own MSM8998 test device.

It was possible to see that the OnePlus 5 was actively looking for known benchmarking tools by working with a "hidden" version of GeekBench 4. When tested with this version of the tool, benchmark scores were significantly lower than when using the version downloaded directly from Google Play.

Carl Pei, co-founder of OnePlus, took to Reddit last night to offer a defense:

We have made it so that when running benchmark apps, the phone performs the same as when running resource intensive apps such as 3D games. We also fully activate our chipset in other parts of OxygenOS, for instance when launching apps to make the launch experience faster and smoother.

We are not making it easier for the chipset to perform, for instance by changing to a lower resolution when detecting a benchmark app. We are not changing the performance of our chipset, for instance by overclocking it.

When users run benchmark apps, which I agree aren't a useful proxy for real life performance, we believe that they want to see the full potential of their device without interference from tampering. That's what we've unlocked.

Every OEM has proprietary performance profiles for their devices, I appreciate that we have a tech enthusiastic following, but feel free to have a look around. :)

Needless to say, Redditors were less than convinced. The full benchmark analysis on XDA Developers is well worth a read, and we recommend you check it out.

In the meantime, does this change your mind about whether you'll buy a OnePlus 5 or not?