OnePlus 3T found to be cheating in benchmarks

Benchmarks are an important metric by which the performance of a given device can be measured. They provide a reliable means of determining whether one laptop, computer, phone or other device is faster than another -- at least that's the idea.

Tests performed by XDA Developers strongly suggest that OnePlus is using underhanded techniques to make the OnePlus 3T's Qualcomm Snapdragon 821 perform better in Qualcomm Trepn and Snapdragon Performance Visualizer tests. Analysis of results points to the fact that OnePlus is targeting benchmark apps by name, such as GeekBench, and is artificially boosting device performance when they are found to be running.

If this story sounds familiar, you need to cast your mind back to 2013/14 when Samsung was among smartphone manufacturers thought to be interfering with benchmark results to make their devices seem better than they actually were. Around a year ago, there were also complaints from AMD that SYSmark benchmarks were biased in favor of Intel.

XDA Developers explains what it found when investigating the app opening speed of the Qualcomm Snapdragon 821:

When entering certain benchmarking apps, the OnePlus 3T’s cores would stay above 0.98 GHz for the little cores and 1.29 GHz for the big cores, even when the CPU load dropped to 0%. This is quite strange, as normally both sets of cores drop down to 0.31 GHz on the OnePlus 3T when there is no load. Upon first seeing this we were worried that OnePlus’ CPU scaling was simply set a bit strangely, however upon further testing we came to the conclusion that OnePlus must be targeting specific applications. Our hypothesis was that OnePlus was targeting these benchmarks by name, and was entering an alternate CPU scaling mode to pump up their benchmark scores. One of our main concerns was that OnePlus was possibly setting looser thermal restrictions in this mode in order avoid the problems they had with the OnePlus One, OnePlus X, and OnePlus 2, where the phones were handling the additional cores coming online for the multi-core section of Geekbench poorly, and occasionally throttling down substantially as a result.

Analysis of the OnePlus 3T by GeekBench creator Primate Labs showed that the phone's software was actively looking for the benchmark apps Geekbench, AnTuTu, Androbench, Quadrant, Vellamo and GFXBench. To test things out, a secret benchmarking tool, disguised as Bob's Mini Golf Putt, was created, and the results were disappointing to say the least:

Immediately upon opening the app, the difference was clear. The OnePlus 3T was idling at 0.31 GHz, the way it does in most apps, rather than at 1.29 GHz for the big cores and 0.98 GHz for the little cores like it does in the regular Geekbench app. OnePlus was makings it CPU governor more aggressive, resulting in a practical artificial clock speed floor in Geekbench that wasn’t there in the hidden Geekbench build. It wasn’t based on the CPU workload, but rather on the app’s package name, which the hidden build could fool. While the difference in individual runs was minimal, the thermal throttling relaxations shine in our sustained performance test.

OnePlus has responded by saying that this technique will be stopped. In a statement to XDA Developers, the company said:

In order to give users a better user experience in resource intensive apps and games, especially graphically intensive ones, we implemented certain mechanisms in the community and Nougat builds to trigger the processor to run more aggressively. The trigger process for benchmarking apps will not be present in upcoming OxygenOS builds on the OnePlus 3 and OnePlus 3T.

The Meizu Pro 6 was found to be using similar techniques to inflate benchmark results.

Photo Credit: Mihăiță Bamburic