Huawei has been cheating in benchmarks
UL -- the company behind benchmarking software 3DMark -- has delisted a number of Huawei phones after the Chinese handset manufacturer was found to be cheating in benchmarks.
As has been the case with handsets from some other manufacturers, the Huawei P20, P20 Pro, Nova 3 and Honor Play were found to increase performance when benchmarking software is running. Interestingly, Huawei has not denied the accusations leveled at it.
The benchmark cheating was first brought to light earlier this week by AnandTech. The site noticed irregularities in benchmark results, saying: "it seems quite blatant as to what Huawei and seemingly Honor had been doing: the newer devices come with a benchmark detection mechanism that enables a much higher power limit for the SoC with far more generous thermal headroom. Ultimately, on certain whitelisted applications, the device performs super high compared to what a user might expect from other similar non-whitelisted titles. This consumes power, pushes the efficiency of the unit down, and reduces battery life".
UL looked into things and found that using a "private" version of 3DMark gave very different benchmark results to those in the version of the app found in Google Play. The company says:
We found that the scores from the public 3DMark app were up to 47 percent higher than the scores from the private app, even though the tests are identical.
With the public 3DMark app, these devices appear to use a hidden "Performance Mode" that overrides the devices' usual power profile -- see the AnandTech article for more details.
The difference in scores tells us that the devices are simply recognizing the 3DMark app by name rather than adapting to the type of work in the test.
As this sort of activity is against UL's rules, the firm says, "we have decided to delist the affected models and remove them from our performance rankings".
Huawei issued a statement to Android Authority in which it said:
Huawei always prioritizes the user experience rather than pursuing high benchmark scores -- especially since there isn't a direct connection between smartphone benchmarks and user experiences. Huawei smartphones use advanced technologies such as AI to optimize the performance of hardware, including the CPU, GPU and NPU.
When someone launches a photography app or plays a graphically-intensive game, Huawei's intelligent software creates a smooth and stable user experience by applying the full capabilities of the hardware, while simultaneously managing the device's temperature and power efficiency. For applications that aren't as power intensive like browsing the web, it will only allocate the resources necessary to deliver the performance that's needed.
In normal benchmarking scenarios, once Huawei's software recognizes a benchmarking application, it intelligently adapts to "Performance Mode" and delivers optimum performance. Huawei is planning to provide users with access to "Performance Mode" so they can use the maximum power of their device when they need to.
Huawei -- as the industry leader -- is willing to work with partners to find the best benchmarking standards that can accurately evaluate the user experience.
Image credit: Grzegorz Czapski / Shutterstock