Google kills off Octane JavaScript benchmark due to 'diminishing returns and over-optimization'

Google has announced that it is to retire the Octane JavaScript benchmark. The company says that while the benchmark was useful when it was introduced back in 2012, "over-optimization" by developers means that it is far less meaningful.

Google goes as far as saying that developers were essentially cheating the system. It says that compiler optimizations needed to achieve high benchmark scores have become common and, in the real world, these optimizations translate into only very small improvements in webpage performance.

In fact, in some instances it was found that tactics used to boost benchmark performance actually had a detrimental effect on real-world performance. Developers exploited known bugs in Octane to achieve higher scores than were warranted, and Google believes the time has now come to retire the system completely.

Placing the blame at the door of those who have been playing the system, Google says:

In the first few years after its release, Octane provided a unique value to the JavaScript VM ecosystem. It allowed engines, including V8, to optimize their performance for a class of applications that stressed peak performance. These CPU-intensive workloads were initially underserviced by VM implementations. Octane helped engine developers deliver optimizations that allowed computationally-heavy applications to reach speeds that made JavaScript a viable alternative to C++ or Java. In addition, Octane drove improvements in garbage collection which helped web browsers avoid long or unpredictable pauses.

By 2015, however, most JavaScript implementations had implemented the compiler optimizations needed to achieve high scores on Octane. Striving for even higher benchmark scores on Octane translated into increasingly-marginal improvements in the performance of real web pages. Investigations into the execution profile of running Octane versus loading common websites (such as Facebook, Twitter, or Wikipedia) revealed that the benchmark doesn’t exercise V8’s parser or the browser loading stack the way real-world code does. Moreover, the style of Octane’s JavaScript doesn’t match the idioms and patterns employed by most modern frameworks and libraries (not to mention transpiled code or newer ES2015+ language features). This means that using Octane to measure V8 performance didn’t capture important use cases for the modern web, such as loading frameworks quickly, supporting large applications with new patterns of state management, or ensuring that ES2015+ features are as fast as their ES5 equivalents.

The company recognizes that Octane had its place -- it led to genuine improvements in JavaScript performance -- but the time has come for a new method of benchmarking. Google has not provided details of its plans yet, but there will be a greater focus on real-world performance rather than just raw numbers.