Google Chrome in a runaway lead for browser performance supremacy

If Apple's Safari is going to make any kind of a challenge for best performing Windows-based Web browser moving into next year, it needs to be now. In Betanews' most extensive testing to date, involving tests that by anyone's guess should not have given it any special advantages, the latest stable edition of Google Chrome runs away with a three point lead over the latest stable Safari -- a lead that now grows by one-half point with each point release.

Chrome now posts test scores in certain heats of our revised CRPI 2.2 test battery that are virtually obscene -- so far ahead of competition that we have to validate our results on various machines to make sure we're not generating false results. For example: On the control flow element of the SunSpider test, both Chrome 3 and the dev build of Chrome 4 post record low time scores of 2.6 ms. This is an element that tests the JavaScript interpreter's capability to keep track of nested loops and its location in a twisted program. By comparison, the latest public Firefox 3.6 Beta 1 -- released late yesterday afternoon after a flurry of delays totaling over one month -- posts a score of 38.2 ms in that category.

Mozilla's team has been making efforts to better Firefox' control flow scores, evidently knowing how much they influence test results like ours. The evidence comes from the latest daily builds of Firefox 3.7 Alpha 1, on the "Minefield" track, whose control flow scores recently quantum-leaped down to 8 ms. That's almost a 5x improvement, but it will need another 3x blast to catch up with Chrome. The latest stable Apple Safari scores 3.4 ms on this element.

Another example: A factorial is the result of multiplying together all positive integers that are less than or equal to a number, and a new element of our testing battery includes a classic algorithm for obtaining the set of all factorials. On this heat, the higher number is better since the objective is to obtain as many factorials as possible over a set period of time, so the score is a relative index. While Firefox 3.6 Beta 1 scores a 17 on this heat, Chrome 3 scores a 150.4, and the latest dev build of Chrome 4 scores 164.5.

There are certain things that Chrome does where the score differences are factors of 10, where one might get the impression that Google is improving Chrome just to score better with Betanews (the company has been expressing its interest to us directly in recent days) or more likely, with its own internal test suite. Indeed, the company's V8 benchmark suite would have users thinking the browser is hundreds of times more capable than its competition -- a claim for which we just don't see the practical evidence just yet, which is why V8 isn't part of our tests.

That's why we've made an effort to pack our CRPI test suite with examinations of a multitude of real-world characteristics in various categories, now with 69 different "heats" in ten separate batteries, plus a multitude of derivative scores (e.g., average of 50 iterations, consistency between the fastest and slowest run, etc.) for each browser.

Once we include all these different elements, we get a much more practical and believable result. There are many reasons why a person chooses a favorite Web browser, with JavaScript performance being just one of them. But that factor is becoming more important now with the onset of applications delivered to you from the Web rather than your hard drive. So with regard to that factor alone, Betanews can say that Google Chrome delivers almost twice the performance (not quite 2x) of the latest stable Mozilla Firefox builds, with Apple Safari probably pulling close, and Opera being left in the dust until it comes time to be thinking about Opera 11...if not Opera 12.

"Probably" pulling close? What, Betanews can't do better than that with respect to Safari? Yea, unfortunately there's still trouble with that: Apple's test builds of Safari come by way of grafts of its daily WebKit engine onto the existing 531.9.1 browser. Usually, after applying one of these grafts, the updated Safari displays better if not superior rendering performance than even Chrome. But its ability to serve as a full-scale browser for other tests vanishes. However, over the last week while Betanews has been trying to resolve this problem, it actually only got worse: Windows testers reported through Apple's forums that not even the grafting mechanism was working.

We validated those claims, discovering that the manifest which the replacement executables were being shipped with (the embedded XML files that point to proper COM components in the System Registry) were pointing instead to incompatible versions of Windows Common Controls, versions that may have worked back in the 1990s. Apple is apparently already aware of this, but as is the company's policy with regard to any kind of problem with its software or hardware, will not publicly comment.

Next: The latest test scores broken down...