Browser blowout: Which is fastest, most standards compliant [benchmarks]?

The browser wars are back, not least due to hectic and ever shorter release schedules, just about every other week there seems to be some new build promising that it's the faster way to get around online.

But which browser really delivers the best performance? We decided to pit the leading contenders against each other in a gruelling set of benchmarks to figure out the truth for ourselves.

The field was made up of the latest versions of the five leading browsers: Chrome 12.0.742.122, Firefox 5.0.1, IE 9.0.8112.16421, Opera 11.50 and Safari 5.0.5. (Those speedy release schedules meant this selection became outdated even before the tests had completed, but more on that later.)

We chose 16 benchmarks to cover every aspect of browser performance: startup time, page load time, rendering speeds, JavaScript and DOM operations, memory use, standards compliance and more.

A 32-bit Windows PC was chosen as our test rig (4GB RAM, Intel Core i7-920 CPU, nVIDIA GeForce GTX 285 graphics card, 1TB Samsung HD103SJ hard drive), and carefully prepared: power settings disabled, background programs uninstalled, everything that could interrupt or distort our tests turned off.

And then the tests began.

JavaScript

SunSpider 0.9.1 is a quick way to assess JavaScript performance, and is probably one of the most regularly-quoted browser benchmarks around. It also gave IE an initial first place, while Firefox, Opera and Chrome trailed behind, and Safari was well out of contention. But is this entirely accurate?

It's perhaps no surprise that Google's own JavaScript benchmark, V8, switched positions by giving Chrome first place and placed IE last. But it's not alone: Dromaeo's JavaScript tests and the older Kraken 1.1 also placed Chrome in front, while giving mixed results for everybody else.

To achieve an overall score, then, we assigned each browser 5 points for a first place, 4 for a second, down to 1 for coming fifth. Despite its SunSpider failings, this saw Chrome take an early lead with 17 out of a possible 20, although Firefox (15), IE (12) and Opera (11) were still within easy reach. Safari, however, was off to a poor start with a lethargic score of 5.

Rendering and DOM operations

We followed up the JavaScript tests with a more mixed set of benchmarks.

The Asteroids HTML5 Canvas, GUIMark2 Flash and GUIMark2 HTML5 Vector tests assessed rendering performance, for instance. Dromaeo's DOM Core tests looked at common Document Object Model operations, vital for ensuring the speedy generation and display of dynamic pages. And Futuremark's Peacekeeper combines a number of tests to give perhaps the most rounded benchmark here.

The results were more mixed this time around: Opera scored a marginal victory over Chrome in the PeaceKeeper test, but Google's browser hit back with a win on GUIMark2?s HTML Vector benchmark, and both Chrome and Opera finished tied for first place with 18 out of 25. Safari recovered from its JavaScript issues with two first places (including the important Dromaeo DOM Core), while IE and Firefox trailed behind.

Standards compliance

Online performance isn't just about raw speed, of course. It also requires that your browser supports the technology that a particular web page is trying to use, and so we quickly assessed this by putting out browser through a couple of standards compliance tests.

The Web Standards Project's Acid3 test, for example, quickly checks a number of DOM, JavaScript and other features in the current browser, displaying the results as a score out of 100. Opera, Safari and Chrome all scored full marks here, while Firefox (97) and IE (95) followed behind.

Acid3 has received a lot of attention, but is a little out of date now, so we also used the more interesting HTML5Test to check how well our browsers supported HTML5 technologies. And the result was an easy win for Chrome, while Firefox and Opera delivered broadly similar results, Safari was fourth and IE fifth.

Real world tests

Synthetic benchmarks are easy to use, but it's difficult to be sure how well they relate to real world experiences. And so we completed our tests by measuring how well each test browser performed some common tasks.

How long does it take to start, for instance? We measured the time it took each browser window to appear from launch: IE was the winner, Chrome close behind, while Opera took a sluggish last place.

What about the time it takes to load popular pages? We tested this with Facebook: Chrome took the crown here, only just pushing Safari into second place, while on this occasion IE and Firefox bring up the rear.

We assessed CPU usage by looking at the processor time required by a browser to run the full Peacekeeper benchmark, and this turned out to be the first win for Firefox, while Chrome was second.

And we completed our roundup by looking at the private memory consumed by each browser, first after running Peacekeeper in a single tab, and second after opening ten popular sites (Google, Facebook, YouTube, Yahoo!, Blogger, Wikipedia, Twitter, Amazon, eBay, Bing), each in their own tab. Safari and Opera were easy winners here, with Chrome unusually trailing behind in last place.

Analysis

Trying to analyse the results of 16 benchmarks, each of widely varying importance, for 5 different browsers, could have been tricky. But fortunately, it really wasn't. Google Chrome romped home, scoring 7 first places, and another 5 seconds, taking an easy victory with a triumphant 60 out of a possible 80.

While Opera never really got close, a set of mid-range scores saw the program take a creditable second place, scoring 52. Firefox and Safari then followed behind, and Internet Explorer was last (although there really was little between the last three contenders).

The margin of victory was so substantial that it overcame any reservations about some of our individual benchmarks. Acid3 really doesn't matter much these days, for instance, and our CPU usage test, while interesting, can be tricky to interpret (low CPU usage is only good if the browser also returns speedy results elsewhere). But neither test made any material difference: we could omit them both and first and second place would remain the same.

Our final concern was the limited shelf life of any browser speed test. Safari 5.1 had already been released while the tests were taking place, for instance; IE 10 Platform Preview 2 was out, and betas of Chrome, Firefox and Safari were available. Would these prove to be game-changers?

To find out, we took Chrome 13.0.782.99 beta, Firefox 6.0b2, Internet Explorer 10 Platform Preview 2, Opera 12.00.1027 and Safari 5.1, and tried them out on Peacekeeper and SunSpider. But while we noticed a few small, incremental improvements -- IE and Safari's latest in particular proved a little faster than their predecessors -- there was nothing to change the overall picture.

Internet Explorer needs work when it comes to performance, and both Firefox and Safari are lagging behind. Opera excels in some important areas, particularly DOM operations, but is weak elsewhere. So if you're looking for great all-round speeds, then you really need Chrome: it's the performance pick of the current browser crop.