Preliminary results: IE9 tech preview performs 7.8 times better than IE8

In the first series of comprehensive performance tests comparing Microsoft's Internet Explorer 9 technical preview, released yesterday, to stable Web browsers in current use today, Betanews confirmed superb speed gains by the IE9 chassis in specific categories. Not everything in the new IE9 was faster than IE8, but in the computational department, the development team's Chakra JavaScript engine shows much-needed gains.

In anticipation of IE9, Betanews has been developing a radically improved set of performance tests to complement (and, in a few categories, replace) those we've used in recent months. Our objective is to determine not just how much faster IE9 is, but how much better and more efficient it will be, in computing data, in rendering on-screen objects, and in adapting to varying workloads.

Betanews estimates that the IE9 chassis on Windows 7 offers 9.32 times better raw computational performance than IE8 on Windows 7, on the same machine. That's a welcome number due in large part to vastly improved scores in the widely respected SunSpider battery, as well as high scores in a new set of variable-workload computational tests produced by Betanews. Specifically on the SunSpider, the IE9 preview scored a 44.77 on Betanews' relative performance index, compared to 5.59 for IE8. Our index is based on cumulative relative performance in each category of the test battery, compared against the score posted by an old, slow Web browser: IE7 on Vista SP2. This means, yes, IE9 (thus far) offers almost 45 times the computational speed of IE7 on the older operating system -- easily the single largest surge we've seen between generations.

A recent dev build of Google Chrome 5 on Windows 7 scored a 69.83 on that same SunSpider index, followed closely by the first stable version of Opera 10.5 with 68.64.

As Microsoft embraces HTML 5, it's also managing to eke out some marginal speed gains in the rendering department, although it must be noted that the IE9 chassis is running in an almost feature-less window with very minimal overhead. As of now, the IE9 preview offers 23% better rendering performance (CSS, DHTML, support for the Canvas element in HTML 5) than IE8.

Looking for the good

What Microsoft did yesterday was give outside developers, for the first time, direct access to just the engine of its next-generation Web browser, long before the functionality and usability features are attached to it. The reason, the Internet Explorer 9 product team says, is to elicit real-world feedback so that the product can be fine-tuned.

That describes exactly what we intend to do. Over the last few weeks, Betanews has been compiling a suite of next-generation browser tests, having taken into account the feedback we've received from both our readers and browser manufacturers, Microsoft included. As rapidly as browsers have evolved in just the past year, it's become clear to us that when we compare brands, at one level, we truly are comparing apples to apple trees, or lawnmowers to bulldozers. When we concentrate on the prowess or power angle, with all the adrenaline-rushing metaphors and superlatives, we sometimes forget that sometimes, what the world really wants is an efficient lawnmower.

Last year, IE General Manager Dean Hachamovitch asked me to take a closer, fairer look at Internet Explorer. Specifically, he said that there were architectural efficiencies to be found in the product line, if only we took the time to look for them.

How I opted to respond to that challenge was to focus on one under-appreciated aspect of the Web browser that will become more important as its components are transported to six-core desktop systems on one end, and Snapdragon handsets and netbooks on the other: scalability. Specifically, I started exploring whether there was a way to effectively measure how well a browser handles increasing workloads, of ever higher orders of magnitude.

Mozilla helped to begin making scalability an issue with its introduction of the TraceMonkey JavaScript engine in Firefox. Tracers make problems that appear complex in coding simpler for their processing engines to execute, by pre-processing instructions ahead of time, converting and optimizing long sequences into easily digestible, assembly language-like instructions. Theoretically, the simpler and longer the sequences, the easier the digestive process should become.

So in this new era, it becomes necessary to test the efficiency of a browser's capability to digest those long sequences, to make harder problems simpler for themselves. This is the scalability element which will represent 30% of the score in our revised Relative Performance Index.

Yesterday, Dean Hachamovitch played down the importance of just-in-time compiling as a factor in improving browser efficiency, promoting instead the option of moving the interpreter to a background process. But doing that alone, as we're discovering now, may not effectively combat what has historically been IE's biggest problem as a Web apps platform: the ability to fall off a cliff (see: "stack overflow") when problems get especially difficult. On new tests involving sorting algorithms, for instance, where recursion easily becomes thousands of layers deep, IE8 can spin off into a coma.
So far, we have not seen the comatose effect in the IE9 tech preview, which could be the first sign of very good news for Web app developers.

What I was surprised to discover in crafting this new set of tests was that IE was not alone. Chrome can fall off a cliff too, just several orders of magnitude later (after 10 million iterations, for example, rather than 100,000). As the problem gets more and more complex, the gap between Chrome or Safari or the new Opera's performance and that of IE becomes wider and wider...and wider. And that's a problem because you could arbitrarily choose some point out in space, where Chrome is a thousand times faster than IE rather than, say, ten. Wait long enough and you might get 10,000.

And that, as IE proponents assert, would not be fair. It's actually the reason we chose not to include Google's V8 benchmark battery in our tests: because there does not appear to be a real-world correlation between the hundreds of times greater performance the V8 battery can report over IE, and the differences we see in ordinary use.

So the goal of our scalability tests is to recognize that smaller engines can still be efficient in what they do, even when they offer lesser horsepower. Maybe IE can't run a 10-million-iteration test. But the difference between its performance in 100,000 iterations and in 10,000 can be compared to Chrome's difference between 10 million iterations and 1 million. That factor may still be meaningful.

In the very first report of browsers' scalability compared to IE7 in Vista SP2, the IE9 tech preview in Windows 7 scored a 6.57 compared to IE8's score of 1.13. That means, we believe IE9's new "Chakra" interpreter offers 581.4% greater efficiency than IE8 at speeding up when workloads increase.
Betanews is applying these new tests to the latest stable browsers from the other Top Five browser makers; and yes, Ross Perot fans, we'll have the charts ready when the numbers come in.