Tests: Firefox 3.1 JavaScript outpaced by Safari 4, Google Chrome

Internet enthusiasts today are getting their first glimpse of the TraceMonkey JavaScript interpreter that Mozilla claims will be one of the principal reasons to own and use Firefox 3.5 -- what the latest Firefox will inevitably be called once the numerology gets sorted out. A fresh round of comprehensive Betanews tests Thursday afternoon indicate that Firefox 3.1 Beta 3 will demonstrate close to eight times the general JavaScript calculation and rendering performance of Microsoft Internet Explorer 7 -- a clear performance gain.

But as we were able to verify Thursday, it may not be nearly enough to earn the latest Mozilla beta the performance lead. In a battery of 20 calculation and rendering tests assembled from four different developers' benchmarks and benchmark suites, Apple's Safari 4 beta for Windows perceptibly and appreciably outperformed every other test browser on our virtual machine.

There is no widely accepted industry standard benchmark suite for Web browser performance just yet, but there are quite a few potential nominees. Over the last few weeks, we've been discovering some that were created by independent developers and shared with the rest of the world through Web sites.

In an effort to produce a fair and unbiased rendering of JavaScript engines' relative performance -- something closer to what you'd expect if we were comparing six different grades of CPU to one another -- we gathered a group of respectable benchmarks that appear to cover the gamut for different types of workloads: cryptography, data handling, DOM handling, AJAX declarations, parsing regular string expressions, string manipulation, mathematics, rendering, compliance with standards. And we re-cleaned and re-assembled our Windows Vista SP1-based virtual machine -- again, not the best candidate for a "super-speedway," but a plausible venue for an unencumbered test of how the latest experimental beta, alpha, and release candidate versions of major Web browsers perform in comparison to one another. (Though Google lifted the "beta" designation from its Chrome browser, its development is continually tested and its clients are automatically updated to reflect results from those tests, so we included the latest Chrome edition with the other brands.)

Firefox 3.1 Beta 3 was released to the general public yesterday. We spent a good part of yesterday comparing that release to the first Apple Safari 4 beta for Windows -- which performed astonishingly well in our initial tests -- as well as Google Chrome, Microsoft Internet Explorer 8 Release Candidate 1, and in response to many of our readers' requests, the latest alpha edition of Opera 10.

Knowing that our test venue would be an arguably slow virtual system, our goal here was to test relative performance -- to illuminate the scale of performance differences that all users would likely see on their own, probably faster systems. The way we chose to represent relative performance is by creating a base index -- something we already know to be slow and that we expect will be outpaced.

So we ran our series of tests on the most recently updated version of Internet Explorer 7, prior to upgrading our test VM once again to run IE8 RC1. For each test, we gave IE7 a score of 1.0, to serve as our baseline. We then used four benchmark "suites" for which every final result is rendered relative to that 1.0; so for example, a score of 2.3 on a speed test would mean the browser was 230% the speed of IE7. And for the Acid3 benchmark, which tests compliance, an index score of 8.0 for a browser would mean its rendering score would be eight times better than for IE7.

Next: The tale of the tape...