But as we were able to verify Thursday, it may not be nearly enough to earn the latest Mozilla beta the performance lead. In a battery of 20 calculation and rendering tests assembled from four different developers' benchmarks and benchmark suites, Apple's Safari 4 beta for Windows perceptibly and appreciably outperformed every other test browser on our virtual machine.
There is no widely accepted industry standard benchmark suite for Web browser performance just yet, but there are quite a few potential nominees. Over the last few weeks, we've been discovering some that were created by independent developers and shared with the rest of the world through Web sites.
Firefox 3.1 Beta 3 was released to the general public yesterday. We spent a good part of yesterday comparing that release to the first Apple Safari 4 beta for Windows -- which performed astonishingly well in our initial tests -- as well as Google Chrome, Microsoft Internet Explorer 8 Release Candidate 1, and in response to many of our readers' requests, the latest alpha edition of Opera 10.
Knowing that our test venue would be an arguably slow virtual system, our goal here was to test relative performance -- to illuminate the scale of performance differences that all users would likely see on their own, probably faster systems. The way we chose to represent relative performance is by creating a base index -- something we already know to be slow and that we expect will be outpaced.
So we ran our series of tests on the most recently updated version of Internet Explorer 7, prior to upgrading our test VM once again to run IE8 RC1. For each test, we gave IE7 a score of 1.0, to serve as our baseline. We then used four benchmark "suites" for which every final result is rendered relative to that 1.0; so for example, a score of 2.3 on a speed test would mean the browser was 230% the speed of IE7. And for the Acid3 benchmark, which tests compliance, an index score of 8.0 for a browser would mean its rendering score would be eight times better than for IE7.
Next: The tale of the tape...