Firefox 4 beta loses to IE9 beta in browser speed, efficiency tests

The latest wave of upcoming changes to the world's two most used Web browsers, jointly responsible for easily three-fourths of the Internet's HTTP requests, has nearly everyone in the business rethinking the meaning of "quality" as it pertains to browser architecture. While their arguments start with the usual reminders that folks just want to see their pages load faster, before too long, they wander into dissertations about the methods architects use to achieve the appearance of loading faster. . . especially when they actually don't.

In preparing to test Microsoft's first Internet Explorer 9 public beta, released last week, and Mozilla's public Firefox 4 beta, released late last month, the advice I received most often fell into two departments: 1) Pay more attention to graphics rendering, since new browsers will be spending more time processing Web apps than just displaying pages; 2) start paying attention to how browsers utilize memory and CPU cycles. Since my smarter readers are typically right, that's what I've done in crafting my all-new browser performance test suite.

In the first round of total scores for the latest beta builds, the first IE9 beta has squeaked by the latest FF4 beta by about 8%. These scores take into account four major departments: computational ability (30%), rendering ability (30%), scalability (losing less processing ability over time as processing jobs scale up, 20%), and now, hardware utilization efficiency (20%). In past tests, as readers will recall, stable Firefox builds have typically scored around nine times the current IE8 score.

With last week's release, Microsoft has made up all of that ground, and then some. If you consider the general performance of the old Firefox version 3.0.19 (stable) as scoring a 1.000 in all categories (my new index browser), then by comparison, IE9 scores a 3.652 - better than three-and-a-half times the performance of what Firefox users were typically experiencing as recently as 2008. Firefox 4 Beta 5 posts a score of 3.377 in the same suite of tests.

What the numbers tell us

The computational engine in IE9, part of the new JavaScript engine dubbed "Chakra" (a Hindu term referring to the seven centers of human energy), is tremendously more capable than its predecessor. In computational scores alone, where IE8 (overall score: 1.145) scored an abysmal 0.502 (about half as powerful as the old Firefox 3.0), the first IE9 beta scored an impressive 7.645 - an improvement of more than 1500%. IE9 beta not only scores better than FF4 beta on the well-respected SunSpider benchmark (relative score: 14.268 versus 13.827, average time elapsed 417.8 ms versus 517.8 ms), but better than the latest stable version of Google Chrome 6 (relative score for build 472.62: 10.672). Although Chrome's cumulative time was better than IE9's (317.2 versus 417.8), my tests compare the relative times of each heat and average the results. Thus the fact that IE9 computed CORDIC variations (an alternate method for computing arctangents) in 1 ms versus Chrome's 10.4 ms and FF4's 29.6 ms, is reflected in this scoring system rather than washed out in the sum.

IE9 was not computationally faster across the board. It only scored a 2.077 on the JSBenchmark battery, which is even below the score for the latest stable version of Firefox (2.161 for version 3.6.10). A recent blog post by IE9 lead program manager Jason Weber, written in preparation for the IE9 beta's release, tries to spin testers' opinions away from certain brands of publicly available tests, describing and even diagraming them as incomplete. But Weber's description matched that of the original Celtic Kane benchmark, not the present test which is a complete replacement, and by the author's own admission, far more applicable to real-world workloads.

The JSBenchmark score, coupled with the blog post, gave credence to some Firefox users and other naysayers who argued that Microsoft has merely been advancing IE's scoring ability on the SunSpider test, and neglecting other departments. Is this true? To find out the answer for myself, I used a series of benchmarks that I wrote myself, for which neither Microsoft nor Mozilla have the source code. They're blistering problem-solving tests that are executed in varying workload sizes, to test for both performance and scalability - the ability to kick in more processing power when it's needed.

The news for the naysayers. . . is very bad indeed. IE9 absolutely flattened FF4 on my own DFScale battery, with a speed score of 6.590 versus 3.539. Maybe IE9 isn't the best performer in every test, but it's not because Microsoft is too focused on test numbers.

Next: Better performance, but at what cost?...

This article originally appeared in Net1News.

©Copyright 2010 BetaNews, Inc.