Firefox 4 beta loses to IE9 beta in browser speed, efficiency tests
The latest wave of upcoming changes to the world's two most used Web browsers, jointly responsible for easily three-fourths of the Internet's HTTP requests, has nearly everyone in the business rethinking the meaning of "quality" as it pertains to browser architecture. While their arguments start with the usual reminders that folks just want to see their pages load faster, before too long, they wander into dissertations about the methods architects use to achieve the appearance of loading faster. . . especially when they actually don't.
In preparing to test Microsoft's first Internet Explorer 9 public beta, released last week, and Mozilla's public Firefox 4 beta, released late last month, the advice I received most often fell into two departments: 1) Pay more attention to graphics rendering, since new browsers will be spending more time processing Web apps than just displaying pages; 2) start paying attention to how browsers utilize memory and CPU cycles. Since my smarter readers are typically right, that's what I've done in crafting my all-new browser performance test suite.
In the first round of total scores for the latest beta builds, the first IE9 beta has squeaked by the latest FF4 beta by about 8%. These scores take into account four major departments: computational ability (30%), rendering ability (30%), scalability (losing less processing ability over time as processing jobs scale up, 20%), and now, hardware utilization efficiency (20%). In past tests, as readers will recall, stable Firefox builds have typically scored around nine times the current IE8 score.
With last week's release, Microsoft has made up all of that ground, and then some. If you consider the general performance of the old Firefox version 3.0.19 (stable) as scoring a 1.000 in all categories (my new index browser), then by comparison, IE9 scores a 3.652 - better than three-and-a-half times the performance of what Firefox users were typically experiencing as recently as 2008. Firefox 4 Beta 5 posts a score of 3.377 in the same suite of tests.
What the numbers tell us
IE9 was not computationally faster across the board. It only scored a 2.077 on the JSBenchmark battery, which is even below the score for the latest stable version of Firefox (2.161 for version 3.6.10). A recent blog post by IE9 lead program manager Jason Weber, written in preparation for the IE9 beta's release, tries to spin testers' opinions away from certain brands of publicly available tests, describing and even diagraming them as incomplete. But Weber's description matched that of the original Celtic Kane benchmark, not the present test which is a complete replacement, and by the author's own admission, far more applicable to real-world workloads.
The JSBenchmark score, coupled with the blog post, gave credence to some Firefox users and other naysayers who argued that Microsoft has merely been advancing IE's scoring ability on the SunSpider test, and neglecting other departments. Is this true? To find out the answer for myself, I used a series of benchmarks that I wrote myself, for which neither Microsoft nor Mozilla have the source code. They're blistering problem-solving tests that are executed in varying workload sizes, to test for both performance and scalability - the ability to kick in more processing power when it's needed.
The news for the naysayers. . . is very bad indeed. IE9 absolutely flattened FF4 on my own DFScale battery, with a speed score of 6.590 versus 3.539. Maybe IE9 isn't the best performer in every test, but it's not because Microsoft is too focused on test numbers.
Next: Better performance, but at what cost?...
©Copyright 2010 Ingenus, LLC.
©Copyright 2010 BetaNews, Inc.