Speed tests: Google curbed Chrome 3 speed prior to stable release

Two days ago, Google signed off on a stable version of series 3 of its Chrome Web browsers; and since that time, users everywhere are noticing two not-so-subtle changes: First, the New Tab panel has a different (and, we feel, better) layout. Second, it's noticeably faster.

Google promised speed increases of about 30% (often quoted as "one-third") for users who'll find themselves bumped up to Chrome 3 (Google's browser diligently updates itself). Last month, Betanews tested that claim, and projected speed increases of more like 24.5% -- still in Google's ballpark, just along the edge. But since that time, we noticed the company had made dramatic strides, with both beta and dev channel (Chrome 4) builds posting record speed numbers in our tests, for gains that could possibly break the 40% barrier.

But was Google able to deliver those numbers to stable, non-testing customers? We've been thoroughly testing our own tests in recent days, in response to an overwhelming number of mostly positive comments from readers that diligently point out that even "real-world" benchmarks tend to be error prone. So we have a project that's "under construction" at the moment, with regard to nailing down numbers that folks can trust, and that represents the actual speed differences that users experience as opposed to the artificial speed differences imposed by many tests under unrealistic circumstances.

While we've been hard at work on this, we have some early numbers for Google's speed increases. And the answer to that big question is actually a surprising no: On our physical test platform, version 3.0.195.21 -- the first officially stable release of Chrome 3 -- ended up 6.5% slower than the record numbers for Chrome 3's last beta version.

On our Windows XP SP3-based physical test platform (it's XP that Google appears to be targeting for its Chrome OS netbook-oriented project), the first stable Chrome 3 posted an index score of 18.68, which is near the results we were seeing when Google first made its 30% speed boost claim. We estimate that the new Chrome 3 will be 24.1% faster overall on Windows XP than the final stable Chrome 2. By comparison, the current stable Firefox 3.5.3 posted an index score of 9.97 on XP SP3, and the latest nightly developers' distribution of Firefox 3.6 Alpha 2 posted a record score for Mozilla of 11.53.

For those of you not familiar with our index scoring system, it's based on a scale where 1.0 is the assessed performance of the slowest browser in current use on the slowest updated platform it can be used on: Microsoft Internet Explorer 7 (not the most recent IE8, which is faster) on Windows Vista SP2, on the same physical hardware. We chose a slow browser on purpose as something against which all our faster browsers may be triangulated.

Where we had been seeing speed gains for Chrome were in three key departments: 1) page load time, where Google was already a major contender (but where Opera still has advantages); 2) JavaScript control flow, which involves the handling of heavily nested functions; 3) handling of arrays in memory. But now, not only have all those gains been scaled back to pre-August levels, but our early tests suggest that the latest Chrome 4 developers' builds have seen the same deceleration. On XP SP3, Chrome 4 has slowed down from a record 20.09 on our index on August 25, to 18.39 now -- actually a few points slower than the stable Chrome 3.

What do a few tenths of a point matter? Google's own JavaScript speed test suite, built around its V8 kernel, would suggest it matters a heck of a lot -- it generates heavy workloads based on the assessed capabilities of the browser on its test platform. But then it tends to magnify the differences as workloads get larger, which is one reason the V8 suite isn't part of ours. However, as features such as dynamic graphics rendering and complex DHTML become more and more a feature of real-world Web usage (and as browsers like Firefox and Chrome get more capable, this will happen sooner rather than later), those elements which make a few tenths of a point of difference on our current scale will actually become more magnified in the real world, as the V8 suite suggests. You'll be playing a 3D game through a pair of browsers, and you'll notice one renders more crisply and cleanly than the other, and this will be why.

Right now, Chrome's nearest competitor in the speed department is Apple's Safari 4, which posted a 16.57 on our index in Windows XP SP3. A recent look at its developers' work on the underlying WebKit rendering engine suggests a future version of Safari could very well blast Chrome off the center of the podium, assuming Apple's team sufficiently blends the rendering speed gains we're seeing with other important features such as control flow and memory conservation.

Yes, yes, we're working on new charts now.