PDC 2009: Microsoft cares about Web browser performance

A word about Dean's comments about our performance measurements

It is no secret to the dozens of you who have followed my work over the past few decades that I am a speed fanatic. I give a damn about the performance of my automobile, my coffee maker, my wristwatch, and my Web browser, and my wife knows it doesn't stop there. It might have something to do with why I edit a publication called Betanews.

The other reason I care is that I'm in the business of improvement, and you can't improve until you're ready to accept your own shortcomings. That goes for myself as well as everyone else. It is never fun to be on the losing side of a fair and competitive battle. It may even seem unfair when the reasons have to do with a legitimate effort to address a serious concern.

But as a high school journalism teacher I knew used to tell her students in the sports department covering one of the worst teams ever to take the field, if you can't do the simplest job in the world -- reporting the score -- then you shouldn't be a journalist but a cheerleader. Your heart may want you to change the score; but if you then do it, you've lost more than a game.

Dean Hachamovitch raised a few concerns about Betanews' testing methodology, and I think they're fair concerns that are worth discussing. First, he contended that it may be confusing to him and others, for us to use Internet Explorer 7 on Windows Vista as a relativity index. I told him I did not want to do as some readers suggested I do -- use IE8 as the index browser -- because that would be unfair to IE8, disabling its ability to be measured for performance at all. I also could not use any older series of browsers (for example, IE6 or Firefox 2) because they could not even run the tests in our performance battery. I know, I've tried.

He also suggested something else: that we go back to using clean installs of operating systems on virtual machines for our test systems, so long as those VMs were running on the same hardware. I explained to him why I changed to testing on physical platforms, mainly in response to reader requests. But I also explained that during our test of IE8 performance after the October Patch Tuesday fix, to validate the numbers I was seeing, I uninstalled the patch and tested again, and reinstalled it and tested again, to verify the values -- they were spot on after the validation.

Hatchamovitch winced at this. He said that by uninstalling and reapplying the patch, I may have reduced the "signal-to-noise ratio," to use his phrase, for IE7 and IE8 on Vista. In other words, I may have polluted the platform and hindered performance. The test results did not suggest that; but my methodology, he argued, could still produce doubt as to the authenticity of my results. This is a fair argument which I will put to the test myself.

Hatchamovitch also suggested we introduce the factor of variability into the test, a plus-or-minus factor, which is often seen in many other scientific measurements of performance. Of course, that's not a bad idea either; on the other hand, I only want to report numbers that make sense to readers. If I'm "fuzzifying" the meaning of a result, I may not be giving readers data that they can use. It would be like putting a plus-or-minus estimate on the Dow Jones Industrial Average.

I stand behind my current methodology, but as you've already seen, I'm open to suggestions for improvement, and I have made improvements based on suggestions. But as I told BASIC interpreter vendors during the 1980s, who for years had seen their performance numbers in my tests slammed again and again and again by Microsoft -- which dominated the interpreter market for as long as it was important -- winning is not a relative state. Ask anyone who's improved and come back to win the next round.

The moment IE9 marks a comeback for Microsoft's Web browser in performance, you'll read about it here. Yes, performance isn't the entire Web just as the drivetrain isn't the entire car. Maybe you don't think much about the drivetrain every day, but that doesn't mean it doesn't exist. Just try riding in a car without one.