XP SP3 speed lead over Vista SP1 narrows under similar workloads

A heavily promoted performance test by an evaluation software firm appeared to situate Windows Vista SP1 performance against Windows XP SP3. But the initial workloads were actually different due to the Office software used, testers admitted to BetaNews today.

Devil Mountain Software's test results comparing similar workloads on systems with varying editions of XP and Vista -- including the latest service packs or their equivalents -- show the Vista system performing astonishingly more poorly, by a staggering 144%.

But a breakdown of the team's initial tests reveal that, although they used identical Dell computers, they actually compared Office 2007 performance on Vista to Office 2003 performance on XP.

"All testing was conducted against fresh installations of the respective OS platforms," the CTO of Devil Mountain Software, Craig Barth, told BetaNews this afternoon. "In the case of Vista, we installed the RTM code only -- no updates were installed in order to preserve a pristine image. We then installed Office 2007 and the DMS Clarity Studio and Tracker tools. Once benchmarking was complete, we upgraded the installation with the v.658 build (RC0) of Service Pack 1."

For the XP test, Barth said his team followed a similar methodology, but one which equipped the older operating system with the older applications suite.

Barth said his tack for the XP rig was "first, installing and testing a pristine XP + SP2 image (via the integrated CD ISO image from MSDN); adding Microsoft Office 2003, DMS Clarity Studio and Tracker; testing under SP2; then upgrading with Service Pack 3 and repeating the tests under the updated configuration."

But some readers noticed that both systems weren't being tested using the same workload. And since the team's testing software focuses on such factors as ActiveX Data Objects (ADO), whose deployments are very different for Office 2003 than for Office 2007, the differences do matter.

So by their request, the team installed Office 2003 on the Vista system, keeping in mind that Vista may still replace much of the infrastructure from the OS upon which the suite relies. When that happens, the performance gap narrows...to a mere 82%.

It may still be enough, though, for the team to uphold its claim from last week that Vista SP1 is "a performance dud." "Extensive testing by the exo.performance.network research staff," reads the team's blog, "shows that SP1 provides no measurable relief to users saddled with sub-par performance under Vista."

Barth revealed to BetaNews further details about the hardware platforms his team used. "The test system was a Dell XPS M1710 notebook," he said, "equipped with a Core 2 Duo CPU at 2 GHz (T7200), 1 or 2 GB of Corsair Value Select DDR-2 667 MHz memory, a Hitachi 7200 RPM 80 GB hard disk and nVidia GeForce Go 7900GS video adapter."

Devil Mountain Software does not appear to have compared performance when running Office 2007 on both Windows XP and Windows Vista.