Vista Security Report Raises More Doubts Than It Relieves

When Microsoft's director of its Security Technology Unit Jeffrey Jones previewed a report that was soon to be published about the number of reported and addressed vulnerabilities in Windows Vista over the first six months of its consumer market shelf life, at TechEd in Orlando two weeks ago, the generally confused and negative reaction among attendees who ended up arguing with Jones for most of the session, prompted BetaNews (who was there) to decide that, amid the other news emerging that week, it wasn't worth covering.

The essence of the report is that Windows Vista had a far fewer number of reported security vulnerabilities during its first six months not only than Windows XP after its introduction, as recorded in the US National Vulnerability Database, Red Hat Enterprise Linux 4 WS, Ubuntu 6.06 Long Term Support Desktop, Novell SUSE Linux Enterprise Desktop 10, and Mac OS X 10.4 (Tiger).

While that statistic is not altogether meaningless, on its face, it is open to varying interpretations; and Jones' presentation of these statistics ended up encouraging, if not demanding, questions about interpretation - including his own.

Repeating his comments from TechEd, Jones' report presents a slide showing Red Hat had patched 281 vulnerabilities during Enterprise Linux 4's first six months, including several pre-existing ones, though had left 65 issues that were widely disclosed as unpatched at the end of that period. By contrast, Windows Vista's security team addressed 12 disclosures during its first six months of public use, though left unfixed 13 disclosed problems with varying degrees of severity.

After Jones' report's release yesterday, the extreme interpretations began to appear. On the one side was the "scoreboard" interpretation, which was essentially: "Red Hat 4: 346; Windows Vista: 25. Vista Far Safer Than Linux!" On the other side came, "Red Hat 4: 81% addressed; Windows Vista: 48%. Vista Riddled With Security Holes!"

Jones doesn't help his side very much, not only by tossing out conclusions that are open to cross-examination, but also by making concessions that would lend credence to skeptics that Vista truly is any more secure.

In his report yesterday, beneath the main slide which appears to spell bad news for Red Hat, Jones wrote, "As can be seen, Windows Vista shows an improved situation over its predecessor and exposed an even smaller vulnerability footprint than the Enterprise Linux distributions or the most recent major Mac OS X release."

The implication here is that greater numbers of disclosed problems only increase an operating system's "vulnerability footprint." Attendees of Jones' solo session at TechEd, including people who described themselves as loyal Windows admins, seized upon that characterization as ungrounded. Just because a problem isn't generally known, one fellow said, doesn't mean it doesn't exist. Another pointed out occasions where his beta testing team found numerous problems with Vista's betas, a great many of which he claimed persist in the release edition. But in working with Microsoft, he got the impression that certain problems, while acknowledged, fail to rise to the level of "vulnerability."

Yet another brought up the question of how vulnerabilities become enrolled in the NVD database to begin with. As Jones' report indicates, Vista vulnerabilities are first made public through Microsoft's own security bulletins. By contrast, others in the audience said, Linux vulnerabilities are often discovered by the open source community at large. They may be part of the public database before the vendors take a hard look at them - a process which Jones appeared to argue was not all that beneficial for the vendors.

Furthermore, part of the presentation of Jones' own report could be condensed at points to read, "Please Insert Your Doubts Here." One footnote regarding the whole issue of the validity of disclosed vulnerabilities appears to concede a degree of fuzziness exists there, even in Jones' own mind:

"Disclosures are harder to track than fixes," Jones writes, "since for fixes one only has to check the vendor site, but for disclosures one has to check many locations where vulnerability information could have been published and then validate that the vulnerability applies. This is as accurate as I can be, but if someone identifies further vulnerability disclosures that I missed, I will acknowledge it and update appropriately."

While Jones appears here to welcome corrections -- and even took on a kind of "Bring it on" stance during his solo session at TechEd -- the "About the Author" section at the close of his report contains a caveat: "Jeff actively encourages readers to challenge his assumptions, analysis and conclusions and provide critical feedback," it reads, "but asks for equal (or better) rigor in methodology and analysis to support the challenges, as opposed to enthusiastic espousal of unsupported evangelistic fervor."

At another TechEd session, Jones was asked to share his slide with attendees of an otherwise very convincing session on Vista security improvements by Michael Howard, Microsoft's security program manager. Howard spoke on specific Vista improvements such as Address Space Load Randomization and Data Execution Prevention, although with regard to the latter, Howard regretted not being able to apply it to Internet Explorer 7 mainly because pages that used Flash, Acrobat, and other Adobe tools would cause IE7 to crash with DEP turned on.

It was amid a technical discussion about features such as these that Jones was asked to show his slide, toward the end of the hour-plus session. With only a few minutes on stage, Jones made a statement that actually subtracted from his position that Vista was more secure than XP by virtue of its smaller footprint, conceding that five years ago, not enough people were around actually finding XP's holes until it was too late.

"I argue that the industry of finding bugs is a much more mature industry today than it was, say, when XP was around," Jones said. "So we're competing in a tougher environment. But [even in light of that], there's still progress."

Jones went on to say there's a general belief that Microsoft's opinion about Vista security is that it has reached the end goal, it's done, and it's moving on to other subjects now. "I think, based on the reporting we use today, we don't believe that," he said. "We believe that we're in a really important step along the way, [in our ability to assess what we can do throughout the next five years] to keep raising the bar, and keep improving the experience for Server 2008, and products after that. That's one of the things I'd like you to take away. [Fewer disclosures represents] a great step of progress along the way."

No one could possibly sit through an hour of down-to-earth technical discussion about Microsoft's security team's thorough renovation of Windows Vista's kernel, without coming away with the idea that for once in its life, the company as a whole is very serious about alleviating the architectural security issues that prior versions of Windows and its applications introduced. Reasonable people will agree that Vista makes a genuine effort, and that there's a lot to be proud of.

But with skepticism of Microsoft almost engrained into the consciousness of not only Linux admins but Windows admins as well, almost any report that uses a "scoreboard" approach as a statement of proof of Vista's progress, is going to be scrutinized very heavily. Inaccurate interpretations can and will be made. These facts probably can't be helped. But certainly they could have been foreseen.