Meet Microsoft's Ultimate Beta Tester

With tens of thousands of external beta testers examining the latest Microsoft software, little is said about the company's largest guinea pig: itself. Long before a final release, Microsoft has put the beta bits into production -- a process it calls "eating its own dog food."

BetaNews recently sat down with Microsoft Chief Information Officer Ron Markezich, often referred to as "Microsoft's ultimate beta tester," to get an in-depth look at how dog-fooding helps shape the software that hundreds of millions of people use each day.

As CIO, Markezich heads up Microsoft's IT department, which is responsible for managing the massive infrastructure that supports over 63,000 employees around the world. Perhaps more importantly, his team also plays a vital role in product development by actively using Microsoft software before it's complete, filing bugs and providing feedback.

Finding individual consumers to beta test products like Windows Vista is easy; even Google offers much of its software as betas. But businesses are a different story. Corporations rely on mission-critical systems to keep things running smoothly, and any hiccup due to buggy software could have major consequences.

Microsoft has attempted to alleviate this dilemma by encouraging businesses to install its enterprise software during the beta phase through the use of "go live" licenses. However, the company has learned the only way to firmly stand behind a release when talking to partners is to first deploy it internally.

That issue came to a head with the release of Exchange Server 2000. Markezich says the product was not running optimally inside Microsoft, but it was shipped to customers nonetheless. The result was a veritable nightmare: customers experienced the same problems and Microsoft had to rush out numerous hotfixes.

"From that day forward we said we will not ship a product that we sell to the enterprise until we run our business on it," Markezich explained. Since then, the dog-fooding program has extended across a variety of Microsoft products and now serves as a quality checkpoint on the road to a final release.

"Not only do we want to get involved in the beta testing, but we also want to give feedback in the product planning, product designs, requirements, and then all the way through the entire lifecycle," Markezich says.

Unlike a typical beta tester, Markezich's team holds considerable sway over the release of a new product. In fact, he has the power to delay a release to manufacturing (RTM) if a software product hasn't hit goals and expectations the development team had previously established as part of the dog-fooding process.

"If we don't meet these shared goals for a particular milestone, we don't allow the product group to move on to the next milestone, all the way to RTM. If we don't meet the goals that we have laid out for RTM, sometimes even a year and a half or two years prior, we don't allow the product to ship," Markezich said.

For Windows Vista, Markezich has shared goals for BitLocker, enterprise search, strong user authentication, network access protection, and a number of other business-oriented features.

"This organization is an extension of the product team; we have sign-off authority on those products before they ship," he adds.