Meet Microsoft's Ultimate Beta Tester

Dog-fooding begins with the development team, which runs a product during its alpha stage. Beta testers are then invited from within Microsoft in "chunks" using internal advertising and awareness campaigns. Over time, the number of employees running pre-release software can encompass much of Microsoft.

Over 60,000 machines were running Windows XP Service Pack 2 during its beta phase, and Markezich expects at least that many for Vista before it ships. By mid-June, Microsoft had 16,000 Windows Vista Beta 2 users and 25,000 on Office 2007 Beta 2.

"We have a group for each product we're dog-fooding that manages the testing process," Markezich said. "As we get to Beta 2 we say: 'Woohoo! People, get on Beta 2!' Part of the role is evangelizing, and part of the role is getting feedback to the product teams -- then we'll determine what build to make available to all the teams."

Typically, there are around eight deployments of the various builds spanning from the alpha stages to RTM. Another full deployment is made across the company at RTM.

Although Markezich is currently overseeing the largest number of Microsoft products ever to be dog-fooded simultaneously, he has no trouble rattling off the numbers. Along with Vista and Office, over 6,000 mailboxes are now on Exchange 12, and more than 30 Longhorn Servers are already in deployment.

1,000 Microsoft employees are running the next version of Windows Mobile, code-named "Crossbow," and 7,000 Systems Management Server v4 -- renamed to System Center Configuration Manager 2007-- installations are live. Over 3,000 machines running the next Microsoft Operations Manager release are also in production.

Markezich's eyes light up when he's asked to expound the benefits Microsoft has received through dog-fooding. Not only does his team play a key part in making sure a software product is of the highest quality, but he says Microsoft also gets to take advantage of new functionality right away.

"Once we rolled Exchange 2003 out, I went from 74 sites with Exchange servers, to 7 sites with Exchange servers throughout dog-food testing. Right now, I'm down to 4 sites with Exchange servers," Markezich explained. "So the fact that I'm dog-fooding means I'm going to get more value out of it very soon."

"My highest scores for employee satisfaction and user satisfaction are years that we have big dog-fooding waves like this," he notes. "Honestly, when an employee finds a problem, we get excited -- if it's before RTM."

Based on results from dog-fooding Exchange, Office and Vista, Microsoft is embarking on what it calls the "7x24 initiative." The company is pledging that employees will get back 7 million hours of productivity after using these products for 24 months. Markezich says Microsoft will "blow that away" with over 10 million hours of extra productivity.

But Markezich admits it's not always smooth sailing when you're rolling out potentially buggy software to employees who have their regular jobs to accomplish.

"Sometimes, as a CIO, you'll get an irate employee who says: 'You guys screwed me, I had a customer presentation and I couldn't do it because of this!' But we tell them: 'You just provided more value to the company because you found that problem than you could have provided otherwise, and now our customers won't have the same issue.'"

Moving forward, Markezich sees more and more products being dog-fooded within Microsoft, and says the extra overhead of running mission-critical systems on beta software is well worth the benefits.

"I'm waiting for the day when we start dog-fooding Xboxes," he jokes, "but we haven't gotten there yet."