After a few days, a mixed verdict on Microsoft Photosynth

Daley's discussion of scenes in Washington, DC got us to thinking about ways we could test Photosynth for ourselves. We arranged two sets of multiple photographs with a high degree of overlap between them, though with certain photographic obstacles that Photosynth would have to overcome. One was a series of multiple shots of the United States Capitol, taken at different times of the day over different years from a variety of angles, but mostly by one photographer (myself). It's a fairly simple building, really: a rectangular block with a large rotunda dome at the center and two smaller domes on either side. Of course, the trick may be in determining which side of the Capitol building each photo represented, as distinguishing the east from the west wall, or the north from the south wall, could be challenging.

The other set featured a very complex object, but something an ordinary user is likely to try: the sculpture of the President seated at the center of the Lincoln Memorial. We selected fifty shots of Mr. Lincoln taken at four different times, from highly overlapping angles, none from the rear.

On two separate test machines -- one with Windows XP Professional SP3, the other with Windows Vista SP1, both using Internet Explorer 7 as the browser -- our US Capitol test repeatedly failed to produce a "synth." We'd jumble the pictures around, trim some of the more difficult ones from the set, or add some simpler ones with more direct angles. In each case, we met with different faults, including an "error A000000B" message, one case in which the program started itself over as if nothing had started, one in which the program appeared never to stop loading pictures, and several common hangs that could be resolved with the Task Manager.

In one very notable and disturbing case of the Capitol test on our Vista machine, the ActiveX control on which Photosynth is based appeared to try to add some functionality to Windows Explorer, which is something ActiveX controls have been known to do in the past. However, in this case the result was to permanently hang both Explorer (the file manager, not the Web browser) and the Windows Sidebar whenever they attempted to launch. In all cases, the hang was so severe that it prevented even the Task Manager from displaying.

We had to try to uninstall Photosynth while in Safe Mode to restore our system. That failed, since Photosynth requires the Windows Installer driver, which is among those that Safe Mode -- in its infinite wisdom -- disables. So we carefully had to set up our system so that neither Explorer nor Sidebar launched automatically, then quickly invoke the uninstall routine in normal mode, before Vista could find an excuse to hang.

All in all, the diagnosis and remedial phase of this test consumed about five hours.

After seven tries of the Lincoln photo set (though we wish we had thought to change the title after the first six, thus we ended up with "Lincoln Test 1"), we were finally able to come up with an arrangement of 48 of our images that enabled our XP machine to generate a synth. The results were, well, surprising.

Photosynth calculated an 60% average overlap, or "synthy-ness," to risk sounding like Stephen Colbert. Twenty-seven of our photos featured the Lincoln sculpture from his right side, ten from his left, with the rest being close-ups of spots a sculptor might take interest in, like the tuft of his hair. Photosynth didn't appear to have much luck blending those close-ups with the rest of the image, which doesn't exactly disqualify it.

But we noticed that Photosynth could not reconcile the right-sided shots with the left-sided shots, even when the angular difference was only a few degrees. In such situations, the program generates multiple "3D groups," assuming that the scene includes multiple sets of objects rather than just one. In our Lincoln test, we had three 3D groups.

To help the viewer's eye reconcile the multiple images in the scene, Photosynth often shows pieces of other photos in semi-transparency, sticking up like tabs of folded origami paper. In some cases, like for photos that were clearly taken from the exact same angle but at different zooms, the blend was spot on. For others, however, Photosynth appeared to take a "best guess," and that guess was typically close, but in a few cases, completely off.

Not that guessing the 3D structure of a complex sculpture from a handful of 2D photographs isn't a difficult, if not impossible, task -- which Photosynth attempts surprisingly well. The problem is, when it's off, it's off.

This is one of the clearest examples, where Photosynth mistook a portion of the chair's left armature as Lincoln's right shoulder.

So how would being "off" translate into a more business-related circumstance where tolerances for failure are smaller? For instance, we asked Microsoft's Alex Daley, what if a synth were to include a series of blends, maybe even made with Photoshop, of a city's architecture with a rendering of buildings yet to be constructed -- say, as a presentation for an architectural firm.

Daley played with the left side of that question and the right side separately, almost like Photosynth itself with our Lincoln Memorial test. He himself made a synth using scenes rendered and captured from the game Halo 3; and New York real estate firm Brown Harris Stevens has been using Photosynth to generate walkthroughs of apartment properties throughout Manhattan. But never, apparently, have the twain met; no one seems to have tried blending an artist's rendering with un-retouched photos.

"Where Photosynth really excels is in actual reality, not this artificial, virtual environment," Daley concluded.

What would be the next step for this technology? For instance, could a feature of a future Windows operating system group images together by their overlaps and similarities -- in other words, producing synths in the background?

As Daley gently implied to BetaNews...Windows has a hard enough time generating textual indexes in the background. "It's a very interesting idea, and I think [in the] long-term, computer technology should be able to address," he said. "But it's definitely a long-term scenario before we can do that kind of offline comparison, computation. One of the biggest barriers to that kind of scenario -- where we look at all your pictures -- is the sheer number of pictures. If you throw 20 pictures or 50 or even 300 pictures into Photosynth, it'll work just fine. But when you start looking at numbers like 3,000 or 4,000, comparing them to every other picture in the collection, it goes beyond the computational memory power of most computers today."

As we found out for ourselves, even fifty can be a tough task for now. But this is the beginning of a new kind of service for Microsoft, one which will provide a much-needed boost in Windows Live registration and usage. Although it doesn't say so up front, Photosynth as a trial of Microsoft's marketing strategy going forward, is very much a beta.