Update: Why did Windows Phone rate so poorly in Pfeiffer's tests?

Just last week, I wrote an article based on a report by Pfeiffer Consulting which suggested that Windows Phone 8 is, in its current state, the least user-friendly mobile operating system available. The article stirred up some feelings, and there were some quite heated exchanges in the comments.

For this follow-up piece, I spoke with Andreas Pfeiffer, the author of the report, to address some of the points, complaints and concerns you raised.

One thing that was mentioned time and time again by commenters is the fact that Pfeiffer Consulting lists Apple as a client on its homepage. The fact that iOS 6 and iOS 7 fared well in the report is seen by many as suspicious. In fact Pfeiffer lists numerous well-known companies as clients, including Adobe and Autodesk. The study was completely self-funded, and the only connection with Apple is hardware benchmarking that was done for the company "a long time ago".

Andreas explains:

Whenever we do a client project like this, we very openly state who has funded it, and we publish the complete methodology and the complete results, which is the only way to lend credibility for the data. Our clients don't pay us because they want to control what we say, but because they trust our methodology, and the last thing they want is to look like they have paid for favorable coverage.

I know what he means. It is very hard to publish any piece of work and have it taken at face value. I have learned from more than a decade of writing that as soon as you praise a company or product you are accused of being on the payroll. Hand out a dose of criticism and you're accused of not "getting it", being a hater or having an axe to grind.

Sure. I understand this. It makes sense.

It's very easy to voice skepticism about funding and methodology, but it would also not be particularly difficult to do a little digging and discover the source of funding -- it is all but impossible to keep such things secret these days. The suspicion was anticipated because the approach taken and perspective given is such a breakaway from the norm.

But here is a simple report, designed to offer a fresh way of looking at things. All of the featured operating systems are criticized in some way. Every company -- Google, Apple, BlackBerry and Microsoft -- has something to take away from it. No one would be foolish enough, user or manufacturer, to claim that its operating system is perfect.

But it is interesting to see just how defensive operating system users get. My mobile OS du jour is Android. I have been enamored with Windows Mobile, BlackBerry and iOS at various times in my many years as a smartphone user, but Android is my current selection. Unlike some people, I do not try to push my OS preference onto anyone else, just as I don’t push my vegetarianism or love of The Smiths (make a link if you like).

I could not care which operating system you use and whether you agree with my choice. It does not matter if you use iOS, Android, Windows Phone or BlackBerry. I. Do. Not. Care. Yet still accusations are thrown around. In publishing the report, Andreas is not trying to sell iOS to you or anyone else -- there would be no benefit in that for anyone involved.

Pfeiffer has taken a very different approach to looking at operating systems -- it is all about context. When analyzing phones, they were looked at through the eyes of a "casual, non-expert user". Yes, an OS might have features, but how easy is it to discover them, how intuitive are the settings; does it all make sense to the average (wo)man? Andreas is quick to point out that this "is quite the opposite of the typical reader of tech magazines or news sites".

What is important to keep in mind from the start is that this was not -- and has never claimed to be -- research based on user surveys. The report differs from other OS comparisons in that one of its key aims is to "quantify the actual value a customer derives from a digital device". Andrea explains that this is not based on "quantitative market research, but rather a purely empirical approach". The methodology behind benchmarking the user experience is rather different to other research.

But being different is, itself, no reason to dismiss something out of hand. Yes, the report takes an unconventional approach, but that is its raison d'etre -- to look at the mobile operating system from the point of view of the user. It is a "small first step, the beginning of a new way of looking at technology", says Andreas.

There are many things that can be said about the report, but it certainly cannot be criticized for being poorly documented. The new approach to analyzing user experience is laid out in some detail in the conceptual framework paper. Here Pfeiffer explains that the concept of user experience is a mixture of not only obvious tangible aspects, but also subjective and objective intangibles. One of the reasons for the development of the framework is that most of the current methods of comparing devices and operating systems are too focused on technical specifications. While this is important, it is far from being the end of the story.

The way an operating system is set up is important. This is where the User Experience Friction Rating (UXF) comes into play -- frustrations and limitations brought about by the OS. Andreas explains:

Each UXF element is rated on a scale of 10: elements that the user gets used to over time receive a lower rating, elements that remain annoying receive a higher mark. To give you an idea: the confusion between apps and widgets receives a low rating -- once the user gets the hang of it he's fine. On the other hand, the lack of camera access from the lock screen gets a high rating, because you will be annoyed by the missing feature even after having used it for years.

It goes without saying that people choose the mobile OS they use because it works for them. It probably means that they have dismissed the competition because it doesn’t work for them. Arguing about which OS is better is an exercise in futility because it is entirely subjective. But there are things that can be looked at objectively, and this is what the report tries to do. "Quantifying the intangible" may sound a little hippy-dippy, but it describes what Pfeiffer has tried to achieve.

This is a new and different way of looking at things -- it is going to cause disagreement and it is controversial. But it's not wrong. The report has been updated with further explanation of the methodology used as this is an ongoing project.

The Pfeiffer website includes an interesting essay about the feature-value gap. New phones and tablets are advertised based on the speed of the processor, the number of pixels crammed onto the screen and other such specs. For the average user -- not the technically minded -- there is really little to differentiate between the most popular phones. For day to day activities, it makes no difference whether a phone has a 1GHz dual-core processor, or a 2GHz quad-core. The operating system is what users must battle with to interact with the hardware… and battling is what many people find themselves doing. This is partly because of the learning curve involved in trying something new, but also due to the intuitiveness (or otherwise) of the software.

This is what the report is about. It is not saying one OS is better than another. Windows Phone comes out at the bottom of the heap largely because of the limited range of customization available, but also because of the unintuitive ways some tasks are performed, or the lengthy steps required to perform common actions. This doesn't mean you should stop using Windows Phone if you're happy with it, but it is food for thought for Microsoft to consider for future updates. The other companies involved can also learn a lot -- this was never intended to be an exercise in bashing Microsoft (or lauding Apple for that matter).

A final word from Andreas: "If a client hinted that they would pay us to write something in their favour -- which none of the big companies we have worked has ever done -- I would just run the other way".

Vent your spleen, agree, disagree, etc below. Opinion exchange... it makes it all worthwhile.

Photo Credit:  txking/Shutterstock