Philips WebLabs

Bring someone into a usability lab and you can see everything: what they click, where they hesitate, what they say when it goes wrong. What you cannot see is how they would have behaved at home, on their own machine, with the television on.

Philips wanted to know whether that difference mattered enough to keep paying for the lab. The answer had to be measured, not asserted.

The question

WebLabs was a web-based platform for gathering user feedback at any point in an iterative, user-centred design cycle. Remote testing promised time and place independence, cheap access to an audience of any size and culture, and results that arrived already tabulated. Against that, you give up control the moment an experiment leaves the building.

So: does remote testing produce the same answers as the lab, for the kind of multimedia software Philips Research was building for consumer electronics?

How it was answered

Two groups, self-selected: participants chose whether to come in to the lab or take part from home. They were recruited with flyers across the High Tech Campus, through the Research newsgroup and the TU/e USI mailing list, with at least eight people needed in each group for the comparison to hold.

The instrument was four standardised questionnaires, run identically on both sides. The stimulus was Edit While Watching, an experimental video editing tool, later joined by a photo book application that generates an album from a collection the user submits.

What came back

No significant difference between the two methods, except that the online tool was the better instrument for testing simple online applications. For the work Philips Research was doing, the lab was not buying what it was assumed to be buying.

Two things that were supposed to be incidental turned out to be results. Edit While Watching proved unstable in consumers’ hands, which is a finding about the software rather than the method. And turnout, around 40% against the 10 to 50% typical of internet studies, became a finding in itself: the practical question about remote research is not whether the data is good but whether anyone shows up.

The photo book application demonstrated a second use for the platform. Run the same collection through different photo-processing algorithms, show the results to enough people, and you can compare algorithms by how much they are actually liked rather than by how they score.

Published as Use of WebLabs as a platform for remote customer research, Eindhoven University of Technology, Stan Ackermans Instituut, 2008. ISBN 978-90-444-0812-6.

Who built it

Carried out at Philips Research Laboratories Eindhoven, on the High Tech Campus, from March to May 2008, in the Experience Processing group. Coached by Maddy Janse and Jun Hu at Eindhoven University of Technology and by Ard Biesheuvel at Philips Research. The software was built by developers in the group, in C# on .NET, working in an agile way.