One question I have been asked about game-like methods - is the data as good as that gathered through offline (or face-to-face) methods? That’s a great question -one that needs to be answered everytime you hear someone say “try this new & different research method”!

Our initial goal with MindCanvas has been to achieve the data quality of paper-based methods, so that is what I will talk about first. There are three things we watched our for - (a) the quality of the data itself; (b) user’s subjective experience of the users (did they enjoy it, did they feel engaged); and (c) efficiency of both methods (effort required to collect data).

a) Data quality: This is the real test, of course. Let me describe using a card-sorting example (note. we have made similar comparisons with Divide-the-Dollar, Freelisting etc.). One of the hallmarks of sorting data is number of meaningful categories. This is important for the statistics and for meaningful design insights. Analysis benefits from people making more fine-grained choices. For example, 10 8-item groups are better than 5 20-item groups. With paper card-sorting, with about 80 cards, we have observed an average of 11 meaningful groups (By meaningful groups, we mean groups that make sense, that have meaningful labels). Number of groups varied from 3 to 20. This was the goal we were shooting for with MindCanvas.

When we started testing MindCanvas, we got about 7 groups for 80 cards. As we added more game-like elements, and increased the intelligence in the system (giving feedback to users at appropriate places, keeping them informed about how they were doing…) the average number of groups increased and stabilized at about 11.5. The length of the labels (and their meaningfulness) seems very similar to that in paper-sorting. Of course, as is typical with online studies, we get some crap data (people who did not make any meaningful categories, or did not complete) - but a lot less than with a typical surveys. Statistically, the data has given us meaningful design solutions. I have not observed any differences between results of analysis on data gathered from paper sorts and that gathered through MindCanvas. However, the statistical solutions are more stable because of larger sample sizes, and its easier to break the sample into smaller groups to understand differences between user groups. (There are other measures we plan to compare in the future- e.g., how many times was same card placed in multiple categories, how many times a new card was created, number of second and third level categories). Most importantly, the user vocabulary that is surfaced by these exercises gets richer and more detailed the more subjects you run.

Our next goal is to exceed the data quality from paper sorts. We think this is possible, especially with more complex exercises (2-3 levels in sortings, divide-the-dollar with lots of options) since the software takes care of some of the mental bookkeeping and arranging people have to do with paper exercises. For example, you always know how many first, second and third level groups you have. You are reminded if you have not named a group. Even with a large number of cards/options, the space remains organized.

b) Now about the subjective experience - once again, our observation is that people enjoy using MindCanvas, in a similar fashion to those doing paper sorts. Some comments are below (note these are spontaneous user comments - we always have a “Any comments” question at the end.) For regular surveys, this box is mostly used for complaints. With MindCanvas, the experience has been pretty different! I should add that these comments are not from a tech savvy user population. Our beta experience has been with older age-groups, people who use the internet mostly for email and occasional browsing … A lot of people used the word “fun”, “enjoy”. A few used the word “game”.

“Filling out this survey was the most fun I’ve had in a long time. It was a brilliant application of computer technology. Ask me again sometime….PLEEEEEEEEEASE.”

“This survey was a novel and much less tedious data collection method than others I have used.”

“This was very cool. I take a fair number of online surveys and I really like the format of this one.”

“It is a GREAT way to see how people group various items.”

“Interesting and instructive way to manage a survey.”

“Great way to do a survey. The format and ease of use was excellent.”

c) Now about efficiency of data collection: That’s very similar to the type of efficiency gains when you successfully create a computer based process for a task. You can collect all the data you want in a day - it would have taken days to schedule that many people to come in person to your lab/office. MindCanvas can also be used for qualitative user insights. We simply use some sort of online screen sharing software (Webex etc.) and watch the person as they do the sorting, or other tasks. We send out the invitation to the meeting beforehand, and anyone from the client company can join in. The participant is encouraged to think aloud, designers/researchers can ask questions. Its great for busy designers, researchers who want qualitative feedback but do not have the time to do the in-person visits.

Insight into user language: One advantage of MindCanvas over paper-based, small-sample studies is the insight into customer language. We focus a lot on gathering user language with the various methods (sorting, free-listing), and do datamining to find the trends and patterns. Working on site language is much easier when you have access to trends in word usage across 200 users. You can see the common words, and the long tail of idiosyncratic word usage. Its like del.icio.us, but for your specific design questions.

This is a topic I expect to come back to as we learn more. So expect periodic updates about data quality…