If a subject is steadily verbalizing while performing a task (e.g. concurrent verbalization), they are assumed to be speaking from short term memory.  Dumas & Redish (1993) conveniently summarize three levels of think-aloud protocols commonly referred to by HCI researchers:  Level 1 verbalizations:  where the emphasis is on pure thoughts with no or minimal explanations; Level 2 verbalizations:  same, but when the participant is dealing with non-verbal information, like shapes, which must be internally “coded” in order to be articulated verbally, and Level 3 verbalizations:  or “thinking plus explanations.”  The latter are also referred to as retrospective reports, because the respondent is recapping and opining about what they actually did earlier.  Dumas summarizes the distinction between Level 3 and the other levels as the stage where the researchers are “no longer getting a read out of short-term memory… rather it is the interpretation of the process they are using or the reasons they have selected a strategy.”  Retrospective reports are not useless to the user researcher, in fact they are necessary to clarify a respondent’s statements and actions, but they are far less valid.

This leads us to the question of active vs. inactive moderation.  In an inactive moderation scenario, the emphasis is on experimental control and creating a unified experience for all test subjects.  This is the old-school style, where the researchers stand behind the glass and the respondent sits in the room by themselves talking out loud like a crazy person.  Participants, faced with the unnatural task of constant verbalization, are typically “coached” on how to deliver a think-aloud protocol.  This often takes the form of a warm-up exercise where the participant and experimenter practice thinking aloud with non-related stimuli, preferably a simple game such as tic-tac-toe (in order to place emphasis on cognitive strategy.)  During the actual experiment, the moderator prompts only when the participant ceases to verbalize: “Please keep talking,” they say.  And that’s all they say!

In a high moderator intervention scenario, the experimenter employs probing questions to focus the participant’s attention on particular features or to elicit and clarify subjective explanations of their behavior.  Moderators are skilled in asking neutral, non-leading questions to minimize bias.  In addition, active listening techniques are employed to emulate the clinician’s empathic stance. This implies paraphrasing of a speaker’s comments to ensure them that they have been listened to, noted, and understood as well as other verbal and non-verbal forms of caring, non-judgmental acceptance.  Mike Kuniavsky, whose work is mentioned frequently in these pages,  lays out the guidelines for “non-directed” interviewing:  questions should be concentrated on immediate experience, nonjudgmental, focused on a single topic, open-ended, and non-binary (e.g. yes-no, true-false).

In a quick look at the literature, you’ll see that Taylor and Dionne (2000) suggest that probes are best deployed for collection and verification of data in retrospective reports, and that they have a detrimental impact on validity if used during concurrent think-aloud protocols.  Preece (1994) suggests that the role of the moderator on the participant is both interruptive and imposes additional cognitive load.  Nielsen (1993) is pragmatic on the subject, suggesting the moderator intervenes as little as possible, yet directing the flow and direction of the interview to maximize the number of usability issues found.   Nielsen is the spiritual father of the commercial usability field, so it’s no surprise that most usability is performed with this degree of pragmatism.  I think it was Jared Spool who once commented that you can only watch participants avoiding clicking on the red button so many times before you are compelled to intervene and ask why.   Plus, most clients of usability research do not share the academic’s interest in validity.  They want enough validity to feel good about the process and the results, but ultimately they want their specific questions answered for a reasonable amount of time and expense.  In Nielsen et. al.’s (2002) interpretation, The human is a psychological being engaged in a psychological interaction, which cannot be reduced to that which is concurrently verbalized.

Some usability researchers, characterized by Whiteside, et. al. (1993), have also posited that observing a user’s behavior is not enough to understand what is happening in terms of higher order thinking and cognitive strategy.  If you set up a priori conditions then you are bounding yourself to learning only what falls within those conditions.  In this view, subjective experience is the most comprehensive criterion for understanding usability.  Concerns of generalizability are side-stepped: the goal is to obtain rich, experiential data.  Since specific questions are the only effective way to elicit and clarify mental models, the researcher must come to terms with at least a partially subjectivist stance. (Tamler, 2001)  This approach values reflexivity:  the subject is full participant in the study, leading the research into relevant areas for exploration as well giving the subject the ability to respond to the researchers interpretations. It is not a search for “truth,” per se, as a philosophical underpinning of this method of inquiry.  The assumption is that an expansion of perspectives leads to the exposure of more aspects of learning.

Here’s some References (I know, for a blog post, it’s over the top… but this stuff’s important!)

DUMAS, J.S. (2001) “Usability Testing Methods:  Think-Aloud Protocols,” in Design by People For People:  Essays on Usability, UPA,  pp 119-129

DUMAS, J.S. & REDISH, J.C., (1993)  A Practical Guide to Usability Testing.  Norwood, NJ, Ablex Publishing Corp.

ERICSSON, K.A. & SIMON, H.A. (1984, 1993) Protocol analysis:  Verbal reports as data (Rev. ed).  Cambridge, MA:  MIT Press

KUNIAVSKY, M. (2003)  Observing the User Experience: A Practitioner’s Guide to User Research,  San Francisco: Morgan Kaufmann Publishers, Inc.

NIELSEN, J., (1993) Usability Engineering.  Chestnut Hill, MA:  Academic Press, Inc.

NIELSEN, J., CLEMMENSEN, T., & YSSING, C., (2002)  “Getting access to what goes on in people’s heads? – Reflections on the think-aloud technique”, paper presented to NordiCHI, Arhus, Denmark, October 19-23

PREECE, J. (1994), Human-Computer Interaction, Addison-Wesley, England

TAMLER, H. (2001) “How (Much) to Intervene in a Usability Testing Session,”  in Design by People For People:  Essays on Usability, UPA,  pp 165-171

WHITESIDE, J., BENNETT., J.L., & HOLTZBLATT, K., (1988)  “Usability Engineering: Our Experience and Evolution,” in Handbook of Human Computer Interaction; edited by Helander, M.  New York, NY:  Elsevier Science Publishers

Photo Credit: curiouslee

Focus groups are probably the most difficult moderating challenge.  And I know this because I was terrible at conducting them until I learned a few tricks…

1) POLL the room frequently. Recap (or let the respondent recap) and then ask the room to raise their hands for who agrees. For those who disagree, ask them to state why? Don’t forget, observers usually want to understand what the group thinks about a certain question, and polling delivers.

2) ASK FOR COUNTERARGUMENTS. Control for ‘groupthink’ by asking “C’mon, Doesn’t anybody want to argue the opposite point?”

3) GO AROUND THE TABLE every so often. Periodically go around the table letting each person answer the question in turn. This will distribute the talking around the room, allow you to control the flow, and respondents will give shorter answers if they realize everyone needs to chime in on a particular question.

4) MINIMAL YES/NO QUESTIONS. Try to ask open ended questions, not ones that are easily confirmed or denied. (e.g. Not “Does this meet your needs?” but “Tell me how you would use this.”)

5) NO LEADING QUESTIONS. Moderating 101. But still the hardest thing to get right unless you are a pro. (e.g. Not “Does this look busy?” but “How do you feel about the amount of information on the screen?”)

6) DON’T FEEL YOU NEED TO ANSWER THEIR QUESTIONS. Remember, you’re from planet Mars. Say. “I don’t know. How do YOU think it works?”

7) FUNNEL OF VAGUENESS. Start broad (“ What is going on here?”) Gradually get more specific (“So is “Pizza Master” the right term for this sort of thing?”)

8) SHUT DOWN HEAVY TALKERS by not making eye contact. Off-topic or overly in-depth comments get the retort “That’s Another Focus Group.” Call on quiet respondents by name and control the flow.

9) KEEP EVERYONE FOCUSED ON THE STIMULI, NOT SPEAKING IN THE ABSTRACT Keep reminding them to tell you how they would improve things on the screen, page, etc., and ask them to imagine they are using it.

10) KEEP SOME SUGAR AND CAFFEINE READY Especially for evening groups, folks get tired. If a group exhibits signs of low energy, bust out the Mini-Snickers, the Mint Milanos, and the Diet Coke.

Ethnographic Insight: Manhattan Vets Store Their Extra Prescription Pet Food Where They Can

It’s been a great week in the field conducting ethnographic interviews with veterinarians and vet techs.  Mostly we’ve been in Manhattan, where food storage in the animal hospitals is a major issue.  We’ve seen bags of prescription chow stacked in hallways and offices, hidden in unused cages, stacked on surgical operating tables and x-ray equipment, and almost always taking over the waiting rooms and foyers of the client greeting area.  Someone should devise a drop-shipping or home delivery scheme wherein vets can earn their margins re-selling the supplies but don’t have to receive and store the inventory.

I doubt this idea will be used by my employer, which for this job is our scientific, technical, and medical publishing division, Wiley-Blackwell.   I’m lucky to work for a company that is committed to understanding its customers with this kind of research before embarking on a product idea.  My favorite study design of late for the early, product discovery phase of user research is to do 10-12 remote contextual inquiry interviews via phone and web conferencing software, followed up by 5-6 in-home or in-office ethnographic visits.   It’s a large enough of a sample to cover a couple of key audience segments and really learn their unmet needs, but still quite cost effective.  And even just a few visits out to a respondent’s real environment crystallizes the findings and brings the research to life.  We’ve refined the mechanics of fielding a study like this without breaking the bank, so let me share a few pointers with you.

Tip 1 – Skip The Video

Getting an edited video as an end result of ethnographic research is great – it’s like having an entertaining and informative documentary movie of the customer’s behavior.  But unfortunately it costs about the same as making a feature length documentary.  The editing time is the deal-breaker here.  You return from the field with a minimum of two to four hours of source footage for every respondent, which becomes two to four minutes of finished video in your report.   Getting from hours to minutes requires multiple passes at sequencing and culling the clips – moving from the ’select reel,’ to the ‘rough cut,’ to the ‘final cut’ – with multiple client and research team review milestones at each phase.   The media management alone (e.g. screening, logging, digitizing, and compressing clips) requires an experienced video editor to be anything less than overwhelming, so this kind of research is virtually always outsourced to specialist market research firms.  My advice – skip the video and bring only a portable DAT recorder and a small digital camera along with you.

Olympus DS-40 DAT Recorder

I love my Olympus DS-40 which I use with their ME-15 lavalier microphone (purchased separately).  This recording rig cost less than $150.   I press record, clip the mic to respondents lapel, drop the recorder into their pocket, and forget about it.  Then I use my point-and-shoot digital camera loaded up with a huge SD card and take photos of everything – including any screens on the computer we might look at.  The audio and pictures can even be combined in the final PowerPoint report for an experience that’s not that far from watching a video.   To pull this off, use transcripts of the audio recordings to identify clips you want to use in your report and have the transcription service pull the clips for you once you’ve identified what you want to use.  Transcripts are expensive, but transcribing a whole research study of 6-10 respondents is cheaper than one respondent’s worth of outsourced professional ethnography with video.

Tip 2 – Hire a Good Recruiting Firm

Don’t even think about doing your own recruiting, unless that’s your business.  Even if you have your own customer lists, turn them over to the recruiting service.   Finding, contacting, scheduling, and confirming qualified respondents for a user study requires constant attention and will become your main job for the 2-3 weeks before the research dates if you try and do it yourself.   Then when the study is over, you will be busy mailing out incentive checks.   Why not sit back and get daily updated spreadsheets of scheduled respondents instead?  Good recruiting firms will also help you write your screening questionairre, advise on the liklihood of finding who you are looking for, handle all participant communication including confirmation emails, reminder calls, etc., and disburse incentives.    Different firms operate in different regions, so it will require some legwork to find a good firm that covers the market where you are fielding your study.  Whenever possible, I try and use the amazingly responsive and organized folks at Focus Forward.  Or if you are using a full-service research firm like Sachs Insights or Catalyst Group Design (in NYC) or Serco (in London) you don’t need to worry about this.

Tip 3 – Ask for The Tour

Don’t take the trouble to visit a respondent in their own environment and then have an interview you could have had on the phone!   For in-home or in-office interviews I structure the discussion guide in a slightly different way than I would for an in-lab interview.  The key difference is that 15 minutes into the visit, after I’ve warmed up the respondent with a little chit chat and background info, I say “Hey, can you take us on a tour?”  This tour becomes the rest of the interview.   Start pointing at things, asking them to open cupboards and drawers, explaining as they go.  Write observation notes into your guide so you remember what sorts of things you should be looking for.  I’m usually interested in how things are organized and setup… what sorts of things wind up out on a table or desk vs. buried in a drawer?  How do they have their browser favorites folders set- up?  What sorts of ad-hoc customizing have they done to support their workflow? etc.   This may not be the fly-on-the-wall method of an anthropologist, but we’ve got a limited amount of time here and need to learn what we need to learn, so it’s time to force the agenda.

Tip 4 – Bring along a Prototype

Designs Always Feel Different Out in the Field

Theoretically, the design minded ethnographer should go into the field with few assumptions about what the final product will look like.  The main goal of any contextual phase of user research is to wind up with a good sense of the audience’s workflow and a list of their unmet needs.  Only after uncovering the customer’s unmet needs should the true product brainstorming begin.   But the reality is that we rarely have the time or budget for multiple rounds of discovery research.   If  we’re lucky, we’ll get the opportunity to get our ideas in front of customers once or twice before we have to commit to building something.   I like to bring concepts into the field.  If it’s for a website, which my work almost always is, I like to see mock-ups on the respondents’ real computer in their real environment and get them to talk about them.   What looks viable from the desk in our offices can be seen in an entirely different light in the field.     Load your sketches or design mock-ups on an extranet and access them from the respondent’s computer in the field.  If you’re interested in watching them react to real websites, create bookmarks on Delicious so you can find them quickly when you need them.

Tip 5 – Be Reflexive

The practical ethnographer still must be an effective and ethical ethnographer.   Most important of all is to exercise reflexivity, which for the field researcher, translates into the mental state of being aware of your own presence and impact on the environment that you are observing.   If there is a super star in the world of user anthropology, it is certainly Jan Chipchase – who vagabonds the world on behalf of Nokia.   Having recently heard Jan speak, I was impressed by his reflexive stance and sensitivity to his surroundings when he works in the field.   For instance, he always stays in the same places his respondents live (which for Jan, who works in the developing world extensively, is really saying something) not the Four Seasons across town.  I’m also pretty sure he doesn’t show up at the respondents’ homes with his iced grande frappacino and his Blackberry.   I recommend Mitch Duneier’s published ethnographic study of the Greenwich Village homeless street vendors, Sidewalk, to learn more about ethnographic field methods.  The appendices of this book alone are invaluable insights into how a truly reflexive researcher works.

Photo Credit: dgilder

Without a doubt, online surveys are the cheapest, easiest way to reach out to a large number of a website’s customers to gauge their opinion- but if not crafted correctly, they can result in misleading information.  You do not need to be a research expert to field a successful survey, but knowing the limitations of each option in your research toolkit is the only way to get accurate, impactful, and persuasive results. There are many issues to cover within this topic, but let’s begin with the basics by crafting a survey that successfully answers three key questions:

- Who is using the site?
- What do they think of it?
- Are they accomplishing what they want to?

If we can get accurate answers to these questions, then we can deduce what needs to change about the site in order to become a greater success.

The Three Basic Types of User Survey Questions

There are three main buckets of questions out of which to construct a simple user survey.   I usually draw from all three.  In future posts, I’ll get into more sophisticated questioning strategies – such as conjoint analysis – but for today let’s keep things simple.

Question Type 1 – Demographic or “Characteristic” (effectiveness: high)

This is basically any factual question about the respondent themselves or any aspect of  their life that occurs off of your site.   The temptation will be to put too much emphasis on this data when doing your analysis, slicing and dicing your results by a customers’ age, income, occupation, etc.  When it comes to user satisfaction – segmentation based on types of behavior is generally much more important than demographics.  But demographic data is always of interest to marketers and website visitors are not just users – they are customers – so we need this type of question in our survey.  And besides, this is a  great method for collecting this kind of information because it’s fast and cost effective to push out to a large, statistically reliable sample and the data is generally easy to analyse.  Typical  survey questions of this type:

• How old are users?
• What do they do for a living?
• Male/Female?
• Where do they live?
• How did they hear about your site or product?
• What sort of other products have they purchased in the past 6 months?

Question Type 2 – Attitudinal or “Satisfaction” (effectiveness: med)

Here we expand beyond the fact-based profiling of our audience into the subjective realm of attitudes and happiness – a treacherous area for those who set out to learn the Truth (with a capital T) about their customers.   The art of building a valid survey questionnaire starts to become of crucial importance here.   All sorts of mechanics can impact the results you get to these kinds of questions. Are the questions and answer choices worded in a way that is completely clear and unambiguous, without the use of “insider” terminology?  Are the questions ordered in such a way that might bias the results?  Are the question statements themselves leading in a way that might bias the answer? (e.g.  Do you think our website totally rocks, or is it just awesome?)  Typical survey questions of this type:

• To what degree do you consider this a primary resource for [insert something the website helps to get done here].
• How satisfied are you with your experience on the site? What would you change about it?
• Rate the following [your products or branded content types] on a scale of [x to x].

This type of question is even more powerful if you can collect attitudinal data about important attributes of your desired site, to help you troubleshoot where the problems are.  Write your attitudinal questions to cover both core brand promise traits as well as specific hypotheses you may have about improving the site.  Here’s an example of taking the attitudinals to the ‘next level’ of granularity:

How true are the following statements:

• The information on this site (or specific feature) is presented in a way that helps me learn
• The information seems relevant and up-to-date
• The information covers the topics I care about
• The use of visuals on this site aid my understanding of the materials
• The information on this site is unique and not available elsewhere on the web

Question Type 3 – Behavioral/Usage (effectiveness: low to med)

This is the least reliable category of questions for surveys, for two simple reasons.  1) surveys are self-reported instruments.  Nothing beats directly observing someone using the site and drawing your own conclusions.  2) it’s very difficult to communicate details about your site’s structure to someone taking a survey in a way that they can meaningfully respond to.  For instance, you might seek to distinguish their satisfaction with the use of your  ’search’ feature versus your primary ‘browsing’ or ‘navigation,’ but a typical user of your site may not make the same distinction that you (or other takers of the survey for that matter) do.  This sort of ambiguity of language is always a problem here, and will make the results hard to trust.  Note that the effectiveness of these sorts of questions goes up dramatically if you are using an intercept (or pop-up) survey, which asks them questions about that specific visit versus a general survey, which may have a respondent reflecting back about previous visits to the site.  Typical survey questions of this type:

• Accessing from home/work/school?
• What is your primary purpose for being here today?
• Did you  successfully complete your primary purpose?

What to Avoid

Don’t put any questions in the survey which can be attained or deduced easily in other ways.

Such as….

• How often do you come to the site? (Use web analytics software to get average repeat visits.)
• Which parts of the site do you use most frequently?  (Why should they know the names of the different parts of your site, or care?  Get this from the analytics.  If your goal is to link certain segments to certain traffic patterns, this can be done by integrating segmentation into your analytics program… but that’s beyond the scope of your simple survey!)

Don’t ask users to tell you which features you should build without making them prioritize!

Such as…

• I’m most interested in seeing…. [insert various features and content-types here].    (To pull this off, users need to prioritize the features and make trade-offs.  This technique is called conjoint analysis and is quite effective when used correctly.)

Don’t focus on specific feature ideas, focus on your user’s needs.

Such as…

• Do you want a blog? Do you want personalization?  (Instead, think about what needs these features address and then get a sense of whether your users feel that your site is missing these attributes.  use the ‘granular attitudinal’ approch mentioned in Question Type 2 above.)

The Absolute Minimum You Need to Know about Survey Validity

Sampling & Opt-In

Defining the appropriate sample (e.g. the group of people who are offered the survey) is a field unto itself.  But there are a few basic principles.

• Surveys are an annoyance to customers, don’t sample any more than you need to
• Samples should be “framed” or “stratified” in a way that models the overall audience (for instance, don’t oversample registered users because they are easier to reach.)
• Size of sample can be roughly 3% for med-trafficked sites (10,000 reachable customers) and 1% of total traffic for high-trafficked sites (100,000 reachable customers) (without getting into the math, this gives you about 95% confidence.)   If you are a very low trafficked site (1000 or less customers, plan to sample at least 15% of them.)
• Those numbers above apply to any audience you want to generalize against (e.g. if that’s total traffic, you can make statements about total traffic… to say something about a specific segment, you need those numbers for each segment.)
• Less known, but equally important, is opt-in rate.  If you get less than 3-5% opt-in (un-incentivized) to your survey invite – you should ditch all your results under the theory that those who answered are likely to be outliers.

Incentives

Offering incentives for survey participants (such as an entry into a sweepstakes or a free toaster) introduces a strong sampling bias that warrants a serious look.  I recommend avoiding incentives for any self-fielded survey. Consult a professional survey company that has their own panel they can reach out to or an expertise in sampling biases if you have trouble fielding an un-incentivized survey.

Completion Rate

Lastly, completion rate matters.  If a majority of your survey takers don’t finish the survey, that makes your results less trustworthy -again signaling that those who do manage to finish are likely to be outliers (e.g. people who love your site enough to put up with your 20 minute survey.)  Keep it Short!   5 minutes max. Be sure to be clear on the survey invite itself that this will take 5 minutes or less, then follow through with a single page survey (or a clear progress bar that promises a short experience.)

Further Reading:

I recommend Mike Kuniavsky’s excellent book, Observing the User Experience: A practitioner’s Guide to User Research, for those who want to dive into this topic in greater detail.