Proofiness is the mathematical version of "truthiness." It lurks in business, politics, media, and yes - user research.

If Charles Seife’s Proofiness has a lasting contribution to offer those in the fields of user experience, design, or even business, it will be in the elegant branding of its own subjectivist epistemology.  This, in itself, is no small victory.  It involves taking a complex debate on the origin of knowledge and in a single catchy word, turning it into a meme.  Picture a future where somebody whips out a clever piece of marketing research in a design or business meeting, something with lots of correlations and a confident sounding sample.   Maybe there is a scientific looking visualization,  like a scatterplot diagram with one of those Jackson Pollack splatters of microscopic pinpoints, something that screams data was collected here.  Its presenter starts speaking with the cajoling air of someone trying to impress the truth, with a capital ‘T,’ upon their audience.  And then suddenly, the attendees stand in protest and accuse their tormenter, in unison, of proofiness.

Then try to grasp what a profound departure that is from today’s climate of info digestion, where almost nothing is spit back if it smells and tastes like it was cooked up from numbers.   Jakob Nielsen, the founding father of pragmatism in HCI research, has condemned number fetishism in our field periodically since he came to prominence in the early 1990s – most exasperatingly in this 2004 post, The Risk of Quantitative Studies.   He writes, “…most statistical research is less credible than qualitative studies.  Design research is not like medical science.” In a 2009 post, five years later, little has changed.  He writes, “People still pay far more attention to questionable quantitative studies than they do to simpler qualitative studies that have much greater validity.”

If those of us in the innovation game are struggling with number fetishism, that is saying quite a lot. User and design research is about human cognition and emotion, pretty squishy topics really.  And design, which is generally carried out in the service of business, is an applied field.  What business person has the time to wrestle with taking the uncertainty factor below a certain threshold of instinctive confidence?  (The genius of Nielsen’s discount usability method was in demonstrating that closing in on the uncertainty factor has a long, expensive tail as it approaches zero – which, of course, it never does.  n=3 represents a lot of bang for the buck, and n=2000 a lot less so.) Why, then, are we so persuaded by numeric evidence?

Unlike the output from qualitative research studies, usually called insights, numbers have a primordial pull on us.  They are more valuable entities in social exchange, a fact which our lizard brains are fully aware of.  Numbers can be transmitted with less nuance, that is to say, less noise, and are therefore choice currency.   Seife provides evidence that human beings are capable of reacting to numbers from somewhere biologically deep, processing them somewhere in the muscles and tissue and nerves but not necessarily with the intellect.  “No matter how idiotic, how unbelievable an idea is, numbers can give it credibility.” (p.8)  He cites an example of MSNBC host Deborah Norville reporting with a straight face that 58% of all exercise done in America is broadcast on television.  3.5 billion situps were done in 2003, she reported.  Two million and 300,000 of those on exercise shows.  “The numbers had short-circuited Norville’s brain,” Siefe notes, “rendering her completely incapable of critical thought.”

Our lust for numbers is like our lust for sweet and fatty foods. Two million years of evolution have taught us to crave them.  But in the modern world, where they are not only in abundance, but easily manipulated and processed with the explicit goal of satiating us, they are dangerous.   The author lays out, in clear language, the heart of his epistemic argument.  Once a number tries to describe the real world, or “acquires a unit” in Seife’s language, it loses its purity.   There is always a measurement bias of one sort or another and therefore it can no longer inhabit the “platonic realm of absolute truth.”(p.10) He writes this without apologizing for it or acknowledging its provocativeness. But this is no mainstream view. This is a rejection of the very idea of structure or universal truth, a return to the Dionysian notion that the sublime lies in closeness of experience and not in critical distance. This is the sort of postmodern thinking that was radical in intellectual circles as recently as the 1960s.  This is Piaget.  This is Derrida.  And maybe because of that inaccessibility, this sort of thinking is still far from being absorbed into the fabric of our daily thoughts and culture in business life.

Enter proofiness.  A great term, a self-descriptive masterpiece of nomenclature. Its dubious etymological structure (with its comic closeness to being a real word, a science word) carries its actual semantic argument.  Of course, the author, Charles Seife, had some help in this.  The word is a clear homage to television comic, Stephen Colbert’s, famous neologism “truthiness.”  But it is Seife who brings it to the concept of proof, not truth.  Truthiness, which applies to information that has the patina of truth, is pure social criticism.  Proofiness, which also applies to information that has the patina of truth, is an attack on status quo views of ontological reality.  Truthiness, like the Bushism ‘strategery,’is aimed at easy targets and misanthropes- liars, manipulators, unilateralists, oversimplifiers.  Proofiness is aimed at all of us with the instinct to prove, and therefore at the natural condition of mankind itself.  “Proofiness has power over us because we’re blind to this impurity (of numbers).  Numbers, charts, graphs all have an aura of perfection.”  Seife’s argument, in short, is the more important one.

The book’s tagline, “The Dark Arts of Mathematical Deception,” implies that the book is about a sort of numeric version of truthiness, some sort of number trickery done with intent to mislead.  In fact, the description of various mathematical anti-patterns ends by about page 40.  These are interesting in their own right.  Potempkin numbers are those which are built out of data that only looks like real data, data based on nonsensical or made-up measurements, such as IQ or the crowd estimates taken at free outdoor concerts.   Disestimation is the act of taking numbers more literally than the uncertainty surrounding them would seem to warrant.  (Seife’s example is the museum docent who dates the brontosaurus bones to 65,000,038 years. What’s the point of including the final 38 years when the error margin on such an estimate is likely in the tens of millions or hundreds of millions of years?)  By page 26, we’re on to fruit-packing and cherry-picking, which refers to the act of ignoring or obscuring the data that fails to support a hypothesis or argument.  The book is already drifting away from mathematics and is onto basic research ethics with 275 pages to go.

Despite the fact that there are the usual warnings here about the pitfalls of overlooking co-variants and making specious correlations that one would see in any tome on statistics, the book isn’t about math.  It’s about numbers. And to Seife, numbers represent powerful disinformation, unfairly persuasive rhetoric.   There is very little statistical depth or discussion of mathematical techniques in this book, other than to say, what is the point of doing statistical slicing and dicing on something that is fictional to begin with?  When Seife explains the concept of margin of error in political polling, which uses statistical formulas to harness some pretty nifty laws of nature in order to determine the amount of random weirdness likely to be in any sample, he doesn’t criticize the math, or even accuse the journalists who cite the polls of being innumerate.  He is more offended by those who accept the mathematical veracity in such a way that they don’t question basic systematic errors in the logic behind the poll’s construction and execution, such as who was sampled or what were they asked and in what way?(p.102)

Particularly revealing of Seife’s lack of concern with actual number manipulation is in his principle of causuistry.  “Casuistry – without the extra ‘u’ – is the art of making a misleading argument through seemingly sound principles.  Causuistry is a specialized form of casuistry where the fault in the argument comes from implying that there is a causal relationship between two things when in fact there isn’t such a linkage.”  He breaks this out as a separate idea from the statistical concept of regression analysis, used to prove causation between multiple independently correlated values. When Seife accuses someone of causuistry he isn’t concerned by the bad math on display, he’s offended by the sheer nerve that someone would try and build an argument out of it.

This gets back to my opening statement.  If the book is to make a lasting contribution outside of journalism or civics (Seife teaches journalism and the book is unfortunately heavy-handed with examples about electoral polling, elections, law and politics.  There are virtually no examples from the world of business or private life), it will require people to start taking this proofiness thesis to heart.  The concept instantly resonated with me.  Numbers, the coldest, hardest facts of all, are twisted and manipulated in order to add an air of proof to some act of data collection.   This touches right to the core of what are the least obvious but most insidious sources of user research failures: those that are related to epistemological hubris and the act of using research to validate ideas rather than to enrich them.

Where research fails most often is when the intent of it is misused.   The more you know about research methodologies, the more aware you are of their inevitable flaws.  The bad researchers are invariably the objectivists, the ones who arrogantly presume to be reporting on reality.   It is in this spirit of hubris that virtually all unforgivable research mistakes are made.     For instance, if you have just conducted a survey and concluded that 85% of potential customers liked your idea for an online information website, I will tell you that you have probably wasted your time.   Most likely your survey was flawed, your sample was flawed, or you created some other sort of systematic error and you cannot make that claim.   You have committed an act of proofiness.   And in the meantime, instead of using that opportunity to talk to your customers to learn something that might enrich your idea (which you always knew you were going to do anyways), you have squandered it trying to convince yourself and others that the idea is less risky than it probably is.   Concentrate on delivering insights, not validation.

Once, I worked as a usability researcher for two arch competitors in the same retailer category at roughly the same time.  Their contrasting styles will always stick with me.  One had a usability manager that emphasized insights.  The other had an old school HCI guy that emphasized Truth (capital T is no accident here) and experimental control.   When interviewing for the first client, I would tell the participant that I didn’t know where this interview was going but I was interested in partnering with them to try and understand what it is like to use this website, how it fits into their life, and how it could be improved.  When interviewing for the second client, I would give the participant a pre-filled out card with a task on it and watch them try and complete the task while somebody from the retailer’s staff timed them in the back room.    I barely opened my mouth for the second client, lest I contaminate the experimental conditions.    The first client had a progressive stance about the research, understood its limitations and was focused on collecting inspiration to refine their design in ways that might resonate with its customer base.  The second client wanted validation only for its existing “agreed upon” design, and set up a priori experimental conditions to get at it.   It is worth noting that for reasons unrelated to this particular usability study, the second client unexpectedly and spectacularly went out of business several years later.  The first one is thriving.   I think it goes to show that proofiness never prospers.

If a subject is steadily verbalizing while performing a task (e.g. concurrent verbalization), they are assumed to be speaking from short term memory.  Dumas & Redish (1993) conveniently summarize three levels of think-aloud protocols commonly referred to by HCI researchers:  Level 1 verbalizations:  where the emphasis is on pure thoughts with no or minimal explanations; Level 2 verbalizations:  same, but when the participant is dealing with non-verbal information, like shapes, which must be internally “coded” in order to be articulated verbally, and Level 3 verbalizations:  or “thinking plus explanations.”  The latter are also referred to as retrospective reports, because the respondent is recapping and opining about what they actually did earlier.  Dumas summarizes the distinction between Level 3 and the other levels as the stage where the researchers are “no longer getting a read out of short-term memory… rather it is the interpretation of the process they are using or the reasons they have selected a strategy.”  Retrospective reports are not useless to the user researcher, in fact they are necessary to clarify a respondent’s statements and actions, but they are far less valid.

This leads us to the question of active vs. inactive moderation.  In an inactive moderation scenario, the emphasis is on experimental control and creating a unified experience for all test subjects.  This is the old-school style, where the researchers stand behind the glass and the respondent sits in the room by themselves talking out loud like a crazy person.  Participants, faced with the unnatural task of constant verbalization, are typically “coached” on how to deliver a think-aloud protocol.  This often takes the form of a warm-up exercise where the participant and experimenter practice thinking aloud with non-related stimuli, preferably a simple game such as tic-tac-toe (in order to place emphasis on cognitive strategy.)  During the actual experiment, the moderator prompts only when the participant ceases to verbalize: “Please keep talking,” they say.  And that’s all they say!

In a high moderator intervention scenario, the experimenter employs probing questions to focus the participant’s attention on particular features or to elicit and clarify subjective explanations of their behavior.  Moderators are skilled in asking neutral, non-leading questions to minimize bias.  In addition, active listening techniques are employed to emulate the clinician’s empathic stance. This implies paraphrasing of a speaker’s comments to ensure them that they have been listened to, noted, and understood as well as other verbal and non-verbal forms of caring, non-judgmental acceptance.  Mike Kuniavsky, whose work is mentioned frequently in these pages,  lays out the guidelines for “non-directed” interviewing:  questions should be concentrated on immediate experience, nonjudgmental, focused on a single topic, open-ended, and non-binary (e.g. yes-no, true-false).

In a quick look at the literature, you’ll see that Taylor and Dionne (2000) suggest that probes are best deployed for collection and verification of data in retrospective reports, and that they have a detrimental impact on validity if used during concurrent think-aloud protocols.  Preece (1994) suggests that the role of the moderator on the participant is both interruptive and imposes additional cognitive load.  Nielsen (1993) is pragmatic on the subject, suggesting the moderator intervenes as little as possible, yet directing the flow and direction of the interview to maximize the number of usability issues found.   Nielsen is the spiritual father of the commercial usability field, so it’s no surprise that most usability is performed with this degree of pragmatism.  I think it was Jared Spool who once commented that you can only watch participants avoiding clicking on the red button so many times before you are compelled to intervene and ask why.   Plus, most clients of usability research do not share the academic’s interest in validity.  They want enough validity to feel good about the process and the results, but ultimately they want their specific questions answered for a reasonable amount of time and expense.  In Nielsen et. al.’s (2002) interpretation, The human is a psychological being engaged in a psychological interaction, which cannot be reduced to that which is concurrently verbalized.

Some usability researchers, characterized by Whiteside, et. al. (1993), have also posited that observing a user’s behavior is not enough to understand what is happening in terms of higher order thinking and cognitive strategy.  If you set up a priori conditions then you are bounding yourself to learning only what falls within those conditions.  In this view, subjective experience is the most comprehensive criterion for understanding usability.  Concerns of generalizability are side-stepped: the goal is to obtain rich, experiential data.  Since specific questions are the only effective way to elicit and clarify mental models, the researcher must come to terms with at least a partially subjectivist stance. (Tamler, 2001)  This approach values reflexivity:  the subject is full participant in the study, leading the research into relevant areas for exploration as well giving the subject the ability to respond to the researchers interpretations. It is not a search for “truth,” per se, as a philosophical underpinning of this method of inquiry.  The assumption is that an expansion of perspectives leads to the exposure of more aspects of learning.

Here’s some References (I know, for a blog post, it’s over the top… but this stuff’s important!)

DUMAS, J.S. (2001) “Usability Testing Methods:  Think-Aloud Protocols,” in Design by People For People:  Essays on Usability, UPA,  pp 119-129

DUMAS, J.S. & REDISH, J.C., (1993)  A Practical Guide to Usability Testing.  Norwood, NJ, Ablex Publishing Corp.

ERICSSON, K.A. & SIMON, H.A. (1984, 1993) Protocol analysis:  Verbal reports as data (Rev. ed).  Cambridge, MA:  MIT Press

KUNIAVSKY, M. (2003)  Observing the User Experience: A Practitioner’s Guide to User Research,  San Francisco: Morgan Kaufmann Publishers, Inc.

NIELSEN, J., (1993) Usability Engineering.  Chestnut Hill, MA:  Academic Press, Inc.

NIELSEN, J., CLEMMENSEN, T., & YSSING, C., (2002)  “Getting access to what goes on in people’s heads? – Reflections on the think-aloud technique”, paper presented to NordiCHI, Arhus, Denmark, October 19-23

PREECE, J. (1994), Human-Computer Interaction, Addison-Wesley, England

TAMLER, H. (2001) “How (Much) to Intervene in a Usability Testing Session,”  in Design by People For People:  Essays on Usability, UPA,  pp 165-171

WHITESIDE, J., BENNETT., J.L., & HOLTZBLATT, K., (1988)  “Usability Engineering: Our Experience and Evolution,” in Handbook of Human Computer Interaction; edited by Helander, M.  New York, NY:  Elsevier Science Publishers

Photo Credit: curiouslee

Focus groups are probably the most difficult moderating challenge.  And I know this because I was terrible at conducting them until I learned a few tricks…

1) POLL the room frequently. Recap (or let the respondent recap) and then ask the room to raise their hands for who agrees. For those who disagree, ask them to state why? Don’t forget, observers usually want to understand what the group thinks about a certain question, and polling delivers.

2) ASK FOR COUNTERARGUMENTS. Control for ‘groupthink’ by asking “C’mon, Doesn’t anybody want to argue the opposite point?”

3) GO AROUND THE TABLE every so often. Periodically go around the table letting each person answer the question in turn. This will distribute the talking around the room, allow you to control the flow, and respondents will give shorter answers if they realize everyone needs to chime in on a particular question.

4) MINIMAL YES/NO QUESTIONS. Try to ask open ended questions, not ones that are easily confirmed or denied. (e.g. Not “Does this meet your needs?” but “Tell me how you would use this.”)

5) NO LEADING QUESTIONS. Moderating 101. But still the hardest thing to get right unless you are a pro. (e.g. Not “Does this look busy?” but “How do you feel about the amount of information on the screen?”)

6) DON’T FEEL YOU NEED TO ANSWER THEIR QUESTIONS. Remember, you’re from planet Mars. Say. “I don’t know. How do YOU think it works?”

7) FUNNEL OF VAGUENESS. Start broad (“ What is going on here?”) Gradually get more specific (“So is “Pizza Master” the right term for this sort of thing?”)

8) SHUT DOWN HEAVY TALKERS by not making eye contact. Off-topic or overly in-depth comments get the retort “That’s Another Focus Group.” Call on quiet respondents by name and control the flow.

9) KEEP EVERYONE FOCUSED ON THE STIMULI, NOT SPEAKING IN THE ABSTRACT Keep reminding them to tell you how they would improve things on the screen, page, etc., and ask them to imagine they are using it.

10) KEEP SOME SUGAR AND CAFFEINE READY Especially for evening groups, folks get tired. If a group exhibits signs of low energy, bust out the Mini-Snickers, the Mint Milanos, and the Diet Coke.

Ethnographic Insight: Manhattan Vets Store Their Extra Prescription Pet Food Where They Can

It’s been a great week in the field conducting ethnographic interviews with veterinarians and vet techs.  Mostly we’ve been in Manhattan, where food storage in the animal hospitals is a major issue.  We’ve seen bags of prescription chow stacked in hallways and offices, hidden in unused cages, stacked on surgical operating tables and x-ray equipment, and almost always taking over the waiting rooms and foyers of the client greeting area.  Someone should devise a drop-shipping or home delivery scheme wherein vets can earn their margins re-selling the supplies but don’t have to receive and store the inventory.

I doubt this idea will be used by my employer, which for this job is our scientific, technical, and medical publishing division, Wiley-Blackwell.   I’m lucky to work for a company that is committed to understanding its customers with this kind of research before embarking on a product idea.  My favorite study design of late for the early, product discovery phase of user research is to do 10-12 remote contextual inquiry interviews via phone and web conferencing software, followed up by 5-6 in-home or in-office ethnographic visits.   It’s a large enough of a sample to cover a couple of key audience segments and really learn their unmet needs, but still quite cost effective.  And even just a few visits out to a respondent’s real environment crystallizes the findings and brings the research to life.  We’ve refined the mechanics of fielding a study like this without breaking the bank, so let me share a few pointers with you.

Tip 1 – Skip The Video

Getting an edited video as an end result of ethnographic research is great – it’s like having an entertaining and informative documentary movie of the customer’s behavior.  But unfortunately it costs about the same as making a feature length documentary.  The editing time is the deal-breaker here.  You return from the field with a minimum of two to four hours of source footage for every respondent, which becomes two to four minutes of finished video in your report.   Getting from hours to minutes requires multiple passes at sequencing and culling the clips – moving from the ’select reel,’ to the ‘rough cut,’ to the ‘final cut’ – with multiple client and research team review milestones at each phase.   The media management alone (e.g. screening, logging, digitizing, and compressing clips) requires an experienced video editor to be anything less than overwhelming, so this kind of research is virtually always outsourced to specialist market research firms.  My advice – skip the video and bring only a portable DAT recorder and a small digital camera along with you.

Olympus DS-40 DAT Recorder

I love my Olympus DS-40 which I use with their ME-15 lavalier microphone (purchased separately).  This recording rig cost less than $150.   I press record, clip the mic to respondents lapel, drop the recorder into their pocket, and forget about it.  Then I use my point-and-shoot digital camera loaded up with a huge SD card and take photos of everything – including any screens on the computer we might look at.  The audio and pictures can even be combined in the final PowerPoint report for an experience that’s not that far from watching a video.   To pull this off, use transcripts of the audio recordings to identify clips you want to use in your report and have the transcription service pull the clips for you once you’ve identified what you want to use.  Transcripts are expensive, but transcribing a whole research study of 6-10 respondents is cheaper than one respondent’s worth of outsourced professional ethnography with video.

Tip 2 – Hire a Good Recruiting Firm

Don’t even think about doing your own recruiting, unless that’s your business.  Even if you have your own customer lists, turn them over to the recruiting service.   Finding, contacting, scheduling, and confirming qualified respondents for a user study requires constant attention and will become your main job for the 2-3 weeks before the research dates if you try and do it yourself.   Then when the study is over, you will be busy mailing out incentive checks.   Why not sit back and get daily updated spreadsheets of scheduled respondents instead?  Good recruiting firms will also help you write your screening questionairre, advise on the liklihood of finding who you are looking for, handle all participant communication including confirmation emails, reminder calls, etc., and disburse incentives.    Different firms operate in different regions, so it will require some legwork to find a good firm that covers the market where you are fielding your study.  Whenever possible, I try and use the amazingly responsive and organized folks at Focus Forward.  Or if you are using a full-service research firm like Sachs Insights or Catalyst Group Design (in NYC) or Serco (in London) you don’t need to worry about this.

Tip 3 – Ask for The Tour

Don’t take the trouble to visit a respondent in their own environment and then have an interview you could have had on the phone!   For in-home or in-office interviews I structure the discussion guide in a slightly different way than I would for an in-lab interview.  The key difference is that 15 minutes into the visit, after I’ve warmed up the respondent with a little chit chat and background info, I say “Hey, can you take us on a tour?”  This tour becomes the rest of the interview.   Start pointing at things, asking them to open cupboards and drawers, explaining as they go.  Write observation notes into your guide so you remember what sorts of things you should be looking for.  I’m usually interested in how things are organized and setup… what sorts of things wind up out on a table or desk vs. buried in a drawer?  How do they have their browser favorites folders set- up?  What sorts of ad-hoc customizing have they done to support their workflow? etc.   This may not be the fly-on-the-wall method of an anthropologist, but we’ve got a limited amount of time here and need to learn what we need to learn, so it’s time to force the agenda.

Tip 4 – Bring along a Prototype

Designs Always Feel Different Out in the Field

Theoretically, the design minded ethnographer should go into the field with few assumptions about what the final product will look like.  The main goal of any contextual phase of user research is to wind up with a good sense of the audience’s workflow and a list of their unmet needs.  Only after uncovering the customer’s unmet needs should the true product brainstorming begin.   But the reality is that we rarely have the time or budget for multiple rounds of discovery research.   If  we’re lucky, we’ll get the opportunity to get our ideas in front of customers once or twice before we have to commit to building something.   I like to bring concepts into the field.  If it’s for a website, which my work almost always is, I like to see mock-ups on the respondents’ real computer in their real environment and get them to talk about them.   What looks viable from the desk in our offices can be seen in an entirely different light in the field.     Load your sketches or design mock-ups on an extranet and access them from the respondent’s computer in the field.  If you’re interested in watching them react to real websites, create bookmarks on Delicious so you can find them quickly when you need them.

Tip 5 – Be Reflexive

The practical ethnographer still must be an effective and ethical ethnographer.   Most important of all is to exercise reflexivity, which for the field researcher, translates into the mental state of being aware of your own presence and impact on the environment that you are observing.   If there is a super star in the world of user anthropology, it is certainly Jan Chipchase – who vagabonds the world on behalf of Nokia.   Having recently heard Jan speak, I was impressed by his reflexive stance and sensitivity to his surroundings when he works in the field.   For instance, he always stays in the same places his respondents live (which for Jan, who works in the developing world extensively, is really saying something) not the Four Seasons across town.  I’m also pretty sure he doesn’t show up at the respondents’ homes with his iced grande frappacino and his Blackberry.   I recommend Mitch Duneier’s published ethnographic study of the Greenwich Village homeless street vendors, Sidewalk, to learn more about ethnographic field methods.  The appendices of this book alone are invaluable insights into how a truly reflexive researcher works.

Photo Credit: dgilder

Without a doubt, online surveys are the cheapest, easiest way to reach out to a large number of a website’s customers to gauge their opinion- but if not crafted correctly, they can result in misleading information.  You do not need to be a research expert to field a successful survey, but knowing the limitations of each option in your research toolkit is the only way to get accurate, impactful, and persuasive results. There are many issues to cover within this topic, but let’s begin with the basics by crafting a survey that successfully answers three key questions:

- Who is using the site?
- What do they think of it?
- Are they accomplishing what they want to?

If we can get accurate answers to these questions, then we can deduce what needs to change about the site in order to become a greater success.

The Three Basic Types of User Survey Questions

There are three main buckets of questions out of which to construct a simple user survey.   I usually draw from all three.  In future posts, I’ll get into more sophisticated questioning strategies – such as conjoint analysis – but for today let’s keep things simple.

Question Type 1 – Demographic or “Characteristic” (effectiveness: high)

This is basically any factual question about the respondent themselves or any aspect of  their life that occurs off of your site.   The temptation will be to put too much emphasis on this data when doing your analysis, slicing and dicing your results by a customers’ age, income, occupation, etc.  When it comes to user satisfaction – segmentation based on types of behavior is generally much more important than demographics.  But demographic data is always of interest to marketers and website visitors are not just users – they are customers – so we need this type of question in our survey.  And besides, this is a  great method for collecting this kind of information because it’s fast and cost effective to push out to a large, statistically reliable sample and the data is generally easy to analyse.  Typical  survey questions of this type:

• How old are users?
• What do they do for a living?
• Male/Female?
• Where do they live?
• How did they hear about your site or product?
• What sort of other products have they purchased in the past 6 months?

Question Type 2 – Attitudinal or “Satisfaction” (effectiveness: med)

Here we expand beyond the fact-based profiling of our audience into the subjective realm of attitudes and happiness – a treacherous area for those who set out to learn the Truth (with a capital T) about their customers.   The art of building a valid survey questionnaire starts to become of crucial importance here.   All sorts of mechanics can impact the results you get to these kinds of questions. Are the questions and answer choices worded in a way that is completely clear and unambiguous, without the use of “insider” terminology?  Are the questions ordered in such a way that might bias the results?  Are the question statements themselves leading in a way that might bias the answer? (e.g.  Do you think our website totally rocks, or is it just awesome?)  Typical survey questions of this type:

• To what degree do you consider this a primary resource for [insert something the website helps to get done here].
• How satisfied are you with your experience on the site? What would you change about it?
• Rate the following [your products or branded content types] on a scale of [x to x].

This type of question is even more powerful if you can collect attitudinal data about important attributes of your desired site, to help you troubleshoot where the problems are.  Write your attitudinal questions to cover both core brand promise traits as well as specific hypotheses you may have about improving the site.  Here’s an example of taking the attitudinals to the ‘next level’ of granularity:

How true are the following statements:

• The information on this site (or specific feature) is presented in a way that helps me learn
• The information seems relevant and up-to-date
• The information covers the topics I care about
• The use of visuals on this site aid my understanding of the materials
• The information on this site is unique and not available elsewhere on the web

Question Type 3 – Behavioral/Usage (effectiveness: low to med)

This is the least reliable category of questions for surveys, for two simple reasons.  1) surveys are self-reported instruments.  Nothing beats directly observing someone using the site and drawing your own conclusions.  2) it’s very difficult to communicate details about your site’s structure to someone taking a survey in a way that they can meaningfully respond to.  For instance, you might seek to distinguish their satisfaction with the use of your  ’search’ feature versus your primary ‘browsing’ or ‘navigation,’ but a typical user of your site may not make the same distinction that you (or other takers of the survey for that matter) do.  This sort of ambiguity of language is always a problem here, and will make the results hard to trust.  Note that the effectiveness of these sorts of questions goes up dramatically if you are using an intercept (or pop-up) survey, which asks them questions about that specific visit versus a general survey, which may have a respondent reflecting back about previous visits to the site.  Typical survey questions of this type:

• Accessing from home/work/school?
• What is your primary purpose for being here today?
• Did you  successfully complete your primary purpose?

What to Avoid

Don’t put any questions in the survey which can be attained or deduced easily in other ways.

Such as….

• How often do you come to the site? (Use web analytics software to get average repeat visits.)
• Which parts of the site do you use most frequently?  (Why should they know the names of the different parts of your site, or care?  Get this from the analytics.  If your goal is to link certain segments to certain traffic patterns, this can be done by integrating segmentation into your analytics program… but that’s beyond the scope of your simple survey!)

Don’t ask users to tell you which features you should build without making them prioritize!

Such as…

• I’m most interested in seeing…. [insert various features and content-types here].    (To pull this off, users need to prioritize the features and make trade-offs.  This technique is called conjoint analysis and is quite effective when used correctly.)

Don’t focus on specific feature ideas, focus on your user’s needs.

Such as…

• Do you want a blog? Do you want personalization?  (Instead, think about what needs these features address and then get a sense of whether your users feel that your site is missing these attributes.  use the ‘granular attitudinal’ approch mentioned in Question Type 2 above.)

The Absolute Minimum You Need to Know about Survey Validity

Sampling & Opt-In

Defining the appropriate sample (e.g. the group of people who are offered the survey) is a field unto itself.  But there are a few basic principles.

• Surveys are an annoyance to customers, don’t sample any more than you need to
• Samples should be “framed” or “stratified” in a way that models the overall audience (for instance, don’t oversample registered users because they are easier to reach.)
• Size of sample can be roughly 3% for med-trafficked sites (10,000 reachable customers) and 1% of total traffic for high-trafficked sites (100,000 reachable customers) (without getting into the math, this gives you about 95% confidence.)   If you are a very low trafficked site (1000 or less customers, plan to sample at least 15% of them.)
• Those numbers above apply to any audience you want to generalize against (e.g. if that’s total traffic, you can make statements about total traffic… to say something about a specific segment, you need those numbers for each segment.)
• Less known, but equally important, is opt-in rate.  If you get less than 3-5% opt-in (un-incentivized) to your survey invite – you should ditch all your results under the theory that those who answered are likely to be outliers.

Incentives

Offering incentives for survey participants (such as an entry into a sweepstakes or a free toaster) introduces a strong sampling bias that warrants a serious look.  I recommend avoiding incentives for any self-fielded survey. Consult a professional survey company that has their own panel they can reach out to or an expertise in sampling biases if you have trouble fielding an un-incentivized survey.

Completion Rate

Lastly, completion rate matters.  If a majority of your survey takers don’t finish the survey, that makes your results less trustworthy -again signaling that those who do manage to finish are likely to be outliers (e.g. people who love your site enough to put up with your 20 minute survey.)  Keep it Short!   5 minutes max. Be sure to be clear on the survey invite itself that this will take 5 minutes or less, then follow through with a single page survey (or a clear progress bar that promises a short experience.)

Further Reading:

I recommend Mike Kuniavsky’s excellent book, Observing the User Experience: A practitioner’s Guide to User Research, for those who want to dive into this topic in greater detail.