Crafting Simple, Effective User Surveys

Photo Credit: dgilder

Without a doubt, online surveys are the cheapest, easiest way to reach out to a large number of a website’s customers to gauge their opinion- but if not crafted correctly, they can result in misleading information. You do not need to be a research expert to field a successful survey, but knowing the limitations of each option in your research toolkit is the only way to get accurate, impactful, and persuasive results. There are many issues to cover within this topic, but let’s begin with the basics by crafting a survey that successfully answers three key questions:

- Who is using the site?
- What do they think of it?
- Are they accomplishing what they want to?

If we can get accurate answers to these questions, then we can deduce what needs to change about the site in order to become a greater success.

The Three Basic Types of User Survey Questions

There are three main buckets of questions out of which to construct a simple user survey. I usually draw from all three. In future posts, I’ll get into more sophisticated questioning strategies – such as conjoint analysis – but for today let’s keep things simple.

Question Type 1 – Demographic or “Characteristic” (effectiveness: high)

This is basically any factual question about the respondent themselves or any aspect of their life that occurs off of your site. The temptation will be to put too much emphasis on this data when doing your analysis, slicing and dicing your results by a customers’ age, income, occupation, etc. When it comes to user satisfaction – segmentation based on types of behavior is generally much more important than demographics. But demographic data is always of interest to marketers and website visitors are not just users – they are customers – so we need this type of question in our survey. And besides, this is a great method for collecting this kind of information because it’s fast and cost effective to push out to a large, statistically reliable sample and the data is generally easy to analyse. Typical survey questions of this type:

How old are users?
What do they do for a living?
Male/Female?
Where do they live?
How did they hear about your site or product?
What sort of other products have they purchased in the past 6 months?

Question Type 2 – Attitudinal or “Satisfaction” (effectiveness: med)

Here we expand beyond the fact-based profiling of our audience into the subjective realm of attitudes and happiness – a treacherous area for those who set out to learn the Truth (with a capital T) about their customers. The art of building a valid survey questionnaire starts to become of crucial importance here. All sorts of mechanics can impact the results you get to these kinds of questions. Are the questions and answer choices worded in a way that is completely clear and unambiguous, without the use of “insider” terminology? Are the questions ordered in such a way that might bias the results? Are the question statements themselves leading in a way that might bias the answer? (e.g. Do you think our website totally rocks, or is it just awesome?) Typical survey questions of this type:

To what degree do you consider this a primary resource for [insert something the website helps to get done here].
How satisfied are you with your experience on the site? What would you change about it?
Rate the following [your products or branded content types] on a scale of [x to x].

This type of question is even more powerful if you can collect attitudinal data about important attributes of your desired site, to help you troubleshoot where the problems are. Write your attitudinal questions to cover both core brand promise traits as well as specific hypotheses you may have about improving the site. Here’s an example of taking the attitudinals to the ‘next level’ of granularity:

How true are the following statements:

The information on this site (or specific feature) is presented in a way that helps me learn
The information seems relevant and up-to-date
The information covers the topics I care about
The use of visuals on this site aid my understanding of the materials
The information on this site is unique and not available elsewhere on the web

Question Type 3 – Behavioral/Usage (effectiveness: low to med)

This is the least reliable category of questions for surveys, for two simple reasons. 1) surveys are self-reported instruments. Nothing beats directly observing someone using the site and drawing your own conclusions. 2) it’s very difficult to communicate details about your site’s structure to someone taking a survey in a way that they can meaningfully respond to. For instance, you might seek to distinguish their satisfaction with the use of your ’search’ feature versus your primary ‘browsing’ or ‘navigation,’ but a typical user of your site may not make the same distinction that you (or other takers of the survey for that matter) do. This sort of ambiguity of language is always a problem here, and will make the results hard to trust. Note that the effectiveness of these sorts of questions goes up dramatically if you are using an intercept (or pop-up) survey, which asks them questions about that specific visit versus a general survey, which may have a respondent reflecting back about previous visits to the site. Typical survey questions of this type:

Accessing from home/work/school?
What is your primary purpose for being here today?
Did you successfully complete your primary purpose?

What to Avoid

Don’t put any questions in the survey which can be attained or deduced easily in other ways.

Such as….

How often do you come to the site? (Use web analytics software to get average repeat visits.)
Which parts of the site do you use most frequently? (Why should they know the names of the different parts of your site, or care? Get this from the analytics. If your goal is to link certain segments to certain traffic patterns, this can be done by integrating segmentation into your analytics program… but that’s beyond the scope of your simple survey!)

Don’t ask users to tell you which features you should build without making them prioritize!

Such as…

I’m most interested in seeing…. [insert various features and content-types here]. (To pull this off, users need to prioritize the features and make trade-offs. This technique is called conjoint analysis and is quite effective when used correctly.)

Don’t focus on specific feature ideas, focus on your user’s needs.

Such as…

Do you want a blog? Do you want personalization? (Instead, think about what needs these features address and then get a sense of whether your users feel that your site is missing these attributes. use the ‘granular attitudinal’ approch mentioned in Question Type 2 above.)

The Absolute Minimum You Need to Know about Survey Validity

Sampling & Opt-In

Defining the appropriate sample (e.g. the group of people who are offered the survey) is a field unto itself. But there are a few basic principles.

Surveys are an annoyance to customers, don’t sample any more than you need to
Samples should be “framed” or “stratified” in a way that models the overall audience (for instance, don’t oversample registered users because they are easier to reach.)
Size of sample can be roughly 3% for med-trafficked sites (10,000 reachable customers) and 1% of total traffic for high-trafficked sites (100,000 reachable customers) (without getting into the math, this gives you about 95% confidence.) If you are a very low trafficked site (1000 or less customers, plan to sample at least 15% of them.)
Those numbers above apply to any audience you want to generalize against (e.g. if that’s total traffic, you can make statements about total traffic… to say something about a specific segment, you need those numbers for each segment.)
Less known, but equally important, is opt-in rate. If you get less than 3-5% opt-in (un-incentivized) to your survey invite – you should ditch all your results under the theory that those who answered are likely to be outliers.

Incentives

Offering incentives for survey participants (such as an entry into a sweepstakes or a free toaster) introduces a strong sampling bias that warrants a serious look. I recommend avoiding incentives for any self-fielded survey. Consult a professional survey company that has their own panel they can reach out to or an expertise in sampling biases if you have trouble fielding an un-incentivized survey.

Completion Rate

Lastly, completion rate matters. If a majority of your survey takers don’t finish the survey, that makes your results less trustworthy -again signaling that those who do manage to finish are likely to be outliers (e.g. people who love your site enough to put up with your 20 minute survey.) Keep it Short! 5 minutes max. Be sure to be clear on the survey invite itself that this will take 5 minutes or less, then follow through with a single page survey (or a clear progress bar that promises a short experience.)

Solid State UX