Do online surveys bore respondents and damage data quality?

[Matt Rhodes]

Research from Engage and GMI claims that high drop-out rates compromise the quality of both sample and data. The study compared drop-out rates from over 550 surveys, asked a sample of 200 online panellists what bored and frustrated them, and compared different types of online survey (simple HTML vs flash, traditional vs innovative question formats).

The results, they claim, show that online surveys both bore respondents and can damage data quality. Among the findings, they found that if boredom sets in, respondents increase the speed at which they answer questions, leading to fewer responses being given generally, and a loss of data quality due to a combination of increased pattern answering and a shift away from using scale extremities.

You can read more about the report on Brand Republic and on the FreshNetworks Blog.

What are anybody else's thoughts on this? Do you have any evidence of how respondents react to online surveys? What's your experience of working with online methods - what makes people take part and what dissuades them?

[Andrew Jeavons]

Yes, they get bored and then they quit or just stop responding with any remnant of thought. This has long been known and true. A lot of this is the fault of survey writers, I heard someone from a very-big-company half boasting that their 40 minute survey was a real pig to complete. And we wonder we have problems with data quality ?

The was a problem in CATI long before the web came on the scene. The notorious "just five more minutes" when you were 40 minutes into a CATI survey that took 60 minutes to complete did nothing for data quality. We just have an updated version of the same problem.

People try to use the minimum of cognitive effort to get through the day, and surveys are not different. There is a growing literature on satisficing in interviews (see http://www.websm.org, search for satisficing) . If surveys are long, boring and hard to navigate you get poor data. The real challenge now is re-invent interviewing and develop ways of getting data that is valid. The question now is: what should we be asking people anyway - do they really know anything about why they behave ? Web surveys may just be another iteration of the same problem.

[Linn Lindström]

With 40-60 minute survey, of course people will loose interest! Not even classes in school have a duration over 45 minutes! ABSOLUTE maximum on a survey should be 30 minutes, if you have 10, great! Less drop outs for you!

To make sure people dont answer randomly, put a control question somewhere in the survey. Or put two.

But with targeting people who may only be vaguely aware of the fact that they singed up for surveys ("get offers and surveys from our partners") and to then expect them to give you an hour of their day... There is no surprises in drop-out rates there...

But with good panlists (who acutally would like to answer survey where they are part of the target group) and surveys under 30 minutes (less is more) - the data quality should be sufficent! :)

And always give respondents the chance to rate the survey - this way at least you'll know 'til next time!

These are at least my belives =)

[Mekki]

This effect is an extension of response fatigue which is present in all questionnaire responses. It is correlated with the length of the questionnaire, and the respondent's interest with the particular topic in question. It can also be related to the complexity of the questions being asked.

The primary way of mitigating its impact on data quality is to randomize the order in which the questions are presented to different respondents. In this manner, the survey fatigue is evenly distributed across the results, and will not skew the later questions more than the earlier questions.

[Andrew Jeavons]

Randomization works of course, but when a series rating scales on a particular concept or product are wanted naturally they need to be together. Questionnaires are simply getting too long.

[Jon Beaumont]

Matt, if I'm correctly remembering the GMI study you refer to, it is grids that are Public Enemy No.1 in online surveys. If you avoid or cut back on those, or use more interesting ways of collecting the same data (GMI study had suggestion I believe), then it will help mitigate respondent fatigue enormously.

However, I agree that some online surveys are too long. In my experience, it's trackers (e.g. customer sat) that can fare the worse, particularly if questions are added over time, but none are culled. It's up to agencies and clients to work together to figure out what they really need, and cut back on the 'nice to have'. Also, some qualitative exploration can sometimes help with the development of a focused quant survey, especially if it's a complex/involved subject.

Cheers
Jon