Survey writers are sometimes guilty of conflation - polluting a potential answer by including two statements when the respondent may only agree with one.
The first example is from Village Cinemas, who want to know how likely the respondent is to return to their establishment. A reasonable selection of answers might be:
- a scale from Very Likely to Very Unlikely
- A binary Likely/Unlikely.
(In either case, of course, a "Don't know" option should be available.)
However, two of the available answers are conflated. The first is the less problematic - the inclusion of the exclamation mark after "Every time I go to the movies" implying a degree of enthusiasm that the respondent might not share.
The second is "Very occasionally, it's a special treat". The inclusion of the second clause specifically excludes the ability to answer the question for those who might more accurately answer:
- Very occasionally, because I hate your cinema
- Very occasionally, because sometimes you're the only cinema showing the movie I want to see
- Very occasionally, because it's the only cinema within walking distance
- Very occasionally, because I don't live in Australia or Greece
... and a large number of other possible reasons for choosing "Very occasionally".
What answer will those respondents give? I have no idea. But in any case, by specifically denying people the ability to choose an unambiguous answer, the data derived from this question is suspect.
A better approach would be to have a clear set of answers untarnished by conflation, with a possible additional question asking for elaboration.
The second example is from the Dominion Post in New Zealand, which asks whether bicycle helmets should be mandatory. The set of answers is odd for a few reasons, but to stay on topic we'll just look at conflation. The two "Yes" answers and the one "No" answer both have additional value statements or clauses that render them problematic.
To focus on only the "No" answer, the available option is "No - it is a personal choice," which specifically excludes answers such as:
- No, because the societal disbenefits outweigh the individual safety gains
- No, because they look ugly
- No, because cyclists deserve to be killed
... and a large number of other possible reasons for choosing "No".
When writing survey questions, you need to provide the ability for all respondents to answer honestly.
Conflation - just say No!
The first example is from Village Cinemas, who want to know how likely the respondent is to return to their establishment. A reasonable selection of answers might be:- a scale from Very Likely to Very Unlikely
- A binary Likely/Unlikely.
(In either case, of course, a "Don't know" option should be available.)
However, two of the available answers are conflated. The first is the less problematic - the inclusion of the exclamation mark after "Every time I go to the movies" implying a degree of enthusiasm that the respondent might not share.
The second is "Very occasionally, it's a special treat". The inclusion of the second clause specifically excludes the ability to answer the question for those who might more accurately answer:
- Very occasionally, because I hate your cinema
- Very occasionally, because sometimes you're the only cinema showing the movie I want to see
- Very occasionally, because it's the only cinema within walking distance
- Very occasionally, because I don't live in Australia or Greece
... and a large number of other possible reasons for choosing "Very occasionally".
What answer will those respondents give? I have no idea. But in any case, by specifically denying people the ability to choose an unambiguous answer, the data derived from this question is suspect.
A better approach would be to have a clear set of answers untarnished by conflation, with a possible additional question asking for elaboration.
To focus on only the "No" answer, the available option is "No - it is a personal choice," which specifically excludes answers such as:
- No, because the societal disbenefits outweigh the individual safety gains
- No, because they look ugly
- No, because cyclists deserve to be killed
... and a large number of other possible reasons for choosing "No".
When writing survey questions, you need to provide the ability for all respondents to answer honestly.
Conflation - just say No!
No comments:
Post a Comment