An Examination of Order Bias (On self-administered questionnaires)
In questionnaire design, marketing researchers should be aware that order bias exists structured responses.
The Problem:
A dilemma that often confronts marketing researchers in the design of questionnaires is that of eliminating or averaging bias to the greatest extent possible. The wording of the questions. The sequence of questioning and ordering of structured responses to certain questions — whether asked by an interviewer or presented to the respondent on an exhibit card or self administered questionnaire — can all influence the respondent's answers. This discussion focuses on one of these potential sources of bias: The order of structured responses on self-administered questionnaires.
The most commonly used method of averaging order bias in structured responses is to rotate the order of presentation of the stimulus: e,g., a frequency of use scale might range from high to low for half the respondents and low to high for the other half. Often rotation is not employed for a number of reasons. Two of the major reasons are: 1) the extra cost and trouble associated with developing various versions of the questionnaire, and 2) the additional data processing effort, time, and cost required to reassemble each version into a common format for analysis. Depending upon the scope of the research these may or may not be valid reasons.
Research Method:
Market Facts, Inc., investigated the issue of order bias in self-administered questionnaires using Consumer Mail Panels, its controlled mail panel research facility. Responses to five questions. directed to female the heads of households in two balanced panels of Consumer Mail Panels members, were analyzed for bias. One panel received questionnaires with the structured responses in one order and the order was reversed for the other panel. The number of respondents in each panel exceeded 760.
The five questions covered a variety of topics: Presidential performance, the weather, television viewing, restaurant patronaqe, and home repairs.
Results:
Statistical analysis of the data obtained from the two panels reveals that significant levels of order bias exist in three of the five questions: those relating to television viewing, restaurant patronage, and home repairs. The reason that order bias was not detected in two questions could be a function of the simplicity of the response and/or the scales used. Because a significant level of order bias was detected in a self administered, mail panel questionnaire, a research medium where order bias is least likely to occur, it can be assumed that the bias will be much greater in telephone surveys or interviews in which the structured response is read to respondents.
Clearly, the issue of order bias deserves serious consideration and the rotation structured responses to permit averaging of the data should be employed whenever feasible.
Discussion of the Data:
Examination of "top-box" scores for frequency of television viewing, reveals that respondents are more likely to admit to greater television viewing when this response option is presented in the first position. The difference in the "top-box" response is significantly different at the 95% confidence level. To test for bias in the complete distribution, further analysis was performed and differences, significant at the 95% level of confidence, were detected.
Similarly, "top-box" scores for the frequency of eating out and household repair questions are significantly different at the 95% confidence level. Respondents are more likely to admit to more frequent eating out and hiring of tradesmen for home repairs when these responses are in the first position. (An examination of difference in the entire distribution detected a significant difference at the 95% level of confidence for eating out but not for household repairs.)
There were, however, two questions in which no significant differences in response frequency were detected in the two versions. The two questions pertain to President Carter's performance and perceptions of the weather this winter (1977-1978) compared to the previous winter. Possibly the simplicity or nature of the scales elicited the similar response rates.
'To improve the quality of research.'
At Market Facts we believe good research must begin with clear understanding of our clients' needs—that is why our research professionals undertake a thorough exploration of the background and objectives before initiating a project.
At Market Facts we also believe that a continual review and refinement of research methodologies is essential to produce the most reliable data possible — that is why Market Facts has an on-going program of Research-On-Research.
Research-On-Research at Market Facts is a continual program of research methodology evaluation with one basic objective — "to improve the quality of research".
This report on sequence bias is the first in a series we will be forwarding to you. Additional reports will cover such areas as:
- Purchase Intent Questions
- Attitude and Rating Scales
- Other Forms of Question Bias
Not all of the findings will be "earth shattering". We feel, however, that they will be useful references. Please watch for these reports in the coming weeks and months.
