Measuring buying intention: How valid is the estimate?

Report 12 in this Research on Research Series summarized the results obtained when the respondents were asked to use a simple "yes" or "no" answer to indicate whether or not they intended to buy a home appliance within a specified period of time.

Research Method:

Two questionnaires were mailed to the same Consumer Mail Panel households approximately 18 months apart. The first questionnaire inquired about present ownership or intention to buy a microwave oven¹. The second questionnaire asked whether a microwave oven had been purchased and, if so, when it was purchased.

Respondents were classified into four groups on the basis of their response to the first questionnaire:

These groups were then cross-tabulated against four similar groups based on the respondents' responses to the second questionnaire:

¹See Research on Research Report Number 12


Results:

Over-all, the estimates of intended purchase were two percentage points higher than the actual purchase using respondents who supplied answers to both waves of research. Initially, nine percent of the respondents said they expected to purchase a microwave oven within the following twelve months. Upon recontact, seven percent of the respondents had actually purchased one.

Even though the sample sizes are large enough to make this difference statistically siqnificant² the relatively small percentages yielded a reasonable estimate of future purchase using data for all respondents who supplied answers.

A more targeted analysis among respondents who initially said they intended to buy a microwave oven produced much less accurate results. Upon recontact, only slightly more than one-fourth (26.5 percent) of the original intenders had actually purchased a microwave oven Because this group represented less than ten percent of all respondents, however, the over-all impact of intenders who did not buy a microwave oven was almost offset by the three percent of non-intenders (almost 80 percent of the sample), three percent of the respondents who initially gave no answer, and nine percent of the initial nonreturners who subsequently bought a microwave oven.


²The difference was tested using McNemar's chi-square test of differences between correlated proportions, correlated by the fact that the same respondents were measured in both time periods. Included were respondents who gave a "yes" or "no" answer to indicate their buying intention compared with actual behavior. The value of chi-square was 4.84. At one degree of freedom this value can occur by chance alone with a probability between .05 and .02.

The latter two groups were of special interest. Their subsequent purchase behavior was analyzed to determine the best method for handling non-response when reporting study results.

Granted, all researchers recognize that the higher the response rate, the more confidence one has that the sample represents the surveyed population. Given the reality of non-response, however, the question is, how does one deal with it in presenting the findings of a survey? In general, should non-response be allocated among the answer categories in the same proportions as the answers received? Specific to this study, when using "yes-no" answer alternatives, should it be treated as a "no" answer or should non-response be reported as non-response?

By cross tabulating the initial "responses" against the subsequent "responses," it was possible to track purchase behavior, not only among respondents who answered the intention question, but also among the "no answers" and non-returners as well.

When the respondents who were initially classified as "no answers" or non-returners are included in the analysis, the estimate of the percent of respondents who would purchase a microwave oven dropped to seven percent but remained two percentage points higher than the percent of respondents who subsequently bought one.

Further, it was found that the subsequent behavior of each of the four initial groups is significantly different³ from group to group. This means that one should neither combine "no answers" with those who said "no", nor allocate "no answers" in the same proportions of "yeses" and "nos" from those who gave answers. Of the two choices, however, combining "no answer" and "no" together would have produced the smallest error. Further, this analysis also revealed that non-returners' subsequent behavior could not be predicted using data obtained from respondents who returned questionnaires.


Conclusion :

The use of "yes" or "no" alternatives appears to have produced an over-estimate of subsequent purchase. This finding is likely to be true particularly in those instances when over-all purchase intention is relatively low.. The fact that actual behavior was statistically different than the forecast is offset somewhat by the relatively small percentages involved. It was estimated that seven percent of the respondents would purchase a microwave oven and five percent had actually purchased one by the time they responded to the second questionnaire.


³Two analyses were performed. One was a standard Pearson's chi-square and the other was a test of symmetry. Both produced results significant at .01 or beyond.

Non-response further complicates pre-post comparison. Purchase behavior among respondents who either gave no answer or failed to return a questionnaire was significantly different compared with those who gave answers regarding their intention to buy. Non-response to the second questionnaire makes it impossible, under this circumstance, to accurately estimate the true level of purchasers for any of the respondent groups.

These findings suggest that reduction of non-response is the most valid way of dealing with it. As suggested by the conclusions of Report 12 of this series, one way to accomplish this is to simplify the questionnaire if possible. A questionnaire which asked for information about only one product produced a significantly smaller non-response as compared with a questionnaire having an eleven product list.

If the researcher wishes to mask the identity of the product of interest, a questionnaire with a small number of products, maybe three or four, could be used to minimize non-response.