Using a Cash Incentive to Heighten Mail Survey Response
Obtaining a high response rate is a recurrent problem facing researchers conducting mail surveys. The use of a cash incentive has been one method of attempting to induce cooperation from respondents. This paper describes an experiment in which sampled respondents were randomly assigned to incentive and non-incentive groups, in order to determine (1) whether a pre-mailed $1.00 incentive produces a significant increase in the rate of survey response and, if so, (2) whether the approach is cost effective.
There are many reasons why mail surveys continue to be a popular form of data collection. Apart from the pragmatic matter of expense, there are several technical considerations that favor mail surveys — the ease of obtaining a geographically dispersed sample, the ability to get to difficult-to-reach respondents (e.g., soldiers, business executives, those away from a telephone), freedom from worries of interviewer bias, greater privacy, the opportunity for respondents to answer thoughtfully and, when necessary, time to access difficult-to-locate information needed for the response.
There are, of course, common situations where a mail survey would be inappropriate, such as when using a sensory stimulus (product sample, advertising copy, etc.) that cannot be presented to the respondent by mail, when operating under tight time constraints, or when interviewer interaction or reinforcement is required (for example, in eliciting a clear, complete answer to a difficult open-ended question).
The most compelling argument against mailed questionnaires, however, has been the typically unsatisfactory response rates often achieved – which can run as low as 10 to 20% in many general public/consumer surveys.1 Given the need to make inferences from the sample to some population, low response rates necessarily raise suspicions about possible non-response bias and, thus, any inferences made.
Cash incentives may be one way to boost mail response — whether sent along with the questionnaire or made conditional upon the respondent's first completing and returning it. The size of the incentives tested has tended to be modest — mostly 25 cents; very few researchers have experimented with incentives as high as $1.00.
For some, even incentives of 25 cents might seem a strain on the research budget, and are rejected as a result. However, cash incentives might not be as expensive as they first appear. Higher response rates are attainable, in certain cases, at little or no additional cost per respondent.
1Response rates using samples selected from ongoing panels recruited to be regular survey respondents tend to be much higher. Surveys using Market Facts' Consumer Mail Panel typically achieve response rates of 60 to 80%.
An experiment Using a Cash Incentive
As part of a larger study, Market Facts recently directed a survey among Canadian Jews about travel to Israel. Most of the questions concerned past travel experiences or attitudes toward possible future trips. The sample (N = 600) was drawn from the telephone directories of four Canadian cities — Toronto, Montreal, Vancouver, and Winnipeg — using a procedure known as "distinctive Jewish name sampling." (Although the method is somewhat controversial in terms of producing a representative sample of the Jewish population, this debate is irrelevant for the current experiment.)
The object of the experiment was to assess the effectiveness of a $1.00 cash incentive in encouraging mail survey recipients to complete and return a rather lengthy (8-page legal size) questionnaire. A dollar bill (U.S.) was included in 300 (half) of the randomly selected mailings ("test group"), while no incentive was sent with the other 300 questionnaires ("control group"). All of the packages contained a stamped return envelope and a cover letter to describe the purpose of the study, explain the sample selection process, and request cooperation. The letter sent to the test sample also referred to the enclosed incentive. A follow-up, reminder postcard was mailed to all respondents approximately eight days later, and a second questionnaire was mailed to remaining non-responders about two weeks later.
Results of the experiment
Table 1 presents the outcome of the experiment, showing the effect of the incentive on response.
The cash incentive cell has a 34% response rate, twice that of the non-cash cell. The difference is statistically significant with at least 99% confidence.
There is little reason to doubt that increases in response rates could also be achieved in a wide variety of other mall surveys. But how much of an incentive should be used? The answer, of course, depends in part on the level of response desired as well as on available resources. Some research suggests that, in general, $1.00 may be more than is needed to achieve a marked increase in returns (compared to using no incentive). In fact, for some surveys, 25 or 50 cents might do just as nicely. The answer also likely depends on the length of the questionnaire, the composition of the sample, the salience of the survey topic, the number of follow-up attempts, and other factors.
In the present study, it was felt that conditions warranted a larger incentive because of the long questionnaire and because of possible suspicions some recipients might have had about the researcher's knowledge of their religion. Whether a smaller incentive would have worked as well in this case, or whether a larger one would have generated an even greater response, is unknown. (The experiment also does not address the issue of pre-mailing the incentive versus making the cash reward contingent upon survey cooperation.) It does demonstrate, however, the effectiveness of a pre-mailed $1.00 cash incentive in heightening mail survey response.
An equally important issue related to the technical quality of the research is the possibility of self selection bias: Does a monetary incentive disproportionately elicit responses from certain types of individuals, causing the obtained sample to be unrepresentative?
To test the self-selection hypothesis in the present experiment, the incentive and non-incentive groups were compared on a series of personal attributes and behavioral measures to determine if any significant differences exist. Using chi-squared tests of independence, no differences between the groups were observed on these items, which included past travel to Israel, intent to travel in the future, religious beliefs, and others.
Relative Cost
How much greater, if at all, is the expected cost per completed questionnaire (CPQ) for a sample sent an incentive, compared to a sample not sent an incentive? For the test sample mailed the incentive, the sum of the direct cost of the labor in preparing the mailing, the non-postage direct expense (paper, envelopes, printing), outgoing and incoming postage, and the $1.00 incentive came to $726. This amount divided by 95 (the number of completed questionnaires received) equals $7.64 per returned questionnaire. In order to elicit the same number of completed questionnaires without the incentive would have required mailing to about 559 recipients, assuming a 17% response rate (.17 x 559 = 95). The sum of the labor, postage, and non-postage expenses for this size mailing would have been $727. Dividing this by 95 equals $7.65 per completed questionnaire — virtually the same CPQ as in the test sample.² The $1.00 incentive turns out to fully pay for it self. In this case, the heightened rate of response represents a technical improvement in the research — at no additional cost.
²Excluding the cost of the incentive in this comparison, the cost per mailed-out questionnaire would be $1.50 for the test group and $1.30 for the control group. The difference is due to being able to spread the fixed costs — those unrelated to the size of the mailing – among a larger number of mailings in the no-incentive group (559) than in the group that would receive the incentive (300).
Appendix
Business Reply Card
In the study described above, it was not feasible to use a business reply envelope so that incoming postage would only be charged for returned questionnaires. Rather, returned postage had to be paid on all mailings. The difference has a dramatic effect on the relative cost comparison. Were the same design applied to a survey in the U.S. (where a business reply envelope could be used, thereby incurring postage costs only for returned questionnaires), the total cost for the group of 300 receiving the incentive would have been $632, or a CPQ of $6.65. The control group sample mailing to 559 would have cost $514, or a CPQ of $5.41. (The same response rates were assumed in this hypothetical scenario.) Under these conditions, simply increasing the size of the mailout and using no incentive would have been more efficient.
In such instances, the decision about whether to mail a monetary incentive is not as easy. Here, the improved response rate produced by the incentive needs to be balanced against the additional cost required to obtain the same number of replies. The actual cost differential of generating returned questionnaires in a given study will depend, among other factors, upon the desired final sample size (and, thus, the number of questionnaires which need to be mailed). Researchers will have to judge the costquality tradeoff in the context of their research needs and priorities.
