Mail Panels vs. General Samples: How Similar and How Different?

Background

Multi-purpose household panels, commonly referred to as "mail panels,¹" offer several advantages for researchers: (1) response rates are generally quite high; (2) strong respondent cooperation facilitates true panel design studies (and diary studies) with relatively low rates of sample attrition; (3) customized samples can be selected "off the shelf" (or via inexpensive mail screening) including samples of low-incidence populations, saving screening costs; (4) samples can be nationally balanced (made demographically representative through quota sampling) on multiple variables; (5) much respondent and household background information needed for data analysis is already available, saving time or space in the survey; (6) use of panel samples facilitates otherwise very difficult or expensive data collection, such as national surveys of children, brand loyalty studies, conjoint measurement surveys requiring complex modes of questioning, and others.

Despite these benefits, some researchers weaned on probability sampling methods are understandably skeptical about using mail panel samples, and distrust conclusions from panel surveys because they typically violate the fundamental premise of statistical inference, i.e., knowing the probability of selection of sample elements. Others resist mail panel surveys because the level of non-cooperation at the recruitment stage is generally high.

Statistical purists may oppose the use of mail panels in all applications. Other researchers make a conscious cost-benefit estimate of the total survey error² - mainly sampling error, sample bias, and measurement error - likely to result from different types of samples, similar to the process used in making other methodological choices in marketing and social research. In some instances, the panel option is selected as likely to offer the lowest total amount of error for a given budget.

Some others take a narrower view, focusing solely on the sampling method relative to the survey's objectives. This camp distinguishes between unacceptable uses, e.g., when precise population parameter estimation is paramount or when the costs of faulty survey results can have serious consequences (as in a major public health survey), and "permissible" applications, e.g., when the main objectives are comparisons between subgroups or assessments of temporal changes at the individual level - in those situations supporting the assumption that any sample biases are likely to be constant across groups or over time.

Aside from criticisms grounded in statistical theory, suspicions persist that persons who join and participate in multi-purpose panels are different: that mail panel samples are unrepresentative of more general samples. Such a priori suspicions also cast doubt on the applicability of panel-based research, perhaps more so than the theoretical issues. However, apart from a few widely accepted generalizations about demographic composition (panel samples under-represent African Americans, persons with poor English language skills, those at the extremes of the socio-economic spectrum - as do most surveys), this premise remains largely untested.


Objectives

The purpose of this paper is to examine how mail panels and general samples are similar and different by comparing results based on parallel national telephone surveys of panel and non-panel samples. The panel sample was compiled from Market Facts' Consumer Mail Panel" (nCMP = 964)³ and a non-panel sample was generated through random-digit-dialing (nRDO = 897). The comparisons will focus on the following issues, which reflect commonly heard hypotheses and speculation about the distinctiveness of mail panel participants:

  1. How willing are mail panel and general samples to participate as respondents in surveys? What are the relative survey response rates? What are the components of non-response in the samples? Why do persons in panel samples participate more readily?

  2. How willing are the samples to answer specific survey questions, including questions about matters that some might consider private or sensitive? What are the relative levels of item non-response?

  3. Do mail panel respondents tend to hold generally more positive attitudes, as some contend? Are they more altruistic or more cooperative?

  4. In terms of "lifestyle" dimensions sometimes used to differentiate the market or general public, how do the samples compare in such realms as media usage? Personal health-related practices? Leisure time and activities?

  5. How similar or different is the consumer behavior of panel and general samples? Do they own or use the same types of products and services? Do they shop with the same frequency at various types of retailers?

  6. How different are mail panel and general samples in their motivation to participate in surveys? What explains the greater motivation and higher participation rate of panelists?

The answers to these questions ought to be valuable in helping researchers who are considering mail panel samples decide the appropriateness of their use in different contexts and what the cost and quality trade-offs are. It should also help current users better evaluate the strengths and limitations of their data.


Research Methods

To represent the general U.S. adult population, a random-digit-dial sample was selected using conventional procedures. For the mail panel sample, a nationwide sample was compiled following the procedures used in typical national CMP surveys: The sample was balanced using five "balancing factors" whose national distributions are regularly updated based on U.S. Census estimates. The sample was drawn to approximate actual distributions-of household income, population density, panel member's age. and household size within the 9 Census divisions. In theory. depending on the needs of the particular survey, other balancing factors can be used as long as the population's target distributions are available.

Interviewing took place evenings and weekends during a 16-day period in the Spring of 1994. The same interviewers worked on both samples to avoid possible response differences resulting from different interviewing skills or styles. A minimum of five contact attempts were made at different times of the day and week before sample replacement. Respondents were selected within households using a form of controlled quota sampling to achieve a proportionate number of male and female respondents. The surveys administered to the two samples were parallel in content and structure so the validity of the results would not be impaired by possible "context effects". The interviews averaged 15 minutes to administer.

As is customary in reporting national poll results, the data from both samples were adjusted through weighting to more accurately represent the U.S. adult population. Three variables were used in the weighting: region, household income, and age by gender.


Sample Response Rates

Survey response rates are commonly used as a measure of data quality (i.e., how well the obtained sample is presumed to represent the population). For a given number of contact attempts, total completion rate can also be viewed as a gauge of operational efficiency: the greater the number of interviews completed given a fixed effort expended (e.g., number of hours of interviewing), the higher the efficiency and the lower the data collection costs.

Cooperation and completion rates for the CMP sample are sharply higher than for the RDD sample, as expected. The CMP sample cooperation rate was 76%, compared to 37% for the RDD sample. The overall response rates were 64% in the CMP sample, compared to 25% in the RDD sample. Most of the non-response is attributable to refusals (initial refusals plus some mid survey terminations).

Because of the inability to distinguish between qualified and non-qualified initial refusals, our strict classification method inflates the number of initial refusals by qualified respondents, and therefore, underestimates the true survey response rates.4


Several reasons can account for the difference in response rates. The most obvious is that the CMP households (at least the designated Panel member in each household) made a deliberate decision to participate in surveys by joining the Panel. So they are likely to have a greater motivation, sense of obligation, or higher tolerance for being a survey respondent. Second, as a result of prior experience with Panel surveys and the CMP's periodic non-survey "maintenance" communications, a pre-esablished rapport and sense of trust is established between the CMP and its member households. Third - a reason which relates to contact rate rather than cooperation rate - is the better quality sample list (accurate, up-to-date phone numbers) available on Panel households. Along with address and demographic information. Panel household telephone numbers are regularly updated and, unlike in RDD surveys, the sampling frame does not intermix working household numbers with non-working and non-residential numbers. Fourth, Panel households that do not wish to be contacted for telephone surveys (approximately 10% of total Panel households) are automatically excluded from the panel sampling frame before telephone survey samples are drawn.

The superior contact and cooperation rates for mail panel samples can reduce data collection costs rather dramatically, compared with execution of general samples, which do not take advantage of the efficiencies of pre-recruited panels.


Item Non-Response

An often-touted advantage of using mail panel samples is the greater willingness of respondents to answer questions, including topics involving sensitive issues or requests for information that some consider private or personal. On 19 questions in the survey which any respondent in either sample refused to answer (including 10 demographic questions),the Panel sample refusal rate was lower 18 times - on all nine non demographic questions and 9 of the 10 demographic questions (all except respondent's race). While the differences in the proportion of refusals are mostly small, the direction of the difference quite consistently favors the mail panel sample. Lower item refusal rates mean less missing data.


Positivity in Beliefs and Altitudes

Some have hypothesized that mail panel samples have a tendency to show a positive bias in attitudes, relative to the population. In other words, regardless of the product or issue, panel samples are said to display a general tendency to respond with a more positive rating. Our research included a test of this theory by querying the samples on their perceptions of the nation's direction, the current U.S. economy, respondents' own personal finances, and their expectation of change in their personal financial situation. All of these questions can be regarded as measures of "satisfaction" or "optimism:" having, in one form or another, a positive outlook.

The data do not support the positivity hypothesis. If anything, there might be a small tendency in the opposite direction, i.e., for general samples to display a slightly more positive relative polarity and mail panels to be a bit more negative/ critical in their evaluations: RDD sample mean = 3.67, CMP sample mean = 3.45 on the 9-point Positivity Index, which combines the answers to the four items (t=2.11, significant at < .05).

Some have similarly speculated that mail panel respondents might be more altruistic than others. (This could be a reason for their joining and participating in panel surveys.) The survey data show no significant difference between Panel household respondents and RDD sample respondents on either volunteering time or contributing money to help others. Nor is there any difference in attendance at religious services - a factor shown to be correlated with charitable behavior. As measured by attitudes and by behavior, the results provide no evidence that the mail panel sample respondents are more "positive" (i.e., less critical, more optimistic or altruistic) than RDD sample respondents.


Product Ownership/Usage and Shopping Frequency

To compare the samples on the key variable, shopping frequency, respondents were asked the following question five times, each time for a different kind of retailer.

How many times do you, yourself, shop or buy anything at a [INSERT STORE TYPE] in an average month?

The store types asked about were: grocery store or supermarket, drug store, department store, discount store (like K-Mart or Walmart), and convenience store (like 7-Eleven). The two samples display very similar averages for all five types of stores:


  CMP RDD
Grocery/supermarket 8.1 7.9
Drug store 3.2 3.0
Department store 3.2 3.3
Discount store 3.7 3.8
Convenience store 7.9 7.8

None of the mean differences even approximate being statistically significant. Nor is there any consistent directional tendency to the small differences. These data support the conclusion that mail panel samples are representative of the population with respect to shopping frequency, or, at least, will yield results which are statistically indistinguishable from those obtained through RDD methods.

The survey also included questions about ownership/usage of two household products and one service - personal computers, telephone answering machines, and long-distance telephone service - which are sometimes viewed as differentiating consumers into market segments. The samples are about equally likely to have a personal computer in the home (33%-RDD sample, 35%-CMP sample) and also about equally likely to have a telephone answering machine in the home (64%-RDD, 67%-CMP). Random-digit-dial sample households however, spend slightly more monthly on long-distance telephone service than CMP households: $23 vs. $19 (median values).


"Lifestyle" Characteristics and Leisure Activities

With regard to consumption of alcoholic beverages, Panel sample respondents are statistically neither more or less likely than RDD sample respondents to drink beer or to drink wine, although they are a little more likely to say they consumed some other kind of alcoholic drink (15% vs. 11%) in the last week. The Panel sample and the RDD sample are not statistically different in drinking behavior overall: they are about equally likely to have had some alcoholic beverage during the previous week (36% vs. 39%).

The same is true for cigarette smoking: 28% of the RDD sample smoked a cigarette in the past week, compared to 26% of the CMP sample.

To capture another aspect of "lifestyle," respondents were asked to rate the quality of their current health. Distributions of responses for the two samples are remarkably similar:


  CMP RDD
Excellent 29% 29%
Pretty good 47 49
Fair 18 16
Not so good 6 6

One reason their health might be similar is because the two samples exercise about the same amount: RDD respondents reported getting at teast "20 minutes of vigorous exercise" an average of 2.9 days out of the previous seven, compared to 2.8 days of vigorous exercise reported by the Panel respondents.

One's travel behavior (especially foreign travel) says a lot about that individual. Measures of travel are commonly used in market segmentation studies as an important dimension of "lifestyle." The survey included a question about whether or not the respondent had ... traveled to another country other than Canada or Mexico during the last 2 years: "The results for the two samples are identical: eleven percent of the RDD sample and 11% of the CMP sample responded affirmatively to the question.

The samples do differ, although modestly, on handgun ownership: 27% of the Panel sample reported keeping a handgun in their home for protection, compared to 22% of the RDD sample (t= 2.55, < .01). Given the sensitive nature of this question combined with the greater reluctance of general samples to answer some types of questions, it is arguable which figure is more accurate.

It is not true that CMP respondents have significantly more free time available to them than RDD respondents – an explanation sometimes offered for mail panel respondents' willingness to participate in surveys: "In the last 7 days," Panel respondents averaged 5.0 evenings spent at home, compared to 4.8 for the RDD sample - another difference well within the range of sampling error.

In terms of media usage, Panel respondents are somewhat more likely to read the daily newspaper regularly: 4.8 times per week vs. 4.3 times per week for the RDD sample (t=4.29, < .OO1). They also tend to watch more television: 17.9 hours/week, compared to 15.7 hours/week for the RDD sample (t=3.31, < .001). As for radio listening, the difference between sample means - 14.6 hours/week (RDD) vs. 15.0 hours/week (CMP) – is not statistically significant.

The results presented in this section, spanning a diversity of topic areas, suggest that mail panel samples are likely to parallel the population in most, if not all, dimensions of leisure activity and lifestyle.5


Survey Participation

This section addresses three questions:

Panel respondents like participating in surveys more than RDD sample respondents:


  CMP RDD
Like 62% 39%
Indifferent 23 34
Dislike 15 27

The contrast is substantial, but entirely expected: If mail panel respondents did not like participating in surveys more than others, use of household panels would carry little unique value for researchers.

Even though CMP household members are asked more often than others to serve as survey respondents, they tend to refuse less often: CMP respondents refused survey participation an average of 1.3 times in the last six months, compared to 1.8 times for RDD respondents. These data reinforce the finding that members of mail panel households are more willing survey respondents than others.

The question is often asked why individuals choose to join mail panels. What benefits do they see themselves deriving from panel participation? What is their motivation? Some have speculated that panel members are less busy than others and thus have more free time to spend filling out questionnaires and being interviewed by telephone. The non-significant contrast presented earlier on the number of evenings spent at home in the past week suggests that greater time availability is probably not the reason for mail panel respondents' greater survey participation. There must be other explanations.

The survey contained a straightforward open-ended question, administered to all Panel members, 7 to address the issue:

Thinking back to when you first joined the Consumer Mail Panel, why did you decide to join?

The reasons mentioned most often were:

Like being asked my opinion .................... 26%
Like surveys, it's fun ................................. 19
Thought it would be interesting .................. 17
I wanted to be helpful, to cooperate .......... 14
I like trying out (new) products ................. 12
Because I had the time, something to do .... 9

The primary motivations for joining a mail panel seems to be an affinity for expressing one's opinion and finding the act of participation interesting or enjoyable. (Fewer than one-tenth of the panel members said they joined the Panel because they have the time or because it gives them something to do.) These are probably the same reasons why others participate in surveys when asked, although this was not tested through a similar question presented to the RDD sample.


Summary and Discussion

This research has examined the similarities and differences of a multi-purpose household panel ("mail panel") sample and a random-digit-dial sample on a broad range of characteristics. Some of the characteristics chosen for comparison were selected because they are commonly used variables in marketing or public opinion research. Others were selected for analysis to test various hypotheses about mail panels, including characteristics where the samples were suspected of showing contrasts. In interpreting the overall meaning of the findings, it is important to recognize that precise representation of the target population is just one of several considerations - albeit, in some contexts, the most critical - in decisions about what survey methodology to adopt. A "total survey error" perspective frames the decision more broadly in terms of the trade offs involving other sources of error as well, not only potential sample bias, for a given budget.


Implications of the Positiity and Altruism Findings

It has been alleged that mail panel respondents are psychologically inclined to rate things more positively than are other respondents. The data from this research are inconsistent with this premise. Examining comparisons based on personal and social perceptions and expectations, there was no evidence of a "positivity bias" among panel respondents. A related hypothesis holds that mail panel respondents are more helpful (or altruistic) than the average person, a perception probably related to their greater cooperativeness in surveys. This perception too failed to gain support in this analysis. In both respects, the panel sample appears to mirror the RDD sample. Based on these results, "positivity bias" should not be a concern in the many types of mail panel research that rely heavily on attitudinal assessments.


Implications of the Consumer Behavior and Lifestyle Comparisons

Even though some of the consumer behavior questions were included in the survey intentionally to test suspected differences, the two samples did not differ meaningfully on 15 of the 20 diverse measures of consumer behavior and lifestyle. Perhaps most important was the finding that mail panel and non-panel respondents shop equally often at grocery stores, drug stores, department stores, discount stores, and convenience stores - supporting use of panel samples for studies involving shopping frequency and bearing indirectly on volumetric estimates of product purchases.

On the five consumer and lifestyle comparisons which showed statistically significant differences, the contrasts hardly appear large or general enough to cause concern to researchers. The only possible pattern detectable in this set of differences is a little greater media usage (TV, newspapers) among the Panel sample. All in all, the analysis of consumer and lifestyle comparisons between the two groups supports the use of mail panel samples as technically sound in the large majority of consumer/survey research applications.


Implications of the Findings on Survey Participation

Few persons join mail panels in order to get free product samples or material rewards, or because they have a lot of free time allowing them to participate. Instead, people join panels mostly because they like voicing their opinion and enjoy responding to surveys. We believe that these reasons are largely the same as those motivating non-panel respondents to participate in surveys. Panel respondents do like surveys more than other respondents - a logical finding which should come as no surprise and, in most contexts, should not cause relevant disparities in survey results.


Concluding Observations

Decisions about sampling methodology are usually made by weighing several criteria ("total survey error" perspective) and depend, in large part, on the survey's specific objectives. In certain contexts, other considerations besides sample representativeness can favor mail panel-based research. These include:

  1. Higher survey participation — The panel sample demonstrated a much higher completion rate. Higher response rates can be particularly beneficial in repeated measures designs by reducing panel attrition. The responsiveness of panel samples also opens up types of multi-contact research which would otherwise be infeasible. One class of examples is phone-mail-phone designs, where persons have to be recruited to perform a task before being interviewed (trying a new product or reading a set of concept statements). Such designs involve three opportunities for attrition or non-response, and are more likely to be effective with panel samples.

  2. Higher item response — The mail panel sample displayed lower item refusal rates. Use of a panel sample can reduce the amount of missing data on commonly asked questions (like household income) and can be of particular assistance in surveys on sensitive topics. Mail panel samples are also likely to facilitate consumer diary studies requiring extensive, careful recording of product purchases.

  3. Availability of pre-existing information on panel households — This data is often invaluable for compiling specialized samples of rare or low incidence populations by eliminating or reducing screening costs. In one recent application, a national sample of primary and secondary school teachers was identified using occupational descriptions of panel household residents. Mail panels have also been used to select national Jewish samples (2.5% incidence).

  4. Low-cost screening to compile specialized samples — When the information is not already part of the panel database, it can often be obtained at low cost via a post card screening of panel households. This is a very common mail panel application, which would be prohibitively expensive (if possible at all) using general samples. Large-scale screening has been employed to identify firs-time purchasers of mutual funds, riders of all-terrain vehicles, travelers to the Bahamas, persons who had recently arranged a funeral for a family member, and in many other applications.

  5. Opportunity to efficiently survey teens and children — Because mail panel household parents trust those who administer the panel not to abuse the relationship or misuse information collected through panel surveys, they are normally willing to allow, assist, and encourage younger members in the household to participate in mail panel surveys. In one application, teens and their parents were surveyed about educational plans and preferences. Panels have been used to survey children as young as 6 about candy preferences.

  6. Lower costs of survey execution — The costs of conducting surveys, either by telephone or mail, are lower with mail panel samples. The magnitude of the savings in telephone interviewing costs was estimated in the current "test." But rarely is lower cost alone the primary rationale for choosing to use a mail panel. As the above illustrations suggest, panel samples can offer research design options for specialized needs which are unavailable elsewhere.

Survey researchers face significant challenges in the years ahead to produce high-quality work at reasonable cost using traditional sampling methods. Completion rates have been declining for two decades. New telecommunications products which enable residents to screen incoming phone calls will likely continue to contribute to declining rates of survey response. Attempts to regulate telemarketing (and other unwanted contacts) could have a similar effect. And there are few signs that, in the future, the public will have more time to participate in surveys or become more receptive to them. If anything, the opposite might be occurrinq. This research should help researchers evaluate the feasibility and utility of using pre-recruited respondents from carefully designed mail panel samples as an alternative to traditional approaches.


NOTES TO THE TEXT

  1. The term "panel," "consumer panel," or "mail panel" to describe pools of households pre-recruited for use in different types of surveys is sometimes confused with the more technical meaning of "panel" in research design terminology. The latter usually implies a sample that is resurveyed over a period of time on the same topic to assess changes at the individual level ("repeated measures" design). As used in this paper, "panel" and "mail panel" refer to a pre-recruited collection of respondents and households from which samples are drawn for different types of surveys, including those based on one-time, cross-sectional designs.
    Although the name may suggest otherwise, mail panel samples can be, and often are, used in telephone surveys.

  2. Ronald Andersen, Judith Kasper, Martin R. Frankel, and Associates, Total Survey Error, Jossey-Bass (San Francisco, 1979).

  3. The total Consumer Mail Panel currently consists of over 400,000 households throughout the Continental U.S.

  4. In this survey, outcomes classified as refusals are, in many instances, unable to distinguish between qualified and unqualified refusals. Although there would likely be a qualified respondent in this survey at most household numbers reached, some "refusers" might be under-age phone answerers, some might have been reached at a business or other non-household number, a few households might have contained no residents at least 18 years old, etc. As imilar problem applies to the telephone numbers classified as non-contacts due to "no answers." Because the interviewing was confined to evenings and weekends (except for a small number of scheduled can-backs), some proportion of the "no answers" in the RDD sample are businesses and other non-household phone numbers. The inability to differentiate businesses from households among the "no answers" results in over-estimating the true amount of non-response due to non-contact in the RDD sample.

  5. The research also contained a series of questions on political topics and current issues. These results are too extensive to fully review in this issue of Research On Research. The main finding was that CMP respondents tend to be somewhat more Republican and less Democratic than those in the RDD sample and more likely to have voted in the last Presidential election - a possible consequence of the under-representation of ethnic minorities in panel samples. Although differing on highly partis an matters (e.g ., ratings of President Clinton), the two groups display only minor and inconsistent differences in policy preferences.

  6. The issue of actual participation rates is not addressed in this paper since, almost by definition, members of an ongoing panel have many more opportunities to participate in surveys.

  7. Two-thirds of the CMP sample turned out to be designated Panel members, while the remaining one-third was made up of other persons in CMP households.