Do you want overall opinions or diagnostic information?

Background:

Researchers often attempt to determine consumers' overall opinions or preference ot two or more products/brands. This can be done in a variety of ways: have respondents rate each product individually on an overall liking scale, a rank ordering at the products, a head-on preference between two products. etc. A determination of consumers' total reaction to products as a whole, or relative to each other, can be valuable for measuring product, advertising or other marketing efforts. This may be done at one point in time, or over time (by conducting a series of studies).

Many times researchers also include a selected set of attributes along with overall opinions / preference. These attributes are frequently included to provide diagnostic information about product strengths and weaknesses. The attributes mayor may not include those characteristics which the consumer would take into account when forming an overall opinion / preference.

Are overall opinions/preferences influenced by the presence of (and the respondent therefore paying particular attention to) a subset of product attributes? If so, then researchers must proceed cautiously. Researchers face fewer restraints if overall opinions / preferences are not affected by the attributes included in the study.


Research Method:

Market Facts, Inc. conducted this investigation using Consumer Mail Panel and its Data Gage service. Three questionnaires were used to provide opinions on a popular imported car and a comparable American car.

Each of the three versions was mailed to a sample of 1000 households. Each sample was "balanced" to be representative of national households in terms of geographic region, age of female head, household income, and population density. The questionnaires were directed to the "person most responsible for making decisions about buying cars."

In one version respondents were asked only which car they would prefer to buy. Another version asked purchase preference preceded by attributes thought to be biased toward the import (good mileage, quality, engineering). The remaining version asked purchase preference preceded by attributes thought to be biased toward the American car (good styling, availability of parts, convenient dealerships). An example of each version is included in this report.

Is preference affected by diagnostic attributes?
If attributes affect preference, then:

If attributes have no effect, then preference should be the same across all three cells.


Results:

Diagnostic attributes did affect preference. Respondents showed a significant preference for the American car over the import when no attributes were asked. The preference for the American car increased with the inclusion of American biased attributes. This pattern reversed on the import biased version, where the American car actually lost to the import. Thus, attributes had a dramatic effect on preference in this study.


  Attributes asked
Purchase Preference None American Biased Import Biased
American Car 52% 59% 42%
Import Car 45 36 51
No Preference 3 5 7


Z tests were performed to analyze the significance of these preference differences between versions. The resulls indicate that differences by version were statistically significant with at least 97% confidence. Attribute results were consistent with the perceived biases. Respondents receiving the American biased questionnaire selected that car more often as having the best styling/appearance, a more convenient dealer location, and as being easiest to get parts/repairs.


  Import Car American Car
Most attractive styling/appearance 37% 59%
Most convenient dealer locations 19 75
Easiest to get parts/repair work 10 85


Attribute results were also consistent among respondents who received the import biased version. The import was most often perceived to have the best overall quality/workmanship, to get the most miles per gallon, and to have the best engineering.


  Import Car American Car
Best overall quality/workmanship 58% 36%
Gets the most miles per gallon 73 21
Has the best engineering 56 37


Other comparisons showed no difference. Return rates were about the same by version (74% overall). The sex composition was about equal by version (54% female, 46% male). The median age of the respondents was about the same by version (46 on average). The level of no answers to the overall preference question was also comparable by version (5% on average).


Conclusions:

The data suggest that it is important for researchers to be clear about their priorities. Attributes influenced purchase preferences in this study. If overall opinions/preference are the primary objectives, the inclusion of attributes should be approached with extreme caution.

It may be appropriate for attributes to precede overall opinions/preference on some studies. There are rare instances when researchers would like overall evaluations based only on a specific set of characteristics. For instance, respondents might be told to focus only on certain product characteristics in an effort to eliminate pricing, availability or brand name effects. In those cases, attributes positioned prior to overall opinions could help sharpen the focus of the respondent. The wording of the overall question in those situations should reinforce the researcher's objective by reminding the respondent to think only about the characteristics previously discussed when forming their overall opinion/preference.

Research methodology should be carefully considered if overall opinions/preferences are to be global ratings not linked to any specific characteristics. The bias caused by attributes can be eliminated in personal or telephone surveys, where it is possible to ask about overall opinions before any mention of the attributes. However, the researcher must then consider the potential impact of preference on the attributes.

Furthermore, the "overall opinion/preference first" approach does not solve the potential problem inherent in self-administered questionnaires — it is still possible and likely for the respondent to read ahead to the attribute section. One solution to these problems is to seek overall opinions/preferences separate from diagnostic attribute data; that is, waves of research separated either by time or by respondents.


Did you notice the size of the questionnaire used for this Research-on-Research report?
We call it Data-Gage.

Data-Gage provides a once-a-month opportunity for :