Table of Contents
-
An Examination of Order Bias (On self-administered questionnaires)
-
In questionnaire design, marketing researchers should be aware that order bias
exists in structured responses.
Measuring Purchase Intent
-
Purchase intent information obtained from five-point and four-point scales are not
directly comparable; nor are transformations from one scale to the other easy to make.
Indications are, however, that "top-box" response for the two scales tends to be very similar.
Variations in Semantic Differential Scales
-
Rating scales that appear quite similar can produce significantly different results.
Research conducted by Synovate suggests that using semantic descriptions at all points
on a scale may be more effective at discriminating among respondents than a scale where
some points are not described.
Number-type Data: Structured vs. Open-ends
-
Collecting "number-type" information (price paid, phone calls per week, etc.)
in a structured or interval approach vs. an "open-end" method can produce quite different
results.
Measuring Purchase Incidence Rates
-
The manner in which a purchase incidence question is worded has a significant effect on the
results obtained.
Brand Perceptions: Relative vs. Absolute Ratings
-
When consumers rate brands on particular product attributes, their ratings can be influenced
by the brands they are asked to evaluate.
Frequency Measurement: Past vs. Average Period
-
When measuring frequency of purchase, use or other activities, the researcher must carefully
consider the purpose and intent and provide the respondent with the appropriate time frame.
Using multiple question grids
-
Data collected through self-administered, multiple-question answer grids can produce
significantly different results than separated answer spaces.
Pricing Research: Single vs. Multiple Presentations
-
Data gathered through multiple presentation methods can produce significantly different
results than single presentations to separate, matched samples.
Dialing Selection Techniques: Random Digit vs. Directory
-
There is a growing tendency for researchers to conduct telephone surveys using a random or
systematic digit selection technique rather than telephone numbers listed in directories.
Because a large number of telephone numbers are not listed in directories, a more representative
sample of listed and unlisted numbers is expected when the random digit selection technique is used.
Market Share Estimates and Their Standard Errors from Store Tests
-
This paper discusses the estimation of brand share and its standard error in test market or
market audit situations. Following a working definition of brand share, a technique called
"jackknifing" is introduced as an estimation procedure for obtaining the necessary statistics.
Examples of its use in some common situations are supplied.
Measuring Buying Intention: Product List vs. Single Product Questionnaire
-
How many times has someone said "well as long as we are talking to them anyway why not ask ....?"
When planning a survey it is difficult to resist the temptation to expand the number of questions
and/or increase the number of products about which each question is asked.
Repeat Measures Design and Analysis
-
The logic, use, and analysis of the Repeated Measures Design is presented. This data collection
strategy in which respondents perform a number of rating tasks on each of a set of objects.
The repeated Measures Design possesses a number of attractive characteristics, making it of
great value in marketing research.
Analysis of Variance: One-factor Designs
-
The analysis of variance (ANOVA) is a powerful technique for analyzing differences among means,
while avoiding the problems associated with multiple t-tests. When several dependent variables
are analyzed, ANOVA can be supplemented by other types of analyses that take into account the
relationships among the variables. The use and interpretation of ANOVA is described in this paper,
and example is presented to illustrate the procedure.
Do You Want Overall Opinions or Diagnostic Information
-
Many times researchers include a selected set of attributes along with overall opinions/preference.
These attributes are frequently included to provide diagnostic information about product strengths
and weaknesses. The attributes may or may not include those characteristics which the consumer
would take into account when forming an overall opinion/preference.
Using Letters to Identify Products or Brands
-
Letters are often used as product labels in order to overcome respondent biases toward particular
brands or manufacturers. However, this introduces the possibility of another source of bias,
since some letters of the alphabet may be perceived more favorably than others. The results of a
study conducted by Synovate to investigate attitudes toward letters are presented. Methods of
reducing possible letter bias are also discussed. It is recommended that more than one set of
letter codes be used whenever possible in order to accomplish this objective.
Graphic Displays of Data: Box and Whisker Plots
-
Box and whisker plots provide a valuable graphic means of summarizing and displaying data.
They are particularly useful for comparing the central tendency, variability, and shape of
distributions of responses from several groups of individuals or on several variables.
The interpretation and uses of box and whisker plots are described and their strengths
and compared to other types of data summaries are outlined.
Analysis of Ratios of Means
-
A procedure is presented for the analysis of the ratio of the two means obtained from
independent samples. The use and interpretation of this ratio is discussed along with
formulas for calculating confidence intervals and significance tests. Comparison is made
between this ratio approach and the more commonly used mean differences (subtraction) technique.
Response Measures, Data Collection Methods, and Conjoint Analysis: A Two-attribute Case Study
-
Conjoint analysis can be one of the most powerful research tools in the marketer's
armamentarium because of its ability to predict consumer preferences for products which have
never been directly evaluated or perhaps even developed. Literally hundreds of conjoint studies
have been commissioned during the last decade. But, there is no consensus on how best to implement
the various steps needed to execute a conjoint analysis study in spite of the rather large amount
of experience the marketing community has had with the technique.
Examining Group Differences: Components of T-Plots
-
Components of t-plots are a valuable graphic aid in summarizing the results of t-tests for
comparing means of independent groups on several variables. These plots show the direction
and magnitude of differences between group means on all dependent variables, as well as
confidence bounds adjusted to take into account the fact that comparisons are made on several
variables.
How to Improve Test Marketing
-
Errors and misguided directions that occur in test marketing are committed by all of us;
every major manufacturer of consumer products, every advertising agency, and every research
company dealing with the test marketing of new products. The degree of fault may vary somewhat
between the three arms, but it is there to share.
An Examination of Weekend Audits
-
In the current state of store audit research, the audit technique receiving the least use is
the weekend audit. However, the weekend audit, which measures store sales and shares from Friday
to Monday, promises to meet some research needs which may not be met by more traditional techniques.
Measuring Buying Intention: How Valid is the Estimate?
-
Report 12 in this Research on Research Series summarized the results obtained when the respondents
were asked to use a simple "yes" or "no" answer to indicate whether or not they intended to buy a
home appliance within a specified period of time.
Interpretation of T-test Results
-
This paper discusses the relationship between t-statistics, obtained from testing differences
between means, and correlations. The relationship should be beneficial in aiding the researcher
to interpret and evaluate results of t-tests. Because potential information provided by correlations
is generally not considered with t-test information, it is very possible that researches are
providing spurious conclusions to management.
Function Plots of Multi-Dimensional Data
-
Through a technique know as function plotting, one can display multi-dimensional data in a
single two-dimensional plot. Such plots are particularly useful for highlighting differences
and similarities among products or groups of respondents when data are collected on several
variables.
Analysis of Forced-Choice Data
-
A procedure is presented for analyzing differences among proportions arising from forced-choice
data. Situations in which this procedure is appropriate are described, and interpretation of
results is discussed. A comparison between this procedure and the commonly used Z-test procedure
is made.
Minimizing Losses (or Maximizing Gains) and Choosing Confidence Levels
-
Some researches are less flexible than they should be in their approach to research.
They tend to use the same number of observations and the same statistical techniques repeatedly.
This rigidity is particularly prevalent in choosing a level of confidence. Contrary to common
practice, choosing a 95% or higher level of confidence is not mandatory when performing a
statistical test. A high level of confidence is often chosen simply because it is customary to
do so. Decisions based on a high level of confidence may be subject to large losses when a decision
is made in error. Whenever possible, a level of confidence should be chosen such that losses are
minimized. The rationale for choosing a level of confidence is discussed in this report.
Measuring the "Importance" of Attributes
-
This report describes the results of a study conducted by Synovate, Inc. in which various methods
of collecting attribute importance data were compared. The methods included 4-point and 6-point
rating scales, pairwise comparisons among attributes, and a "checklist" format, where respondents
indicated only which attributes were most important, second most important, and third most
important. Except for the checklist format, the rank order of attribute means was virtually
identical for the various methods.
Displaying Group Differences Using Biplots
-
"Perceptual maps" are among the most commonly used tools for describing and portraying group
differences on multiple attributes. The term "perceptual map" is a general one which has been
used to refer to a variety of statistical techniques, including discriminant analysis,
multidimensional scaling, plots of group means on principal components or factors, and a
relatively new technique know as the "biplot." Biplots, unlike most other "mapping" techniques,
can be used with many types of data, such as means, percentages, and frequency counts.
This report describes the use and interpretation of biplots and describes the relationship of
the method to discriminant analysis.
Testing Differences Among Several Means
-
The t-test is often used to determine the significance of differences between group means.
Frequently a marketing researcher will use t-tests in order to determine which among a number
of groups differ significantly. In this situation the researcher may need to perform several
t-tests. The set of comparisons performed is commonly referred to as a 'family'.
Bootstrapping
-
Several statistical methods involving resampling of observations have been developed in recent years.
Resampling allows statistics of interest to be estimated when statistical assumptions are
inappropriate, or no known statistical approaches exist, or known statistical assumptions are too
complex to carry out routinely. One such resampling technique called jackknifing was intoduced
earlier in Research on Research Report Number 11. Another method, called bootstrapping, is applied
to the problem of estimating brand share and its variability.
Estimating Sample Sizes for Mailouts
-
This report presents a formula for estimating the number of mailouts needed to obtain a sample
containing at least a certain number of people who possess a particular characteristic.
A conventional approach is modified to control the risk of not obtaining the required sample size.
The method is most useful in situations in which the questionnaire return rate can be estimated
reasonably well. The calculations are illustrated through an example.
Simultaneous Measurement of Discrimination and Preference
-
Several taste testing procedures reviewed, all of which provide information about both
discrimination and preference. Through use of a "discriminator-nondiscriminator" model,
it is possible to assess the efficiency of each procedure in providing estimates of the
proportions of the population who truly prefer one product to another, and the proportion
unable to tell the difference between products. The procedures examined are found to differ
considerably in efficiency. Two procedures, the "repeat pair" and "double pair", are found
to have particularly desirable properties.
Confidence and Tolerance Intervals
-
Marketing researchers are often interested in estimating the average value in a population.
Information about the population average, in the form of a sample estimate, can be supplemented
by drawing and interval or range of values around the sample average likely to include the true
population average. Such intervals are called confidence intervals. The researcher can be confident,
to a chosen degree, that the interval contains the true population average. However, sometimes the
range of values in a population is of greater importance than the average. In such cases another
type of interval, a tolerance interval may be useful. Tolerance limits define the bounds of an
interval which contains a specified proportion of the individuals (or, in general, objects) in
the population, with a chosen level of confidence. This report discusses both confidence and
tolerance intervals, and their application.
An Analysis of Multiple Brand Usage
-
Brand usage within a product category can be very diverse, with most respondents using more than
one brand. Although a specific brand may be preferred, respondents may actually use several brands
within a short period of time. Usage of specific brands may be occasion-based, or purchase may be
stimulated by coupons or other price incentives. In any event, multiple brand usage requires that
the marketing researcher know and understand the usage relationships, or at least the combinations
of brands used, and the frequency with which such combinations occur.
The Logic of Statistical Significance Tests
-
Statistical significance testing, or "hypothesis testing," plays an important role in marketing
research. Significance tests are routinely carried out on sample means, proportions, etc. When
used appropriately, such tests allow the user to control the risks of drawing erroneous conclusions
or inferences about characteristics of the population based on data obtained from a representative
sample.
Sample Sizes for Analyses of Means and Proportions
-
An issue that must be resolved early in any research project concerns the sample size required
to satisfactorily address the research objectives. In many cases, a researcher's questions can
be answered via statistical analyses of sample means and/or proportions. This report describes
the statistical issues involved in sample size estimation and presents formulas for determining
-from a statistical point of view - appropriate sample sizes for analyses of means and proportions
from one sample or from two independent samples.
Correctly Selecting the Best Product
-
Product tests represent one of the cornerstones of marketing research. The tests involve many
stages of preparation and planning, culminating in design, data collection and analysis.
Experience suggests that many of the tests have as a goal the identification of one product,
or a small subset of products, which is "best," in the sense of being most preferred or most
likely to be purchased. However, a potential inconsistency exists between the goal of selecting
the best product and the use of standard statistical tests to assess the significance of differences
among products.
Using a Cash Incentive to Heighten Mail Survey Response
-
Obtaining a high response rate is a recurrent problem facing researchers conducting mail surveys.
The use of a cash incentive has been one method of attempting to induce cooperation from respondents.
This paper describes an experiment in which sampled respondents were randomly assigned to incentive
and non-incentive groups, in order to determine (1) whether a pre-mailed $1.00 incentive produces
a significant increase in the rate of survey response and, if so, (2) whether the approach is cost
effective.
Measures of Relationship for Binary Data
-
Binary ("yes/no") data arise frequently in marketing research, especially in the form of
multiple-response questions. Resolution of important or interesting marketing issues often
requires going beyond basic tabulations of marginal response frequencies - to exploration
of relationships among respondent's answers to the various questions or response alternatives.
This paper describes various statistics that can be used to quantify the degree of "similarity"
or "relationship" among binary items. The statistics differ with respect to how the underlying
concept of "similarity" is defined, and some of them can be used as a basis for cluster analyses
of the items.
Some Methodological Issues in Product Testing
-
The development of food products follows a lengthy path culminating in consumer acceptance testing.
The product tests are very complex, with numerous issues to consider to obtain clear, unbiased
measurements of consumer preference. These issues cover areas of product technology, respondents'
sensory abilities and measurement of reaction to products. This paper is strictly concerned with
the third issue. The objective is to discuss four interrelated statistical/methodological facets
of food product testing. Consideration of these facets can help strengthen the interpretability
of the test results.
Clustering Concepts
-
There is great interest in the accumulation of information concerning the performance of concepts.
This information, typically in the form of some measure of purchase likelihood coupled with
marketing information, is used as the basis for models which predict future product profitability
from concept performance. Predictive ability improves considerably with the quality, breadth and
relevance of the accumulated information. However, many researchers may have scant data that can
be used with these models: potentially many recently surveyed concepts, but few brought to market,
and little idea as to levels at which to set marketing variables. In lieu of these data, a simple
approach to identifying successful concepts will be presented which relies only on some measure of
purchase intent. The techniques employed here will not estimate sales or ultimate product
profitability. Rather, the goal is to simply identify some subset or cluster of concepts which is
"better" (evaluated more positively) than others. A statement of statistical significance of the
distinctiveness of these clusters is also available.
Balancing Confidence and Power for Decision Making
-
Tests of consumer preference are performed to reduce risk to the manufacturer.A new product
formulation may be compared to an existing product using this type of testing with the intent
of "improving" the product line. Improvement may come from increased profits from existing share
or from increased share. Inherent in any product change is the chance that the product modification
is to the detriment of the manufacturer. The manufacturer risks loss in sales if the new product is
worse than the current one. Conversely, there is the risk of losing the chance to increase profits
by not producing a cheaper, yet equally preferable product. Unfortunately, statistical analyses
performed on product test data rarely take these risks into account. Research on Research Paper
Number 27 touches on this aspect. This paper presents an example of a product test, relating
monetary risks to levels of significance and power of the test of product preference.
The Use of Concern Scales as an Alternative to Importance Ratings
-
Research was undertaken to assess the virtues of a "concern" scale. Considered similar to importance
ratings in interpretation, the concern scale is an attempt to reduce the clumping of responses at
the upper end ("extremely important") of the rating scale and, in general, to increase the amount
of variability in the scale responses. A concern scale differs most noticeably from an importance
scale by the objective wording of the statement to be rated.
Sample Size Tables For Significance Tests
-
Tables are provided in this report to simplify the task of estimating sample sizes required for
significance tests concerning means and proportions (or percentages) obtained from a single sample
or from two independent samples. A detailed description of the logic of statistical significance
tests is presented in Research on Research Paper Number 36, and procedures for estimating sample
sizes are described and illustrated in Research on Research Paper Number 37. The tables presented
here were developed by application of formulas contained in the latter paper.
Color Testing: Color vs. Black and White Product Photographs
-
The purpose of this paper is to report the results of an experiment comparing ratings of food
product concepts obtained from two executions: A 4-color photograph stimulus, and a black and
white photograph stimulus.
The Effect of Population Size on Precision and Sample Size
-
Many marketing research projects are primarily enumerative in nature, the purpose being the
need for a description of the population of interest. For example, a small geographic division
of a national cable television company may be interested in various characteristics of their
subscribers to aid in programming decisions. A project could be undertaken to address this
objective by eliciting the desired demographic, behavioral, and/or attitudinal information from
a sample of their subscribers, in essence obtaining a description of the population of subscribers.
An Alternative to the Mean
-
Information on a population of interest is typically gathered by drawing a sample from that
population. The distribution of sample responses on a variable of interest is used as an
approximation of the population distribution. Usually the sample mean. This statistic is,
then, viewed as the value that "typifies," or characterizes the population of interest.
While this practice of relying upon the sample mean to represent the population is generally sound,
there are instances when the mean should not be relied upon. This paper discusses the use of an
alternative statistic, the sample median, in such instances.
Statistical Designs for Ordering and Rotating Products in Product Tests
-
This paper summarizes aspects to consider when designing a product test and is a sequel to
Research on Research Paper Number 41, "Some Methodological Issues in Product Testing."
Issues of design addressed in this paper concern the practical and statistical aspects of
physically presenting the products to be tested.
Statistical Analyses of Extreme Proportions
-
A number of product categories are highly fragmented, with dozens of brands or market entries.
The proportion of consumers who use many of these brands can be quite small. Yet, total industry
sales may be sufficiently large that even a small proportion of users translates to millions of
dollars in sales. A study of category users may yield brand usage proportions and confidence bounds
around such proportions are desirable to obtain upper and lower bounds on possible sales.
However, with small proportions, the usual confidence interval calculations may yield uninterpretable
results. This paper discusses the use of arcsin transformation of extreme, very small or large,
proportions as a way of estimating these confidence bounds correctly.
Factors Involved in Conducting Product Tests via Central Location Facilities
-
This paper presents an example of, and guidelines for, product testing that ensure proper control
of the set- up and execution factors listed above. The example is based on experience with tests
conducted in central location testing facilities. Observations can be generalized to other modes
of data collection, such as mall intercept facilities.
Censored Scales
-
Discrete scales are often issued to record responses concerning characteristics which really
vary along a continuum. Examples in attitudinal measurement are most prevalent where respondents
are given several choices (e.g., a 6-point agreement scale) which are used to capture attitudes
that fall along a continuum. Number of units consumed (e.g., glasses of beer), or demographic
characteristics, such as age or income, are also obtained by splitting a measurement continuum
into several discrete intervals. Respondents are then instructed to check the scale position or
interval which most accurately describes their intended response. Quite often the discrete
measurements of consumption or demographics are characterized by open-ended intervals at the
scale extremes. "More than 20 glasses" or "over 65 years of age" are examples of open-ended
upper bounds on scale for beer consumption and age, respectively. "Under $10,000 a year" reflects
the open-ended nature of a lower bound on income. This paper addresses some issues which arise
when discrete scales with open-ended extremes are used to measure characteristics which really
vary along a continuum.
The Effect of the Number of Scale Points in Measuring Product Perceptions
-
Lists of characteristics are often used by researchers to assess consumers' perceptions of a
product or concept. For example, respondents may be asked whether or to what extent they agree
that the characteristics describe the product or concept, or how much they like various attributes
of the product. Prior to data collection, the researcher must decide what type or form of scale
should be used. Scales may vary not only in the intent (e.g. performance, satisfaction, description,
agreement) and the semantic descriptions of the scale points, but also in the number of scale points.
Two experiments were conducted focusing on the latter issue, with particular reference to agreement
scales. This paper summarizes the results of these experiments. Specifically, the research issue
addressed is whether the conclusions about product or concept differences are affected by the number
of points used in an agree/disagree scale.
Test of Differences Between Correlated Proportions
-
This report concerns the statistical significance of differences between proportions obtained
from the same sample of respondents. Such proportions are correlated and thus require tests
different from those used to compare proportions from two independent samples. Procedures for
constructing confidence intervals for differences between correlated proportions are also
described. Finally, some comments are made regarding estimation of appropriate sample sizes
prior to data collection.
Use of a Bayesian Orientation in Product Testing
-
A Bayesian approach allows the researcher to incorporate into the research process prior
knowledge or information which can be merged with new data (such as the results of a most
recent preference test) to form a more complete understanding of this "state of nature."
Triangle Plots: Graphic Display of "Just Right" Scale Data
-
Data gathered in this fashion are typically summarized by dividing responses to the
"just right" scale into three categories: "too little," "just right" and "too much,"
and then determining the percentage of responses falling into each of the three categories.
The intent is to develop and compare profiles of the products being tested. When numerous
characteristics are involved, tabular approaches to this task can be unwieldy, especially
if several products are involved. This paper presents a graphical technique for displaying
such profiles, thus facilitating product comparisons and the assessment of test product
strengths and weaknesses. The technique is called the triangle plot.
Estimation of the Effect of Attributes on Overall Liking
-
One of the fundamental issues in brand related marketing research concerns the ability to
estimate the importance of an attribute, such that manipulation of that attribute causes a
change in overall acceptability of a brand. It may be fairly simple to define the characteristics
of an important attribute. For example, an attribute which in some statistical sense has an
association with responses given to overall liking of a brand is one of the more highly respected
criteria. However, estimation of that association from which importance is derived is quite another
issue. Traditionally, measures of importance have been expressed as functions of statistical
relationships such as regression coefficients, correlations, the amount of variance explained
and so forth. This paper describes a statistical relationship, expresses the association between
an attribute and overall liking is attributable to or influenced by positive perceptions of the
attribute. Conversely, the statistic can be interpreted as the change in the proportion of
respondents liking the brand if it no longer provided adequate performance on the attribute
being studied.
Conjoint Analysis for Product Strategy Decisions
-
The tremendous appeal that conjoint analysis has had for the marketing research community during
the past few decades is due to the fact that from a relatively small amount of data, the Conjoint
Model provides a way of predicting preference for a potentially large number of products and
services which have never been directed evaluated. After the quantification of value systems
has been completed, the marketing researcher is in a position to simulate the consequences of
numerous marketing scenarios. For example, one could estimate the impact of the introduction
of a new product to the marketplace or changes in the specifications of existing products:
both yours and your competitors. Armed with this kind of information, it is possible to develop
products which maximize share/revenue/profits, minimize the cannibalization of existing business,
and target specific groups of people.
Mail Panels vs. General Samples: How similar and how different
-
Multi-purpose household panels, commonly referred to as "mail panels," offer several advantages
for researchers: (1) response rates are generally quite high; (2) strong respondent cooperation
facilitates true panel design studies (and diary studies) with relatively low rates of sample
attrition; (3) customized samples can be selected "off the shelf" (or via inexpensive mail
screening) including samples of low- incidence populations, saving screening costs; (4) samples
can be nationally balanced (made demographically representative through quota sampling) on multiple
variables; (5) much respondent and household background information needed for data analysis is
already available, saving time or space in the survey; (6) use of panel samples facilitates
otherwise very difficult or expensive data collection, such as national surveys of children,
brand loyalty studies, conjoint measurement surveys requiring complex modes of questioning,
and others.
An Analysis of Importance Ratings
-
Importance ratings, in one form or another, have become a rather ubiquitous commodity in
marketing research, found in a variety of studies. Yet, this approach to assessing the weight
or value given to various product characteristics and benefits during the purchase decision
making process has often been criticized. Many view the resulting measurements as lacking
discrimination, both between benefits evaluated and among respondents (e.g., "too many people
said too many items are extremely important"). Further, the resulting ratings are often perceived
as not reflecting the true nature of product benefits or motivations, hence the need for
"derived importance" to get at what consumers truly think and feel.
Effects of Various Rating Scale Descriptors and Administration on Response Profiles With
Telephone Interview Data
-
Ratings given to attitudinal questions administered over the telephone are influenced by the
order in which response alternatives are described to respondents, the inclusion of a scale
mid-point descriptor, and the choice of end-point labels. If the questions are different,
researchers shouldn't expect the answers to be the same.
Online Consumers: Beyond Fiction to Fact Distilling Reality from the Hype
-
The transformation of the research industry is a direct result of the astounding technological
advances impacting all facets of business and society today. At the root of this myriad of
change is the day-to-day impact on people, the very people who are our consumers and our
respondents. This paper will examine the Internet's impact on consumers, both attitudinally
and behaviorally, with particular emphasis on those facets directly related to the research
industry’s quest to gather sound, viable consumer inputs via this burgeoning medium.
An Examination of Online Sampling Techniques
-
One of the key challenges of executing online research is that of sampling: where and how
to acquire Internet sample, how to control the composition and consistency of samples and
what is the optimal process for sampling on the Internet. Given these issues, along with
the proliferation of badly designed surveys being forced upon web users and the heightened
sensitivity to spam-like invitations, online panels are being created in hopes of providing
better solutions to these sampling issues.
Is this Art of Science?
-
Market segmentation is a necessary step before creating any integrated marketing communications
plan in order for it to lead to brand equity!
