Social scientists have struggled for generations with the issue of what people say versus what people do. If we as researchers cannot trust what people tell us, how can we predict what will happen with the products and services we research? Trying to comprehend that understanding and prediction are not the same thing makes many of us uncomfortable because we have been taught that understanding is the key element of prediction and it is, but it is not the same thing.

When I was a child and got sick, my mother would prepare remedies that had been passed down from generation to generation. There was a particular order, almost a ritual that she followed in preparing the medicines. She had no idea why a particular remedy worked, but she knew that if she prepared the medicine in the way she had been taught, the medicine would work. She had no understanding of the underlying biochemistry, but she was able to prepare something that would help. She had prediction with no understanding.

When I took my first course in research methodology, I encountered a different example of prediction without understanding. A small town in Minnesota was experiencing an interesting pattern. Alcohol sales there were highly correlated with the amounts the local churches received in their Sunday offering baskets. The first hypothesis for this was based on guilt. People got paid on Fridays, drank on Saturdays and came to church on Sundays with a guilty conscience and a hangover. But social scientists examining the problem discovered that most of those living it up on Saturday, the drinkers, didn’t go to church. So what was going on?

There was a latent variable driving both phenomena. When times were good, income would increase and some people chose to spend it on alcohol, while others chose to donate more to their local church. Alcohol sales and church offerings are highly correlated. You don’t have to know about the underlying latent variable to predict what is going on. So we have prediction with misunderstanding. How often in life does this happen? Many, many times.

Can we have understanding without prediction? Well yes. When mainframe computers became commercially available in the 1950s, one of the first applications they were used for was processing information on weather patterns. The hope was that we could process enough information to predict natural disasters days, perhaps weeks, ahead of time. But as history has shown, that kind of forecasting is almost impossible. From 1960 to 2008, we have been able to move to a 14-minute warning period for a tornado for example. The factors that impact the development of natural disasters are so many and involved in such subtle relationships that modelling them with any precision to allow for more warning escapes us.

Human thought and behaviour too are complex. There are many reasons why what we do looks to be inconsistent with what we say. First, there are issues of recall. Recall is problematic and far from perfect and gets worse the farther back in time we go. Think about something that happened yesterday. You can probably describe the event accurately, perhaps even the exact time it occurred. Now think about a similar event that occurred a year ago. How much can you recall and how accurate is your recollection about that event?

Recall is also impacted by salience – how important a certain event was in your life. If we tried to process all the information that hits our five senses, we would suffer information overload. The human mind condenses information, processes what’s important and dumps the rest.

Think about walking down the street. Do you study the sidewalk? Do you notice the colour of the cement? Do you know about how many steps you take in between the expansion joints in the cement? The distance between the expansion joints in the sidewalk is a fact that you encounter, but quickly throw away. It is irrelevant and not salient to the problem of getting from where you are to where you need to be. Context is critical to determine what is important and what you will recall at a later point in time.

Another interesting factor about communication is that it is interactive. Kind of obvious isn’t it? But think about it. The first problem with human oral communication is that it is imperfect. What a person says and what a person hears can be two different things. From our early years we are taught that words have definitions and these definitions provide an aura of precision. But language is built on a base set of words that are not precise. Select a word, look up its definition and then look up the definitions of the words in the first definition. Keep doing this and somewhere around the fifth level of the exercise you will start seeing some of the words used in the very first definition. Language is not as precise as we think.

The second interesting characteristic of communication is that people impact each other when they communicate. That seems obvious, but it also has serious research implications. Since the 1960s, the National Science Foundation has supported a programme in the United States called the National Election Studies. At first these were surveys carried out before and after national elections using the same panel of people. As the research progressed, more money was added and the people recruited to be on the panel were interviewed as early as the primaries.

As this research progressed through the years, the final results were very close to the distribution of the final vote. However, the proportion of respondents who said that they voted was significantly higher than what was experienced in the general population. For years, social scientists assumed that respondents at the post-election survey were lying about actually voting. It is socially acceptable to do your civic duty so respondents were supposedly trying to face save with the interviewers.

Someone got the bright idea to sample the respondents and spend the time and effort to search public voting records to determine if the respondents voted. To everyone’s surprise, they discovered that people had not been lying. They did vote. The fact that these people were repeatedly interviewed about a national elections heightened their interest in the elections. Some of these people who had not paid attention to the elections were now sensitised to it because of the interview experience. This forced us to face the fact that when we talk to people, we impact them. We cause them to think about what we talked about and we modify their attitudes and subsequently change their behaviours.

The implications for survey research and especially for panel designs with no cross-sectional control groups are obvious. When we interview people, we affect their attitudes and behaviour and the respondents will be different because of this interchange.

The anthropologist Margaret Mead pointed a whole generation of social scientists in the direction of analysing context. Context is what we have to deal with when we develop questionnaires to query respondents. A questionnaire by definition is a closed system. No one would argue that we account for every factor that impacts a person’s decision to do something. We limit what we ask in order not to burden the respondent and gain acceptance. The interview is a closed system. It helps us to understand the relationships between variables, but it closes off from other factors that alone, seem small, but can in an open-system context cause a serious deviation in behaviour. Mathematicians even have a name for this phenomenon: They call it the impact of small deviations.

One of the famous observations that came from the simplification of chaos theory was that a butterfly flapping its wings over North America impacts the weather in Europe. That is what makes weather patterns so difficult to predict. We can understand general patterns and how one factor impacts another yet once we get into the real world, we can’t control all the little things that accumulate and have a major impact on what are observing.