Receive our complimentary News,
Tips & Insights eNewsletter
  • Election Poll Failure Offers Lessons for Business Research
  • Businesses can learn important lessons from how the polls failed to predict the winner of the recent presidential election. As a result of this failure, there is greater awareness of the limitation of survey research methodology. A November 21, 2016 Wall Street Journal article “Advertisers Search for Middle America” by Alexandra Bruell and Suzanne Vranica discusses how, in the aftermath of what happened with the election polls, major advertisers are re-evaluating the way they do market research.

    As I write this, however, there is talk of an election recount. Regardless, methodological issues associated with surveys and other data collection methods, as well as with data analytics, are becoming increasingly important in today’s more data oriented era. So, recount or not, it is a good time for companies to reevaluate their research and data gathering methodologies, just as some major advertisers have been doing in the aftermath of election polls.

    I am not a political analyst. But, my expertise does cover issues to consider when re-evaluating the approach to survey research and the use of data. The limitations of and alternatives to survey research is a topic I’ve been involved with going back a number of years. In the fall of 1994 issue of Marketing Research: A Magazine of Management and Applications, I authored a peer reviewed article on this topic titled “Database Marketing Research”.

    My article essentially compares and contrasts the advantages and disadvantages of using surveys versus using behavioral data, such as the information in a database of a business’s transactions with its customers. Behavioral data reflects what customers actually did--for example, what they purchased--rather than what they said when asked. My article was published long before Amazon became well known for using behavioral data to suggest additional items to its customers based upon what others who bought a particular book were also buying.

    About a year after my article ran, renowned Medill School-Northwestern University professor Don E. Schultz, who is known as the founding father of integrated marketing communications, wrote an article urging marketers to use behavioral data rather than relying on attitudinal data. In his article, Schultz cited my article and included it in a very brief list of recommended reading.

    My article discusses how the surveys in such widespread use were not the only way to obtain information about the market, and in many situations, they might not be the best way. Although my article focuses heavily upon the use of transactions databases, which reflect the purchasing behavior or other actions taken by customers, database information is not the only alternative to survey research.

    As the Wall Street Journal article points out, advertisers are considering greater use of ethnographic research in the wake of polls failing to predict the election. Ethnographic research involves observing people rather than merely asking them questions. Use of this type of market research was on the increase more than 10 years ago. Back in 2004, when the use of ethnographic research was on the upswing, I was quoted in a Minneapolis Star Tribune article, which reported on the increasing popularity of this methodology.

    In fact, an example for my “Database Marketing Research” article entails a circumstance where the transactions data differed markedly from the findings of a survey asking about purchase frequency. This was not a high volume situation, so observation was able to confirm that the transactions data, not the survey, more accurately described purchase frequency. This was probably due to the survey over representing heavier users.

    Although these kinds of survey representativeness issues were occurring back then, which was over 20 years ago, today they are of even greater concern. With modern technology, such as cell phones and caller ID, it is now far more difficult to get a random sample for a telephone survey, and survey representativeness can suffer. Thus, the kinds of methodological issues that the election polls brought to the forefront are even more important today than they were years ago.

    Furthermore, as the Wall Street Journal article points out, advertisers are considering hiring more staffers from rural areas as a result of the election polling failure. This could improve the understanding of rural markets and help avoid misinformation that might occur if surveys under represent those areas. As the Wall Street Journal article explains, this is being done as “advertisers are reflecting on whether they are out of touch with the same people–rural, economically frustrated, elite distrusting, anti-globalization”—that fueled the election’s surprise outcome.

    Since the right methodology can enhance the value of information, it is encouraging to see the election polls lead to reinvigorated emphasis on exploring these kinds of alternatives to survey research. This does not mean that ethnographic research, with its greater emphasis in the wake of election poll failure, is always the preferred method. But, it underscores the value of thinking through what the information really means, especially in light of how the data was obtained and how it might differ if other methodologies were used. This is the case regardless of the methodology—whether it’s surveys, the observational methods of ethnographic research, Big Data/data analytics, or any other approach.

    And, like survey research, other approaches also have their limitations. With data analytics, there can be glitches when data is not cleaned up. For example, problems can occur when a person is listed in the data under two different names, or when an entry to the data is sometimes misspelled or abbreviated, but sometimes entered correctly in full. And, with the more recent emphasis on AI (artificial intelligence) and machine learning, the results can be troubling when an algorithm is incapable of making the distinctions that a human can.

    Thus, the lesson from the election is not merely to rely less or surveys and polls. The lesson is to pay more attention to a methodology’s advantages versus its limitations. Despite possible glitches and methodological issues, research and data have tremendous potential to play a valuable role. So, recognize that there can be reasons why different methodologies can yield different results and try to understand what is going on when that happens. Look for and use approaches that work well for the kind of information you are seeking.

    ©2005-2017 All Rights Reserved. For reprint permission, just give us a call.

    Ezop and Associates
    La Grange Park, IL
    (708) 579-1711