Damn lies and statistics: Bad research proliferates online

If you're having trouble figuring out where to spend your online marketing dollars, your confusion is understandable. Consider the research: “Are Brands Ignoring Facebook's Interactive Potential?” asked eMarketer early this year in presenting the results of an A.T. Kearney analysis that showed that 28 of the top 50 brands on Facebook didn't respond to a single customer comment in 2011.

Just a few months later, though, Marketing Charts reported a different trend. “Customer Conversations a Rising Corporate Strategy,” it said, citing results of an InSites Consulting study.

The irony is that both surveys reported similar findings, yet two respected sources analyzing two similar sets of data came to two very different conclusions. Such is the conundrum facing anyone searching for truth in today's information-saturated environment, where statistics are plentiful but facts are elusive.

Forecasting is even less of a science, often amounting to little more than an informed finger in the air. Channelnomics recently presented a roundup of statistics about the size of the cloud computing market. AMI-Partners said it would hit $95 billion in 2014. 451 Market Monitor, on the other hand, said it would be $16.7 billion by then. Projections for 2016 ranged from $86 billion (by International Data Corp.) to $112 billion (Gartner Inc.).

Wow.

Depending on whom you believe, this market is either in hypergrowth or big trouble. How are you to know the truth about social media marketing, or indeed anything, when most forecasters don't even publish their methodologies?

Decision-makers have a seemingly insatiable appetite for research, and there are plenty of marketing departments, one-man consultancies and bloggers who are happy to oblige. The problem is that a lot of them don't know much about how to conduct research.

Social media and the Internet have compounded the problem. Anyone can now create an online survey for short money and publish a PowerPoint deck that looks like it came from Gallup. But access to a SurveyMonkey account is not the key to wisdom.

Bad research hurts everyone. It leads to indecision, skewed perceptions and bad investments. It can also be a self-fulfilling prophecy: If enough marketers believe that Facebook is a bad place to spend their ad dollars, then perception becomes reality.

Before you make a decision based on research by an organization you've never heard of, ask a few questions.

• How was the research conducted? Online surveys are a popular tool because of their low cost, but they're inherently flawed because they measure, well, the opinions of people who take online surveys. Sampling errors are compounded when research asks leading questions or fails to list all possible answers. The most reliable sampling techniques are, unfortunately, also the most expensive: telephone and U.S. mail. Online polling has all kinds of reliability problems.
• Who answered the questions? Sample quality is a big problem. While it's possible to conduct a statistically valid survey with fewer than 40 respondents, you need considerable research expertise to do so. Even then, the results may be more or less valid. A large number of responses doesn't indicate validity if the sample isn't representative of the people whose opinions you want to measure.
• What questions were asked? I've seen survey forms from well-known media companies that would make a statistician cringe. Leading questions, assumptions about the respondents' knowledge level and insufficient response options are all common problems. Survey design is a science, not an art.

  Did you know there is a statistically significant difference between asking “Do you plan to buy laundry detergent within the next 30 days?” and “Do you plan to buy laundry detergent within the next 30 days or not?“ Before you believe the results of a survey, ask to see the survey form and run it by someone who knows how these things are done.

• Does the researcher have an agenda? Marketers know that research is a great way to validate their businesses if the results are in their favor. That doesn't mean marketing-generated research is bad, but you should look twice at the study by, for example, a direct mail company that forecasts explosive growth for direct mail. Ask the questions above before buying in.

When factoring research into a decision, consider the source and don't be afraid to ask for details. If it fails the smell test, walk away. The old cliché about lies, damned lies and statistics has never been more true.

Paul Gillin is an Internet marketing consultant and the author of three books about social media. He also writes the New Channels column in BtoB. He can be reached at paul@gillin.com.