RESEARCH AND THE RELIGIOUS RIGHT

Nothing Seems As Foolish As A Fundamentalist Who Has Lost His Fundamentals.

By Erwin Ephron

Truth is dead and faith is lost along with misdirected emails. There's is no certainty in survey research, only probability samples. And hasn't been one of those in decades.

Still, the research clerics circle the wagons to defend the faith. Their doctrine remains stubbornly statistical. Probability sampling and high completion rates separate the clean from the unclean.

Statistics has a strange provenance for Holy Writ. It was invented at the gaming tables of 17th century Paris to calculate the odds. This gambler's friend grew into the mathematical underpinnings of most survey research. But human behavior is more rowdy than roulette and people are less docile than dice and therein lies the problem.

Most of our research polls people on behavior, testing that fraction of the population we hope will mirror the whole. We expect our Nielsen sample of 11 thousand to tell us what 272 million are viewing. Flim-flammy as it seems, it works if it's the right 11 thousand.

But for the Nielsen sample to ring true statistically, each person in the US must have an "equal or known probability" of being chosen. That is the deal-breaker. More than half of the people chosen for the Nielsen sample don't participate, so the basic principle of sample-based research is in a word, "twaddle." [1]

Researchers have stamped the coin of this deceit by not admitting to the problem. Most studies still come packaged with "standard error" tables, which work only with probability samples. These misplaced estimates of reliability serve as a narcotic to numb us to the awful truth. We have no firm idea how good, or bad, the numbers are. It would be more honest to mark the pages "reliability unknown."

Then there is the equal issue of the validity of the measurement itself. Facts reported in audience surveys appear to be much more than they are. All we know for certain about an NSI-reported viewer is someone in the household made a mark on a line in a diary. An NTI-reported viewer is someone who pushed a button. An MRI-reported reader answered, "yes" to a readership question.

These acts -- marking, pushing and nodding -- are far removed from the viewing and reading interpretations we give them. Researchers refer to the difference between what we want to measure and what the research is actually measuring, as "validity." Different techniques can produce vastly different numbers; witness the new Arbitron PPM measurements of TV and Radio.

Why is this tiresome carping important? If the part of the population not in the sample uses media differently, we may be spending too much (or too little) and buying the wrong things. And right off we know they behave differently in some ways because they chose not to participate.
If the techniques we're using aren't measuring what we think they're measuring, then the search for accountability may be drowning in disinformation.

If we're selling bad numbers to clients, we're part of a con.

But I go too far. We don't want a cure that kills more than the cold. Research is not supposed to be Truth, just information, some good, some bad, but most of it useful. The best way to cope with this muddle is to walk around it. Discard some of the old rules of best practice, since they are badly broken, and think about the many ways there are to collect data and judge its usefulness.

Internet-based research is a fine example. It is fast and cost-effective. I don't see how we can dismiss it out-of-hand because it's "biased". All research is biased. Here at least, the infirmity draws attention.

The diary is dreadful, (you can almost feel the knees jerking). That's true enough for TV with 75-plus channels and short-interval viewing. But it's a cost-effective technique to collect limited habitual behavior, like radio listening or commuting patterns for outdoor.

Modeling isn’t making up numbers. It’s a useful and cost effective statistical technique for filling in missing data.

And it's all right to argue with research. It is not ordained, it's gathered by man. When the decision is small, don't sweat it. But when the stakes are high, don't buy the book either.

ARF's Jim Spaeth points out that we rarely do a study without knowledge from prior research, but we seldom apply it as well as we should. Others suggest we need to search out patterns instead of chasing aberrations. We need to "triangulate" using several sources and experience.

And let's get past our sanctimonious certainty. Today using research is like having sex with a stranger. You have to think about it a lot and be careful.

I wish I had better answers, but there is no easy fix.

Truth is dead and faith is lost along with misdirected emails. There's is no certainty in survey research, only probability samples. And hasn't been one of those in decades.

Yes it’s uncomfortable, but a good time for smart people to be alive.

[1] MRI is remarkable with a cooperation rate of 70%. Other cooperation rates cluster in the low 30 percents.

- December 1, 2002 -

^ top of page ^      < back <