When Good Samples Do Strange Things

Despite my utter faith in sampling theory and data collection quality, there's a part of me that's still amazed whenever I crack open a book of tabulations and see how closely sample findings match known characteristics in the population. Plus, there's that mixture of satisfaction, pride, and relief I feel when, in the middle of a presentation, I hear the sweet sound of someone remarking that "your numbers line up almost exactly with our internal data."

On the other hand, there's little that contributes more to the graying of my hair than the flip side of this phenomenon--opening a set of tabulations and seeing a distribution that just can't be right (too many older respondents, or younger respondents, or men, or women, or product users, or non-users . . .), or getting to chart three in a presentation and hearing someone say--"that's strange, our product penetration is lower than five years ago, not higher."

Fortunately, such situations are the exception rather than the rule (meaning I'm not quite completely gray yet). But when they do happen, I try hard to resist panicking and instead look at the issue as rationally and systematically as I possibly can. In fact, my strategy tends to fall into four categories--prevention, investigation, communication, and resolution.

The prevention step (as you'd imagine) takes place before the problem even arises--and involves things like making sure we're using question wording and scale points that are consistent with what's been used in the past, and getting preliminary data runs to spot potential problems before the fieldwork is completed.

Investigation tends to be a no-holds-barred look at everything and anything that could have caused the seeming discrepancy--from checking that codes and weighting are correct, to looking for internal consistency with comparable questions, to calling back respondents to check on whether their interpretation of the question was different from what we intended.

Communication is another key step, assuming steps one and two haven't solved the puzzle. Indeed, we can often get to the bottom of things by discussing the discrepancy with other professionals on our staff, or the client, or outside experts, to pick their brains about what may be going on.

And hopefully, all of this leads to resolution, whether it be confirmation or correction--and another month of keeping the Grecian Formula at bay.