Last week a cartoon in The New Yorker caught my eye: a middle-aged man in a conference room, pointing to an easel in front of confused-looking coworkers. On the easel is a drawing of the presenter, sitting on a Pegasus-like winged horse, flying over a rainbow.

The caption reads: "Impressive, Meyers, but let's stick to your quantitative projections."

Suddenly I felt like I was back in our conference room, in the middle of a staff meeting we had about a month ago. We had gathered to discuss a journal article on focus group research that was passed along by one of our long-time clients. Everyone had already read the article, and we convened to go over some of its more pertinent points.

While we have qualitative and quantitative specialists here at The Taylor Group, most of our staff members are experienced in both qualitative and quantitative work. But I've noticed that most people have their favorites. And these feelings were certainly on display during our discussion.

Now you should know that when our staff gets together to discuss an issue, people really get into it. There's no holding back. Recently, during an internal seminar on writing, a 10-minute debate on contractions broke out, with both sides passionately making their cases, trying to persuade the fence-sitters to join their group.

This meeting, like the infamous contraction discussion, started out calmly enough. We were pretty much in agreement that we enjoyed the article and began to discuss some of its more significant aspects.

But after a while -- and I have to admit I'm not exactly sure how it happened -- the tenor changed. Sides were taken. Pro-qual and pro-quant teams formed and suddenly we were debating the merits of focus group research. It was as if the man on the winged horse had swooped into the room and changed the dynamic of the discussion.

The arguments were made: The sample size is too small, and participant selection is not sufficiently random, to be representative of the whole population. There's always a chance that some group members echo the sentiments of others rather than disagree, or make comments for the sake of effect. Moderators in focus groups are too important and end up serving as participants, which could affect the findings.

Political observers emphasize "controlling the message." And this conversation was suddenly being controlled by the pro-quant side. After all, we'd gone from discussing focus group techniques to talking about whether focus groups were legitimate research. Advantage, pro-quanters.

But in response, most of our staff settled into the pro-qual camp -- a coalition of partisan pro-qualers and moderators who found the anti-focus-group position of the pro-quanters too militant. They kept hammering on one main point -- that focus groups, when conducted properly, do exactly what they are designed to do: provide the emotional depth and breadth of customer opinion within which statistical data is lodged. And that argument resonated with most of our staff. The pendulum had swung toward the pro-qualers.

Of course, that didn't stop the vocal minority from trying to persuade their coworkers. We have a few newer colleagues who have joined us within the last six months, people who haven't yet formed methodological preferences, and they represented the undecided voters that each camp tried to drag into their own.

On and on it went. About an hour later, with perhaps a few feathers ruffled, we had completed our discussion.

Much like in politics, few minds were probably changed. But just as a thorough airing of the issues is always good for democracy, this discussion was good for us.

It allowed the quantitative specialists to raise some legitimate issues they have with focus group research. And it forced the qualitative researchers to defend what they do, how they do it, and what they believe in.

An article in the Week in Review section of last Sunday's New York Times titled (or at least subtitled) "A Polling Mystery" centered on the following point: "Support for the initial invasion of Iraq, as measured by a question The New York Times/CBS News Poll has asked since December 2003, increased modestly compared with two months ago."

Specifically, in mid-July, "42% of those polled said the United States did the right thing, and 54% said the United States should have stayed out of Iraq. The last time the question was asked, in May, 35% said taking military action against Iraq was the right thing and 61% said the United States should have stayed out."* With a margin of error of 3 points, this 7-point change was statistically significant.

The article went on to explain just how "counterintuitive" this finding was, because "none of the other war-related questions showed change. Mr. Bush's approval rating had not changed. Nor had approval of his handling of Iraq. The level of support for Mr. Bush's decision to send more troops to Iraq -- the 'surge' -- was about the same as it had been in past polls. Support for the decision to go to war had risen modestly and nothing else in the poll could explain it."

So the Times and CBS decided to do another poll, and found the very same thing. The article ends with this polling "mystery" unsolved: "What was driving the change still wasn't all that clear, but at least the paper had confidence in the results and was able to report the findings."

I have two big problems with this article.

First, on this polling "mystery," I am surprised that the New York Times/CBS polling people didn't explain this "mystery" to the reporter. They are solid survey research professionals who know what they're doing, and I'm certain that deeper analysis of the data would have shed much light on what's behind this sudden change in public opinion. At the very least, an examination of change among demographic and political affiliation would have helped explain the sudden shift. There was none of this in the article, however; so I can only assume that "a polling mystery" is a sexier story than "an apparent polling mystery solved" would be.

• It makes me curious, though: What do you think was going on? What might cause this particular shift in public opinion, when nothing related had changed at all?
• And there's an obvious lesson here as well for all survey research professionals. Whenever we see a "mysterious" finding in our data, we need to leave no analytic stone unturned in solving that mystery. It's our responsibility. (And as troubling as it can be -- and we've been through it many times -- it's part of the fun of deep data analysis.) One has to be a genuine data detective to solve such mysteries, but that's what we're here for. I'll bet anything the Times/CBS polling people have some answers.

My second big problem with this article is a lot more subtle, and a lot more insidious as well. I was surprised to see the word "modest" used twice in this article to describe the recent, sudden 7-point increase in public support for the initial invasion of Iraq. Surely if the president's job rating were to decline suddenly by 7 percentage points over the same two-month time period, the word "modest" would not be used to describe such a drop. Words like "precipitous," "dramatic," and "major" would be used; and such terms would be entirely appropriate with a change of this magnitude in such a short period of time. I can't figure out why such words weren't used in this case, and "modest" was. Editorial bias is all I could come up with in answer to "why."

• Do you have an alternative answer?

*Actual question wording: "Looking back, do you think the United States did the right thing in taking military action against Iraq, or should the United States have stayed out?"

Partnership -- it's our approach to each vendor relationship. We don't just want a fieldhouse to open the phone lines and make thousands of calls, or a focus group facility to call through their database searching for the right person. We want to partner with each vendor to map out a research plan, review the most practical and cost-effective way to conduct the study, and work together through the entire lifecycle of the project.

Forming partnerships is important for two reasons:

1) Top-notch results -- our experience on both easy and challenging studies with our vendors has given us insight into pooling our collective research minds to deliver top-notch data on each project; and

2) Trust -- even on the tough studies, we know our vendors will stick with us and do what it takes to get qualified focus group participants or continue making calls on a low-incidence telephone study until every complete is achieved.

Each vendor we work with -- on both qualitative and quantitative projects -- goes through a rigorous initial review process and is continuously monitored. All quantitative vendors provide information about their operating procedures and practices, including interviewing staff/supervisors and their experience, how sample is handled in the field, and quality control measures. Online vendors also provide information about their panel, how it's maintained, their programming capabilities, etc.

On the qualitative side, knowing the facilities we work with helps on many fronts. When we discuss a project in the RFP stage, our facility contacts offer guidance on how to reach a tough segment, whether their market can support the research, and how to move forward. During the recruiting phase, our relationships with facility directors help tremendously when we run into problems. Facility directors and project managers will work with us to brainstorm creative recruiting techniques (such as advertising spots, referrals, e-mail blasts, etc.) and pull out all the stops to ensure we have full groups.

We've found that knowing our vendors, and nurturing and valuing these relationships, has many times made the difference between research that just meets expectations and a project that exceeds expectations.

Until the 1970s, face-to-face was the preferred method to conduct quantitative market research studies. But as telephones became so prevalent, it was more efficient for most studies to be conducted by phone.

In the late 1990s and early 2000s, the Internet emerged as the newest opportunity. Some studies that would have previously been conducted by phone were done online. More and more experts started (and have not stopped) talking about the opportunities and dangers posed by this method.

Now another opportunity -- and challenge -- has arrived: the emergence of cellphone-only households. Almost 15% of American households do not have a landline phone and rely solely on a cellphone.

So what is the solution?

For random-sample studies, we can add a sample portion where we call people on their cellphones. For cost-related reasons, we typically screen out people who have a landline and only include those people in the cellphone portion who cannot be reached on a landline. Afterwards, we weight the data accordingly. These cellphone portions are substantively more expensive than landline portions, for three reasons:

(1) We typically suggest an incentive to the respondent since we cut into their everyday minutes and most people are on a limited-minute plan. (This also lengthens the survey since their address needs to be taken down.)

(2) This is a population that is hard to reach: young people are less likely to take time to respond to surveys than older people, so the response rate is typically lower.

(3) Once we reach a person, we need to make sure he/she is not distracted by traffic or other things people do when talking on their cellphone.

If we do not adjust to this development, we are in danger of missing a crucial segment of the market: the cellphone-only segment tends to be younger and more mobile. If we do not include them in our samples, our data would be incomplete.

This adage has probably never been truer than it is today. More often than not, the success of a product can hinge on how effectively the marketer reaches out to those ages 10 to 34 -- capturing their attention and establishing them as loyal, lifelong customers. As such, The Taylor Group's clients are more interested than ever in knowing what makes kids tick -- and in ever-growing numbers they are turning to research to help figure this out.

Conducting research among kids can be immensely fun, exciting, and rewarding. However, it can also present some major-league challenges -- challenges we, our clients, and our fellow research professionals need to fully recognize and come to terms with. Anyone who thinks they can simply duplicate their approach to conducting survey research among adults in their kids research could be setting themselves up for a rude awakening -- and potentially a legal nightmare.

While there are many challenges involved with interviewing kids, in this entry I'd like to focus primarily on what is perhaps the thorniest and most high-stakes issue -- the legality of interviewing minors online. This is definitely not a good time to be blissfully unaware.

By way of illustration, Taylor conducts an annual, online survey of high school juniors and seniors about their college and career aspirations. When we kicked off this study several years ago, there were relatively few clearly defined limitations governing who we contacted and how, and what we offered them in return for their input. Over time, this has changed radically.

With the evolution of the Internet, the laws and codes of conduct governing its use for survey research have also undergone major transformations. As such, this year we took a long, hard look at whether our approach to surveying high school students was still appropriate. We consulted with legal counsel, experts in academia, and colleagues in the research field. We also requested clarification on AAPORnet (a leading research-industry forum) from anyone who might have insight.

What came of all this? Initially, considerable confusion -- and a bit of panic that we might have to radically alter our approach to this study. Among other things, we were told that to interview anyone 18 or younger online (19, in some states) you have to seek parental permission in writing. We were also told no one under age 18 can be required to abide by "contract terms," such as contest rules governing the disbursement of prizes to contest winners in a prize pool. You name it, we heard it. Clearly, even seven years after the enactment of the Children's Online Privacy Protection Act (COPPA), there is considerable confusion within the research community (and even among lawyers) about what you can and cannot do when it comes to interviewing minors online. Sifting through all of this information, we ultimately uncovered the following key details:

• In all cases, you need to secure written, parental permission for all minor children under the age of 13 to participate in an online survey.
• To distribute incentives or prizes to anyone under 18, you may request only his or her e-mail address and use this only for identification and delivery of the promised incentive. You may use that address only once and solely to inform the minor of his/her winnings or to deliver the incentive. Since this does not allow for the collection of physical delivery information, it is in your best interest to make the incentive or prize distribution electronic. Once delivery of the incentive or prize announcement has been made, the minor's e-mail address must be deleted from all databases maintained by the research organization and its affiliates.
• To contact a minor more than once, you must use a so-called multiple-contact exception, for which you will also need to collect the parent's e-mail address(es) and provide him/her with direct notice of your information practices (i.e., privacy policy) and offer an opportunity to opt out. The law prohibits you from using the child's e-mail address for any other purpose, and you must ensure the security of the information, which is particularly important if the contest runs for any length of time.
• If you need to obtain a mailing address and wish to stay within the one-time exception, you may ask the child to provide his/her parent's e-mail address so you may notify the parent if the child wins the contest. In the prize notification e-mail, you can ask the parent to provide the home mailing address to ship the prize, or invite the parent to call a telephone number to provide the mailing information.
• Whether those over 13, but under 18 (19 in some states), are legally bound by contest rules or other eligibility requirements they formally accept in return for the right to participate in an online survey and potentially benefit from any incentives offered is determined by local, not national, law.

I hope this assessment of the laws surrounding minor participation in online surveys will be helpful and time-saving to those reading this entry. The FTC website (www.ftc.gov) has further information about COPPA and related topics. We welcome and strongly encourage replies that expand upon or clarify anything we have said above, and would be interested to hear what impact these requirements might have on other researchers' plans for conducting online research among minors.

Despite my utter faith in sampling theory and data collection quality, there's a part of me that's still amazed whenever I crack open a book of tabulations and see how closely sample findings match known characteristics in the population. Plus, there's that mixture of satisfaction, pride, and relief I feel when, in the middle of a presentation, I hear the sweet sound of someone remarking that "your numbers line up almost exactly with our internal data."

On the other hand, there's little that contributes more to the graying of my hair than the flip side of this phenomenon--opening a set of tabulations and seeing a distribution that just can't be right (too many older respondents, or younger respondents, or men, or women, or product users, or non-users . . .), or getting to chart three in a presentation and hearing someone say--"that's strange, our product penetration is lower than five years ago, not higher."

Fortunately, such situations are the exception rather than the rule (meaning I'm not quite completely gray yet). But when they do happen, I try hard to resist panicking and instead look at the issue as rationally and systematically as I possibly can. In fact, my strategy tends to fall into four categories--prevention, investigation, communication, and resolution.

The prevention step (as you'd imagine) takes place before the problem even arises--and involves things like making sure we're using question wording and scale points that are consistent with what's been used in the past, and getting preliminary data runs to spot potential problems before the fieldwork is completed.

Investigation tends to be a no-holds-barred look at everything and anything that could have caused the seeming discrepancy--from checking that codes and weighting are correct, to looking for internal consistency with comparable questions, to calling back respondents to check on whether their interpretation of the question was different from what we intended.

Communication is another key step, assuming steps one and two haven't solved the puzzle. Indeed, we can often get to the bottom of things by discussing the discrepancy with other professionals on our staff, or the client, or outside experts, to pick their brains about what may be going on.

And hopefully, all of this leads to resolution, whether it be confirmation or correction--and another month of keeping the Grecian Formula at bay.