If you've followed this blog, you may know that we have a monthly movie group and that we conduct research for the movie industry. With the Oscars approaching, I've been trying to get out and see as many movies as possible -- and also hoping that Netflix members begin to return movies faster, so Long Wait becomes Wait, and Wait becomes Shipping Today. (This process can be quite frustrating.)

And earlier this week I came across a fascinating article on the industry : Tad Friend's look inside the marketing of movies (which focuses on Lionsgate's intriguing co-president of theatrical marketing, Tim Palen). First, from the article:

Modern campaigns have three acts: a year or more before the film débuts, you introduce it with ninety-second teaser trailers and viral Internet "leaks" of gossip or early footage, in preparation for the main trailer, which appears four months before the release; five weeks before the film opens, you start saturating with a "flight" of thirty-second TV spots; and, at the end, you remind with fifteen-second spots, newspaper ads, and billboards. Studios typically spend about ten million dollars on the "basics" (cutting trailers and designing posters, conducting market research, flying the film's talent to the junket and the première, and the première itself) and thirty million on the media buy. Between seventy and eighty percent of that is spent on television advertising (enough so that viewers should see the ads an average of fifteen times), eight or nine percent on Internet ads, and the remainder on newspaper and outdoor advertising.

Good stuff. The article was subtitled "Inside a movie marketer's playbook," which initially caught my attention because of a recent movie-industry study we conducted that examined how consumers use various sources to make movie-going decisions. The study led to the creation of a playbook for our clients to leverage the Internet most effectively as an advertising and informational resource for consumers.

I was particularly interested when the article got around to the subject of test screening and how it's used. Again, from Friend's piece:

[T]he percentage who thought the film excellent or very good, the so-called "top two boxes," went from sixty-five at the earlier screening to seventy-four--in other words, from worrisome to respectable. (Studios love to see scores in the eighties.) Yet testing is fraught: it rewards comedy, narrative, and familiar stars or plot elements, and often undervalues the new. Executives' testing stories take divergent paths to the same punch line. Either they decided not to tamper with a "Pulp Fiction," despite testing results invariably described as "the lowest scores in the studio's history," or they were confounded when an "Akeelah and the Bee" faltered commercially despite "the highest scores in the studio's history." In both scenarios, the numbers lied. "Testing is a sham," one marketing consultant says. "All you've learned is what people thought of a movie they didn't have to pay for. It does not mean they're going to go pay for it."

First, this glosses over the fact that movies can be, and are, tested both qualitatively and quantitatively. While the quantitative side usually consists of a big audience and a short questionnaire, the qualitative research is done by bringing viewers into smaller focus groups. These methodologies should be used together to find out what people think of the film and what could make it better.

But Friend is right -- to a degree: in both scenarios, the numbers from the quantitative research lied. But the marketing consultant's conclusion that testing is "a sham" doesn't follow.

Numbers lie all the time if they're not interpreted correctly. That's where the art of research comes in. For example, in our advertising testing in particular, there have been many instances where we've tested a half-dozen different creatives and asked respondents to give scores to each one. And sure, we look at the data, and one or two creatives usually stand out as scoring the highest.

But sometimes lower-rated pieces that represent a somewhat higher risk also represent the possibility of much higher reward. It's only upon listening to the discussion from a point of understanding what the client is trying to communicate that we can assess what's worth considering. And that's what Friend leaves out.

When entering the United States as a foreigner, people are assigned by Immigration to one of three categories:

1. The fortunate ones with a permanent residence card, a.k.a. a green card. They can pass Immigration through the U.S. citizen line at the airport.

2. Those who have a visa. Immigration requires you to fill out an I-94, a small form with information on where you live and where you intend to stay.

3. Visitors from countries that have an agreement with the U.S. about a visa waiver. For example, if a Briton intends to stay for a couple of weeks on vacation, he doesn't need a visa. These visitors hop on a plane, and, while on the plane, they fill out the I-94W. The front is similar to the I-94, asking for residence and where you intend to stay. The back, on the other hand, contains a series of straightforward, yes-or-no questions:

A. Do you have a communicable disease; physical or mental disorder; or are you a drug abuser or addict?

B. Have you ever been arrested or convicted for an offense or crime involving moral turpitude or a violation related to a controlled substance; or been arrested or convicted for two or more offenses for which the aggregate sentence to confinement was five years or more; or been a controlled substance trafficker; or are you seeking entry to engage in criminal or immoral activities?

C. Have you ever been or are you now involved in espionage or sabotage; or in terrorist activities; or genocide; or between 1933 and 1945 were you involved, in any way, in persecutions associated with Nazi Germany or its allies?

D. Are you seeking to work in the U.S.; or have you ever been excluded and deported; or been previously removed from the United States; or procured or attempted to procure a visa or entry into the U.S. by fraud or misrepresentation?

E. Have you ever detained, retained, or withheld custody of a child from a U.S. citizen granted custody of the child?

F. Have you ever been denied a U.S. visa or entry into the U.S. or had a U.S. visa canceled? If yes, when? Where?

G. Have you ever asserted immunity from prosecution?

IMPORTANT: If you entered "Yes" to any of the above, please contact the American Embassy BEFORE you travel to the U.S. since you may be refused admission into the United States.

I can assure you this form provides enough material for evening discussions among expats. The questions about mental disorders and terrorist activities are good ice-breakers. Also, hearing on the plane that you should have contacted the American Embassy before hopping on said plane has entertained thousands of tourists. I wonder about the validity of the responses the U.S. Customs and Border Protection get from these questions. If you have some mental disorder, is now the best time to admit that? If you worked for MI6 for 10 years, do you think the immigration officer would like to compare notes with you?

In market research, our surveys are slightly more respectful of the respondent's privacy, but then again, we don't have to secure a border. Even so, I often wonder about the value about one specific type of question:

Do you or does anyone in your household work in the following industry: [fill in client's industry]?

The theoretical purpose seems obvious:

1. If we test pricing for a new product, the client doesn't want the product or price options revealed to competitors.

2. We like to prevent competitors from seeing our methods; especially since copyrighting questions and methods is essentially impossible, it is very hard for us to protect our intellectual property.

3. Last, we might want to avoid respondents who have a professional affinity to those products and measures.

Regarding the last goal, in my opinion, people who work in this industry are part of the market. They might make their decisions differently than John Doe, but they are still part of the market -- and if the sampling method is appropriate, then they would be equally represented in the sample as they are in the population. Why would we want to exclude them? Sooner or later the product will be available in the stores anyways.

My greater quibble has to do with goals 1 and 2: people who want to understand what their competitors are doing would certainly not stumble over such a question. You can call me a pessimist (I prefer to call myself a realist) when it comes to my evaluation of the human nature, but I have a hard time believing that the people we want to keep out will sheepishly admit their industry. On the contrary -- I think we offer those people our research area on a silver platter with a big sign saying: WATCH OUT! If you work in any of the following four industries, listen now. The following survey is not intended to be seen by you, but if you click "no" four times, we will show you the hottest products and the deepest secrets!

Theoretically, we could make it a bit harder by having a long list of industries where only a few result in a termination of the respondent. But I am confident most people we want to keep out would know which industries are pretty safe bets to not be terminated ("education," "government," and "other" seem good choices). We might lose some respondents because they don't want to go through a list of 35 industries, but otherwise it won't do much good.

So what can we do?

Well, for starters, we might want to consider whether the survey really contains information these people wouldn't already have (if they already have it or could easily find it out, we're trying to solve a nonexistent problem). Maybe all we're doing is telling them things they know already while occupying 10 minutes of their precious time. Sounds like a fair deal to me: you lie to us and we take your time.

If there are really previously undisclosed pieces of information in the survey, we might want to choose an open-end approach where we do not disclose beforehand what industries we want to exclude (respondents could still say they work in education or government).

By asking respondents what industry they work in, we actually suggest there is some so-far-undisclosed information in the survey. If that is the case, we would attract less attention by not having that screening question in the first place. Removing it would also save a few seconds of all respondents' time and reduce respondent fatigue.

We in the United States are often accused by those who live beyond our borders of having cultural blinders on -- of being ignorant (perhaps even dismissive) of other people and cultures. As stereotypes go, this is, sadly, a pretty good fit.

According to Census Bureau statistics, over 82% of the U.S. population speaks only English, while just 8% can hold a passable conversation in a language other than English. According to the Philadelphia Tribune, only 8% of U.S. college students take courses in a foreign language, while two-thirds of all U.S. high school students graduate without ever having studied a foreign language.

Contrast this with the fact that American tourists can typically find their way around Europe without knowing a word of any language other than English, simply by imposing on the English-language abilities of their European hosts. Increasingly, the same can be said for Asia. I've traveled extensively within both Europe and Asia and have been amazed that even in the smallest, most remote hamlets, I've always encountered someone who can converse in my native tongue.

Then there's the sad fact that 88% of Americans cannot find the country of Afghanistan -- so much a focus of our collective thoughts and energies for most of this decade -- on a map, while 66% cannot locate either Saudi Arabia or Iraq. Sadder still is that nearly half cannot find the continent of Europe (the "motherland" for the majority of us).

So, what does this all have to do with market research?

With the globalization of economy, American businesses and their employees are increasingly doing business with people from other cultures. And all too often, these businesses, so singularly focused on their "captive" U.S. market, have little more than the most basic understanding of how to go about this. Typically, they hire locally within the markets into which they are looking to expand, hoping this will make it all that much easier. All too often, though, a wall appears between their U.S. and local employees -- a wall whose foundation lies on the U.S. side's lack of understanding of their local counterparts' language, culture, customs, and business practices. This wall impedes all subsequent efforts at becoming more familiar with this market and the people who comprise it. All too often, the end result is failure and retrenchment.

That's where market research, done locally by researchers who are familiar with conducting international research, often becomes critically important to the success of American businesses looking to expand overseas. We've conducted research on six of the seven continents (we're still waiting for that client who's hell bent on breaking into the Antarctic market), in dozens of countries, including China, Japan, Korea, Taiwan, Spain, England, France, Mexico, Brazil, and Argentina, to name but a few. And we're proud to have on staff a number of researchers who have extensive international research experience and, importantly, are familiar with cultural nuances that come into play when conducting research outside the United States.

Why is this important? Well, for example, in Indonesia, time can be somewhat of a relative thing. As such, the standard recruit of 12 for 10 to show becomes more like recruit 25 for 8 to show when conducting focus groups in Jakarta.

Focus groups with physicians in New Delhi, India, are seldom conducted in the native Hindi, but instead in the more academically and socially acceptable English.

And in China, you should never give a clock as a thank-you gift (e.g., for an interview), as clocks are symbolic of death in many Chinese cultures.

While some other U.S.- and Europe-based research firms have built global networks of wholly owned, "local" research organizations, we've instead taken the approach of partnering directly with locals (on occasion, the local branches of these large, international firms). We believe this gives us greater flexibility to choose the local firm we feel has the greatest expertise in the field we're researching.

The bottom line is that the American model for conducting business and market research often does not fit beyond our borders. It is critically important that American companies take the time to get to know the people they are hoping to make customers. Market research, conducted locally, with a strong sense of the local culture and sensibilities, can often make the difference between failure and success.

Previously, on Praxis . . . Come on -- didn't our staff MAs understand that Sachs was just reporting on tendencies among this segment? That "more likely than" is understood in all of this? That they were using marketing shorthand to make the presentation more powerful? That the people in the clips don't necessarily represent all MAs?

Well, we've already gotten some thought-provoking answers to these questions from Sarah Floyd, Nikki Lavoie, and Amy Barry. Maybe the whole "tendency" issue is not the point. If they saw this as (to put it kindly) pigeonholing their entire generation, and if they took some offense to being pigeonholed, could you blame them? (By the way, ask any member of selected racial or ethnic groups in this country how they feel about attempts to describe them in broad, general terms. Odds are the term they'd use isn't as kind as "pigeonhole.")

But to think of this as primarily an emotional, visceral reaction to pigeonholing is to miss an important point. The fact that our MAs felt they had nothing in common with the people in the clips may say something about the relative value, or lack of value, of research that ends up offering broad generalizations about a diverse generational group. Do these initiatives help at all? Or, by missing the nuances and the diversity that are necessarily part of any consumer segment, are they limited to simply stating the obvious?

That's not an easy question to answer, but this might help: Several weeks ago, The New York Times Sunday Magazine ran a fascinating feature story about Netflix. It turns out that for several years, Netflix has been looking into ways to improve the accuracy of its movie recommendation feature -- "if you liked this movie, you'll also like this one." As part of this effort, the company established a contest, open to any data geek who might be interested. The first person, or team of people, to create a statistical model that improves the accuracy of the current recommendation system by at least 10% wins a million dollars.

All of the teams are using Netflix customers' past ratings of movies to predict how those same customers would rate other movies. You might think that demographic characteristics would play a role -- gender, age, generational group, etc. But here's the surprising part:

• "Interestingly, the Netflix Prize competitors do not know anything about the demographics of the customers whose taste they're trying to predict. The teams sometimes argue on the discussion board about whether their predictions would be better if they knew that customer No. 465 is, for example, a 23-year-old woman in Arizona. Yet most of the leading teams say that personal information is not very useful, because it's too crude. As one team pointed out to me, the fact that I'm a 40-year-old West Village resident is not very predictive. There's little reason to think the other 40-year-old men on my block enjoy the same movies as I do. . . . Even though Net¬flix has a good deal of demographic information about its users, the company does not currently use it much to generate movie recommendations; merely knowing who people are, paradoxically, isn't very predictive of their movie tastes." The New York Times Sunday Magazine, November 23, 2008.

In other words, it's a no-brainer that my 17-year-old stepson is more likely to enjoy so-called "torture porn" movies than my 82-year-old mother. But whether he likes Saw V better than Hostel II has very little, if anything, to do with the fact that he's a late Millennial -- a MilleniJuvie, if you will.

So, if you ask me (and I would tell you even if you didn't ask, so don't go anywhere yet), I'd say our MillenniAdults' reactions do provide important lessons about the limitations of studies that attempt to describe the common characteristics of a generational segment: that these studies can't capture the diversity and nuance inherent in a generation, that sweeping generalizations do little more than state the obvious, and that it's misleading at best -- and offensive at worst -- to confuse tendencies with absolutes. And most important, their reactions remind us that when you notice any of this in a generational study, the results need to be taken with a big grain of salt.

For myself, I continue to believe in the value of studies that profile a consumer segment -- as long as everyone involved clearly understands what they can and cannot do. Where they work best (concrete, behavioral, and contextual) and where they can fall short (broadly attitudinal and absolute).

But my feeling about all of this shouldn't surprise anyone. I'm a Baby Boomer, after all. I just went online, and it says I'm "adaptive" and have a "positive attitude."

Wait, isn't that what someone said about Millennials?

After reading Peter's first two entries on MillenniAdults (MAs -- and my generation), I did some thinking. I remembered that staff meeting clearly, and I remembered the Sachs video clips making me feel, well, a little annoyed.

But I couldn't remember just what had irked me about the clips, so I went to the site to check them out again.

Clip 1: A twenty-something woman talks about her ambitions and experiences, which have included traveling, working for AmeriCorps and Habitat for Humanity, and applying to Ph.D. programs.

Hmmm. Nothing offensive there. I'd be proud to have a girl like that representing my generation.

Clip 2: Another girl, probably about my age, talking about her interest in creating a multifaceted career for herself, incorporating numerous goals and personal interests, such as being a bread baker, jeweler, farmer, and artist, and studying alternative medicine.

Err, well, that sounds good -- I also like the idea of being able to express myself creatively while earning a living, and I love that this girl is leaving herself open to all her passions and the possibilities they present. I began to get suspicious; had this clip always been here?

Clip 3: A collage video with words and drawings of important concepts and goals for MillenniAdults -- things like friends, travel, moments, family, and a work/life balance.

Yep, those things sounded good to me. I swore they'd changed these clips to make the MAs look better.

At this point, I started to wonder why I had felt so pigeonholed when we watched these clips as a staff. Then I kept watching.

Clip 4: A young woman talking about the "M" word: "Whenever I have no money, my parents will always help me. . . . They don't give me money on a regular basis, or pay for anything; well, no, that's not true, they pay for my cellphone. . . . At any age . . . they would still be there to be like, 'We'll give you money to get on your feet.'"

I think this clip was one of the first we watched together as a staff, and it left a bad taste in my mouth. With that clip setting the stage, we next watched one where a young man talks about traveling to Europe, and how "now is the time" to do it, presumably before he has serious family or career obligations.

Maybe it was just the order in which we watched the clips that set the tone in my mind, but I didn't like what these two clips implied: that my generation is the offspring of well-off parents who have given us the resources to do whatever we want, who have worked hard so we don't have to.

Maybe we'll get in a pinch because our rent is too high or we charged too much on the credit card, but mom and dad will help us out and get us back on our feet, so we're free to move on to our next life adventure. We're able to roam across Europe for a few weeks because we've got that safety net, and presumably the money, to take that kind of trip. Don't we?

With these implications coloring my outlook on Sachs's clips, I started to get defensive. My family, solidly middle-class and loving as all hell, is great -- but they don't write me checks when I get in a bind. And sure, I guess I could have put off getting a "real job" for a few years after college to enjoy my youth and travel and see the world, but upon graduating from college, I had pretty close to a zero balance in my bank account. Not to mention the student loan payments that were soon due -- and my $160 a month was nothing compared to some of my friends, who were looking at sums of up to $800 a month (to pay down $70,000 in school debt).

So, there I was, sitting in a room with my colleagues, a handful of whom are part of my parents' generation -- the Baby Boomers -- and who are also my professional superiors. And on the screen, there's a girl my age saying, "I know a Louis Vuitton bag is not going to make me happy, but I still want it."

Fine. But the last thing I want is Peter, or anyone, to think my fellow MAs and I are materialistic, spoiled, and willing to sacrifice hard work for "balance" in our personal lives.

It's not that I don't share many of the sentiments and goals voiced by the MAs in the Sachs clips. And it's not that I don't have friends who are very much like the people in the clips. I do. But during work hours, with my colleagues sitting next to me, I don't want to be seen as one of them.

It is, to use Peter's words, an unsettling conundrum. If a few clips provoke such a negative reaction in me -- and my fellow MAs in the office -- are they still valid? To what extent can they be safely applied?

Previously, on Praxis . . . In the end, what began as a helpful FYI initiative on Jeff's part turned into a somewhat unsettling conundrum, one I've been thinking about quite a bit ever since.

Before getting back into this, I need to state for the record that on the whole, I'm personally a proponent of the type of research we've been discussing -- research that profiles a consumer segment of interest. Marketers realized long ago that the ability to identify specific segments of the consumer population -- where members share common attitudes, aspirations, and behaviors -- opens up a gold mine of opportunities to create products, advertisements, etc., that "speak" to those segments. I've worked on many such studies myself and stand by the approach. (In fact, we'll soon be releasing a report that deconstructs the music consumption habits of consumers across the generational spectrum, and identifies the unique challenges the music industry faces from each age segment.)

Now, that was a very broad, generalized statement about the value of this type of research (broad, generalized -- please hold that thought). Let's break things down a bit and focus on three concepts: concreteness, interpretation, and tendencies. Any research that profiles a consumer segment is on safest ground when it:

• Focuses on phenomena that can be measured concretely;
• Focuses on phenomena that can be interpreted unambiguously; and
• Doesn't lose sight of the fact that it identifies tendencies rather than absolutes.

I'm not saying these studies are worthwhile only if they meet these criteria. But there are implications once you move beyond this.

• First, let's say you read somewhere that Millennials are more likely than Boomers to own an iPod. No controversy there at all -- a concrete measure of behavior that's about as unambiguous as you can get (you either own an iPod or you don't). And we're clearly talking tendencies -- group A is more likely to own an iPod than group B. (Explaining the degree of difference would be even more helpful -- 1.1 times more likely, 10 times more likely -- but let's not get too greedy here.)

• Now let's take things to an attitudinal level. You read that Millennials are more likely than Boomers to agree with the statement: work/life balance is important to me. Not as easy to interpret as owning an iPod, but the measure is concrete (agreement with a statement) and the focus is still on tendencies. Check.

• OK, what about this: Millennials are more likely than Boomers to value work/life balance. That doesn't sound much different, does it? But now we're talking about group A's greater tendency to feel a certain way rather than their greater tendency to agree with a statement. What does "work/life value" mean, after all? Does it mean that Millennials are less likely to work on a weekend? To be more likely to take a day off to drive grandma to the doctor? Maybe, but things are beginning to get murky. (Although at least we're still emphasizing the tendencies bit.)

• And now, a little devil appears on your shoulder. Big execs just want to get to the bottom line -- they don't want to keep reading "more likely than." It sounds weak. It's not punchy. And everyone understands we're talking about tendencies anyway. Voilà: As a generation, Millennials value work/life balance.

Welcome to the dark side.

Now take this:

• "Mr. Tapscott identifies eight norms that define Net Geners. . . . Net Geners value freedom and choice in everything they do. They love to customize and personalize. They scrutinize everything. . . . They expect everything to happen fast." The Economist, November 13, 2008.

And this:

• "Over the past few decades the prevailing disposition among college students -- today labeled Generation Y or Millennials -- has slid into full-blown narcissism. . . . The 'all about me' shift . . . points, says the study's author, to a generation's lack of empathy, its inability to form relationships -- and worse." Christian Science Monitor, March 2, 2007.

And:

• The video wall.

And:

• The people in the same generation who look at the video wall and say, "That's not me at all."

Come on -- didn't our staff MAs understand that Sachs was just reporting on tendencies among this segment? That "more likely than" is understood in all of this? That they were using marketing shorthand to make the presentation more powerful? That the people in the clips don't necessarily represent all MAs?

Tune in to my third installment for some startling answers!

It started out innocently enough.

Last month, Jeff Radcliffe (our Director of Client Services) shared a presentation with the rest of our staff, a presentation he had seen while attending a marketing research conference in October. The presentation, developed by the research firm Sachs Insights, focused on a consumer segment Sachs has labeled MillenniAdults -- 22- to 28-year-olds, the "first wave of Millennials . . . coming into their own as adults."

The purpose of the Sachs project: to paint a picture of that generational segment -- to demonstrate what makes MillenniAdults tick, what they value. And, by implication, what the marketing world needs to know about these consumers to most effectively reach them.

The presentation was, among other things, technologically impressive. It consisted of 24 short video clips of MAs discussing their lives, desires, hopes, and frustrations. The 24 clips were arranged as thumbnails on a "video wall" -- click on a thumbnail, and the video comes to life. It was a compelling, eye-appealing approach to presenting research results.

After playing a healthy sampling of the video clips, Jeff opened the meeting for reactions and discussion. Several staff members commented on the coolness of the interface. But then, in what proved to be a portentous comment (for me at least), Sarah Floyd -- a MillenniAdult herself -- said (I'm paraphrasing here):

"I just don't feel like I can relate to any of those people. They don't sound anything like me."

With that floodgate opened, many of the other MAs in the room chimed in with similar comments -- essentially, "I don't see myself in any of these people." What was especially interesting was the seemingly raw emotion underlying these comments. Our MillenniAdults seemed insulted, maybe even a bit angry, at what they had seen.

So what happened here? It was clear that the Sachs people had put a lot of thought and effort into the project and its ultra-high-production-value presentation. With all of that intellectual and technological clout, how was it possible that members of the very generational segment profiled in the study could react so negatively? And with an emotional edge? Were Sachs's points about MAs simply wrong? Did Sachs miss an entire subsegment of this population? Were our staff members reacting defensively? Are Taylor MillenniAdults special in some way -- well, we like to think so, but I mean out of the ordinary, not representative of the typical MA?

In the end, what began as a helpful FYI initiative on Jeff's part turned into a somewhat unsettling conundrum, one I've been thinking about quite a bit ever since. Tune in to my next entry, where I'll offer some thoughts on what may be going on.

Last week I was reviewing some screening questions for a sports-industry survey that is now in the field. Several of us were debating one question: It was designed to assign a level of fandom for a specific sport, and we were providing four options: In general, do you consider yourself . . .

• A big fan of [sport];
• A casual fan;
• Only slightly interested;
• Not interested at all?

This is a standard question, but it was especially important here because of the need to fill several buckets of complicated segments for this study. So I asked myself, where would I fall? And where would my colleagues fall? We talked it through and settled on the categories, and I hadn't given it much thought since.

Until today. Last night, with the Red Sox trailing Tampa Bay 7-0 in the seventh inning, I went to sleep. It was about 11:00 and the game was pretty much over. Sure, it would be the last Red Sox game I'd watch until April, but I didn't even think twice about it. Good night.

Of course, Boston came back to win 8-7, forcing Game Six tomorrow night in Tampa.

Now I'm sure many big sports fans, and big baseball fans (I consider myself a member of both categories), did the same. But for me, this is the third time in six years that I've done something similar during a crucial Red Sox postseason game:

• I went to sleep at 5-5 in Game Seven of the 2003 ALCS (the Aaron Boone game), after Grady Little had fallen asleep in the dugout and coughed up a 5-2 lead in the seventh inning without using his bullpen.
• I went out with friends on the Sunday night of Game Four of the 2004 ALCS, with the Sox trailing the Yankees three games to none and facing elimination, only catching the end of the game by luck (Boston won in extra innings).
• And then there was last night.

If asked, I would never say I'm only a casual sports fan, a casual baseball fan, or a casual Red Sox fan. But here's the question: While I might feel like a big fan, have I reached a point where my behavior conflicts with that self-analysis?

And if so, how reliable is my self-described fandom?

Your thoughts?

In Friday's edition of The Wall Street Journal, Carl Bialak makes two points about interviewing cellphone-only consumers by phone:

1. Gallup now supplements its landline phone samples with cellphone-only samples, but it does not reimburse its respondents.

2. Including cellphone-only consumers in political surveys might not be worth it.

First, we believe it is important to reimburse respondents in exchange for their time and insight. Since most people are not on an unlimited calling plan, they might have to pay for those minutes out of their own pocket. Paying a flat incentive is the most economical way to ensure the respondent has no extra cost.

We believe the respondent's cooperation is something we cannot take for granted, so we comply with these basic rules:

1. The respondent's privacy is crucial to us.

2. A respondent should not incur costs by cooperating with us beyond the time he/she spends.

3. Any respondent has the option of opting out of a study at any point in the survey.

While not reimbursing respondents for their used minutes might save money in the short term, we sincerely believe this practice will hurt market research in the long term.

As far as the second point is concerned: We have conducted a number of studies with dual samples, cellphones, and landline phones, and frequently we do find significant differences among the cellphone-only sample.

Bialak's article tackles the issue of political polling; since voter turnout is low among the segment with a high cellphone-only percentage, not including this segment might not make a big difference to the poll. But when you look at the market potential for an HDTV or at consumer behavior of Millenials, it does make a difference if you exclude 30% of that market from your research.

The bottom line is: Each study is different. Sometimes you can save the extra money by skipping the cellphone-only sample, and sometimes you need to spend it.

It's not unusual for the New Hampshire Primary to rejuvenate the campaign of a candidate who had been counted out. Last night's 2008 New Hampshire Primary, however, rejuvenated two campaigns, giving Arizona Senator John McCain (R) a much-needed victory and New York Senator Hillary Clinton (D) a shocking one.

This morning much of the post-Primary coverage was focused on polls -- specifically, how they got it so wrong on the Democratic side, while calling it right for the Republicans. As Josh Marshall noted on his website Talking Points Memo, "It's hard for me to remember an election where the trend of polling and the final poll results so failed to predict the actual vote. Certainly, there's no example I can remember of it happening in such a high-profile contest."

According to Real Clear Politics, the final New Hampshire poll released by all seven major polling outfits showed Illinois Senator Barack Obama with a healthy advantage.

• Suffolk/WHDH: Obama +5
• American Research Group: Obama +9
• Reuters/C-Span/Zogby: Obama +13
• Rasmussen: Obama +7
• CNN/WMUR/UNH: Obama +9
• Marist: Obama +8
• CBS News: Obama +7

The findings were consistent, and not dependent on the size of the sample. For example, the Rasmussen survey of 1,774 likely voters and the CBS News survey of 323 likely voters both had Obama up seven points.

Clinton won by two points. So what happened?

Our president here at The Taylor Group, Scott Taylor, has deep roots in election polling, having worked for Lou Harris, one of the founders of campaign and election polling in the U.S., and having done state-level election polling for Rutgers University's Eagleton Poll and Clark University's Public Affairs Research Center. Earlier today I talked with Scott about last night's results.

Q: Is political polling becoming more difficult?

A: Surveys of all types are becoming more difficult. There's so much of it out there, more than there's ever been. And particularly this year here in New Hampshire, with the race as wide open and hotly contested on both sides, there has been just an onslaught of polls. On top of this, when you combine all the calls from the candidates' organizations, people begin to turn a deaf ear to it all. Refusal rates for surveys have climbed steadily over the years. To the extent that the blizzard of calls in this campaign further turned people off, yes, polling became more difficult. New Hampshirites are notorious for the value they place on their privacy, and Caller ID makes it easy to avoid the pollsters. When you have a huge refusal rate, you need to worry about the extent to which the non-responders to your poll are significantly different from the responders in their attitudes and their behavior.

But all this aside, it has always been the case that election polling in primary contests is dramatically more problematic than in general elections. Turnout is harder to forecast; there are two elections (Rep/Dem), not one; the vote is divided across more candidates; and to top it all off, voters make up their minds much more slowly and closer to Election Day. This year in New Hampshire was a classic case of all of these factors. For pollsters, I feel their pain (if I may use that line).

Q: What specific challenges exist in political polling that don't exist in other types of survey research?

A: Well, they're similar in most ways. The one key difference is that, for political pollsters, there is a truth that happens at the very end. There is an election, and votes get distributed. So pollsters have this endgame that tells them if they were right or wrong. In the kind of work we do, there is never a one-day endgame, or an outcome that serves as a measuring stick for the accuracy of our projections. For example, when we study the market opportunity for a new product or new service, the purchasing of that product or service takes place over an extended period of time, and it takes place as a result of an expenditure of funds on a marketing campaign with a certain amount of resources that are used to a certain extent of effectiveness. In political polling there is a decision, Election Day, and at the end of that day when the votes are counted, you as a pollster learn exactly how close or far away you were. That's incredibly exciting. It's also incredibly anxiety-producing. You live and die on the basis of your projection; and when you're wrong in a given election, you spend a great deal of time trying to figure out why.

Q: New Hampshire saw record turnout yesterday. What effect does huge turnout have on polling a race?

A: The accuracy of a pollster's work ultimately depends on turnout. And turnout varies election-by-election. Typically in higher-turnout elections, the advantage goes to the more liberal candidate. But all of these pollsters came up with their final number based on some determination of what the turnout would be.

Q: How has the rise of cellphone-only households affected the industry?

A: We know from our own work that upwards of 15% of households are cellphone-only and they cannot be reached by traditional phone surveys. So you're losing 15% right off the top in a phone survey (and though web surveys are being used more and more, they have problems of their own). Cellphone-only people are much younger and more mobile than those reachable via landline. The question is: Are they different from their demographic counterparts who can be reached via landline? In our own work, we are seeing more and more instances where they are different in both attitudes and behavior.

Q: What do you think happened yesterday?

A: I read one explanation this morning that I think was wrong, and that's that Clinton had an advantageous position on the ballot. By the luck of the draw, Clinton's name was near the top and Obama's name was near the bottom. This commentator was making the case that Clinton's ballot placement advantage made a three-percentage-point difference in the outcome. I don't believe it. New Hampshire voters are known to be attentive and savvy voters, and this campaign was so intense that voter attention was even greater than usual. Ballot position as an explanation for an advantage assumes a more apathetic voter population than New Hampshire has.

I think there are several things that happened. First, historically there has always been a built-in unpredictability in the New Hampshire electorate. New Hampshirites are proud of being unpredictable, and it leads them to consciously behave unpredictably. In addition, New Hampshirites have some history of looking at the Iowa results, and voting differently.

Also, typically pollsters push undecided voters by asking if they at least "lean" toward one candidate, and they tend to group the leaners with the choosers in their projected results at the end of a campaign. This year in NH we saw a particularly large number of undecided voters, and I wonder if this year's leaners weren't perhaps still truly undecided up to the end. I wonder if what happened was that a lot of the leaners who said they were going for Obama ended up going to Clinton at the last minute. Maybe it was a function of her Monday tears here in Portsmouth, maybe as a function of the ornery, contrarian New Hampshire voter saying, "I'm not going to let the pollsters and pundits tell me how to vote," maybe it was a function of white voters getting behind the curtain and not voting for the black candidate. It was probably a combination of all of those things.

In fact, I was undecided right up to the moment when the kindly, blue-haired poll worker at Rye Elementary School asked me which ballot I wanted. I had come in to the polling place with two strategies, one for the Republicans and one for the Democrats, and I chose at the last minute on the basis of what my slight strategic preference was at that time. But going in I wasn't 100% sure what I was going to do. My vote was strategic, not committed. I'm a New Hampshire native, and I don't think I'm alone when it comes to "strategic" as opposed to "committed" voting in a presidential primary election.

A lot of people saw the polling data that was suggesting a strong move toward Obama and said to themselves -- even those who were perhaps leaning toward Obama -- "We don't want this to be over." And if Obama wins, everyone was saying it's all but over. Pundits even started saying if Hillary loses New Hampshire, then she ought to drop out. And so the voters said, "I don't think we've seen enough here, and I don't want to hand this to Obama so early." So it's entirely possible the polls were not wrong, but voters changed their minds at the last minute because they wanted to see this contest continue.

I think a lot of people voted strategically, voting for a continuation of this contest.

Q: Personally, what do you think of the early front-loading of the primary schedule this year?

A: I hate it. I think it's awful. I have no problems with Iowa and New Hampshire coming earlier than ever, mind you (and I don't say only because I'm a New Hampshirite, by the way). My problem is with the front-loading of the rest.

As a big fan of the campaign process, I would love to see a primary a week. It would give voters an opportunity to make these candidates work -- to see how they react under pressure over time. The front-loading creates an opportunity for the process to be over prematurely. We want to see these candidates over an extended period of time. Now, money has the potential to matter more than ever, even though so far with the Republicans that hasn't necessarily been the case. On balance, though, money matters more under this system. And more than ever, it makes it difficult for a dark horse to succeed.

Last week a cartoon in The New Yorker caught my eye: a middle-aged man in a conference room, pointing to an easel in front of confused-looking coworkers. On the easel is a drawing of the presenter, sitting on a Pegasus-like winged horse, flying over a rainbow.

The caption reads: "Impressive, Meyers, but let's stick to your quantitative projections."

Suddenly I felt like I was back in our conference room, in the middle of a staff meeting we had about a month ago. We had gathered to discuss a journal article on focus group research that was passed along by one of our long-time clients. Everyone had already read the article, and we convened to go over some of its more pertinent points.

While we have qualitative and quantitative specialists here at The Taylor Group, most of our staff members are experienced in both qualitative and quantitative work. But I've noticed that most people have their favorites. And these feelings were certainly on display during our discussion.

Now you should know that when our staff gets together to discuss an issue, people really get into it. There's no holding back. Recently, during an internal seminar on writing, a 10-minute debate on contractions broke out, with both sides passionately making their cases, trying to persuade the fence-sitters to join their group.

This meeting, like the infamous contraction discussion, started out calmly enough. We were pretty much in agreement that we enjoyed the article and began to discuss some of its more significant aspects.

But after a while -- and I have to admit I'm not exactly sure how it happened -- the tenor changed. Sides were taken. Pro-qual and pro-quant teams formed and suddenly we were debating the merits of focus group research. It was as if the man on the winged horse had swooped into the room and changed the dynamic of the discussion.

The arguments were made: The sample size is too small, and participant selection is not sufficiently random, to be representative of the whole population. There's always a chance that some group members echo the sentiments of others rather than disagree, or make comments for the sake of effect. Moderators in focus groups are too important and end up serving as participants, which could affect the findings.

Political observers emphasize "controlling the message." And this conversation was suddenly being controlled by the pro-quant side. After all, we'd gone from discussing focus group techniques to talking about whether focus groups were legitimate research. Advantage, pro-quanters.

But in response, most of our staff settled into the pro-qual camp -- a coalition of partisan pro-qualers and moderators who found the anti-focus-group position of the pro-quanters too militant. They kept hammering on one main point -- that focus groups, when conducted properly, do exactly what they are designed to do: provide the emotional depth and breadth of customer opinion within which statistical data is lodged. And that argument resonated with most of our staff. The pendulum had swung toward the pro-qualers.

Of course, that didn't stop the vocal minority from trying to persuade their coworkers. We have a few newer colleagues who have joined us within the last six months, people who haven't yet formed methodological preferences, and they represented the undecided voters that each camp tried to drag into their own.

On and on it went. About an hour later, with perhaps a few feathers ruffled, we had completed our discussion.

Much like in politics, few minds were probably changed. But just as a thorough airing of the issues is always good for democracy, this discussion was good for us.

It allowed the quantitative specialists to raise some legitimate issues they have with focus group research. And it forced the qualitative researchers to defend what they do, how they do it, and what they believe in.