Polling “Mystery”? (Hardly.) Editorial Bias? (You Decide.)

An article in the Week in Review section of last Sunday's New York Times titled (or at least subtitled) "A Polling Mystery" centered on the following point: "Support for the initial invasion of Iraq, as measured by a question The New York Times/CBS News Poll has asked since December 2003, increased modestly compared with two months ago."

Specifically, in mid-July, "42% of those polled said the United States did the right thing, and 54% said the United States should have stayed out of Iraq. The last time the question was asked, in May, 35% said taking military action against Iraq was the right thing and 61% said the United States should have stayed out."* With a margin of error of 3 points, this 7-point change was statistically significant.

The article went on to explain just how "counterintuitive" this finding was, because "none of the other war-related questions showed change. Mr. Bush's approval rating had not changed. Nor had approval of his handling of Iraq. The level of support for Mr. Bush's decision to send more troops to Iraq -- the 'surge' -- was about the same as it had been in past polls. Support for the decision to go to war had risen modestly and nothing else in the poll could explain it."

So the Times and CBS decided to do another poll, and found the very same thing. The article ends with this polling "mystery" unsolved: "What was driving the change still wasn't all that clear, but at least the paper had confidence in the results and was able to report the findings."

I have two big problems with this article.

First, on this polling "mystery," I am surprised that the New York Times/CBS polling people didn't explain this "mystery" to the reporter. They are solid survey research professionals who know what they're doing, and I'm certain that deeper analysis of the data would have shed much light on what's behind this sudden change in public opinion. At the very least, an examination of change among demographic and political affiliation would have helped explain the sudden shift. There was none of this in the article, however; so I can only assume that "a polling mystery" is a sexier story than "an apparent polling mystery solved" would be.

• It makes me curious, though: What do you think was going on? What might cause this particular shift in public opinion, when nothing related had changed at all?
• And there's an obvious lesson here as well for all survey research professionals. Whenever we see a "mysterious" finding in our data, we need to leave no analytic stone unturned in solving that mystery. It's our responsibility. (And as troubling as it can be -- and we've been through it many times -- it's part of the fun of deep data analysis.) One has to be a genuine data detective to solve such mysteries, but that's what we're here for. I'll bet anything the Times/CBS polling people have some answers.

My second big problem with this article is a lot more subtle, and a lot more insidious as well. I was surprised to see the word "modest" used twice in this article to describe the recent, sudden 7-point increase in public support for the initial invasion of Iraq. Surely if the president's job rating were to decline suddenly by 7 percentage points over the same two-month time period, the word "modest" would not be used to describe such a drop. Words like "precipitous," "dramatic," and "major" would be used; and such terms would be entirely appropriate with a change of this magnitude in such a short period of time. I can't figure out why such words weren't used in this case, and "modest" was. Editorial bias is all I could come up with in answer to "why."

• Do you have an alternative answer?

*Actual question wording: "Looking back, do you think the United States did the right thing in taking military action against Iraq, or should the United States have stayed out?"