Garbage in, garbage out: the perils of polling

This post isn’t so much about polling as it is about what happens if you use statistics as an excuse to turn your brain off.  In an attempt to gauge the impact of Mitt Romney’s acceptance speech, a company called SurveyUSA polled Floridians who watched the Thursday night coverage and found that the speech (along, I suppose with Eastwood’s and Rubio’s) had changed the votes of 34% of them: Romney picked up 22%, Obama 12%.  The great majority of the vote-changers (26 of 34%) started the evening in the undecided category.

SurveryUSA is a fairly respected neutral polling firm, but this is garbage.  I don’t have a problem with Romney getting the better share of votes—it would be a surprise if that weren’t so—but the idea that 34% of viewers changed their votes is wildly implausible. I’ll try to explain why, but have a look at the results yourself:

The first sign that something is terribly wrong is the overall sample.  They talked to 1100 registered voters, and over 750 said they heard the speeches, almost 70%, including over 60% of vrespondents aged 18-34.  Those would be awesome ratings for the Super Bowl, I think.  Here in the real world, just under 10% of Americans watched the coverage.  Presumably the percentage was higher among registered voters…maybe even as high as 15%.  A sample in which 70% of the respondents watched Romney’s speech is one crazy-ass sample.  My guess is that this is the result of two factors, people who watched the coverage being more likely to answer the survey (which is maybe not a huge problem) and people lying about having heard the speech because they think it’s what they’re supposed to say (which is a bigger deal).

It just gets weirder.  Within the group that claimed to have heard the speech, 26% said they started undecided and ended up supporting either Obama or Romney.  Now it’s inconceivable that every undecided voter had his/her mind made up by the speech; being super conservative, let’s say that 1/3 of undecided voters remained undecided.  That puts the total of undecided voters at the start of the speech at 39%

How plausible is that?  Recent Florida polls have pegged the number of undecideds at about 5%; to be sure, these are “likely voter” samples, and if we expanded to all registered voters, we might get 10 or 15% undecided.  On the other hand, what kind of people watch political conventions?  It sure as hell isn’t apathetic people who probably won’t bother to vote, rather, it’s people who are intensely interested in the election, and, as the article points out, mostly people who already favor the party holding the convention.  This is always true, and all the more so this year when there is no buzz that might draw ordinary people to watch,as there was for Obama and Sarah Palin in 2008.  The dramatically reduced ratings and the fact that a larger share were watching Fox News both indicate an audience of only the hard core.

So there is no way in hell that almost 40% of these viewers came in undecided.  If SUSA had conducted a real poll before the speech, they would have found 5% or at most 10%.  Presumably what happened is that a bunch of people who in reality were already going to vote for Romney (mostly) or for Obama pretended that they had been undecided.  Psychologically, this makes sense: people like to say something interesting, and it seems more interesting to say that a particular event affected them than to say nothing much happened, they pretty much always end up voting for the same party.

The moral of all this is that statistics are not magical.  You can go through the motions and generate numbers, but they won’t mean anything unless you’ve asked the right questions and have a reason to expect frank answers.  A study like this always worries me, whether it’s a poll or not,because so many readers don’t realize they can do a reality check.

PS for kicks and giggles, have a look at the amazingly volatile results for African-American voters: an actual majority of them apparently changed their voter votes after listening to Romney.  As you probably know, in the real world black people continue to be rather reliable supporters of the President, and I have seen state polls that give Romney a percentage of the African-American vote that is too small to measure.

This entry was posted in Uncategorized. Bookmark the permalink.

5 Responses to Garbage in, garbage out: the perils of polling

  1. Mary Evelyn White says:

    Oh my! I never would have figured that out. do you have any idea why they would have created such a poll?

    • Roy says:

      It would have been too difficult and/or expensive to do a proper investigation of the question, so they figured a crappy study tarted up was better than nothing.

  2. Ann Foxen says:

    If women, African-Americans, poor people, older people and young people vote for President Obama, do we really need to worry about the votes of white middle-class males? How likely is it that the presence of Joe Biden on the ticket is going to make any difference?

  3. apolena says:

    This would work really nicely as an example in an Intro to Statistics course. And, of course, after all that you’ve described, there comes the question of what formulas were used to process the data. Sociologists who rely on statistical methods talk often about “massaging the data” without even blushing..

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s