Politico's Horrible, Terrible Poll

So Politico came out with a “Battleground” poll this morning that shows Romney ahead of Obama 48-47. I can deal with polls like this, but I’m often interested in the internals of such polls, especially when the previous poll done by the same group showed pretty wildly different results.

There were a few wacky things in the polls, but they mainly dealt with crosstabs and not with the topline – for example, the fact that they asked people if they were very/somewhat conservative or liberal with no option to say you were moderate, which is a pretty crappy way to ask that question (about 6% said moderate anyway, and it resulted in the crosstabs saying that 57% of the nation was conservative – a number I’m sure the GOP will run with but is completely wrong).

However, there was one thing that I saw that WOULD make a difference on the top line. Buried deep down on page 113 of their PDF, I saw these results:

• California: Romney 51%, Obama 43%
• Florida: Romney 56%, Obama 43%
• Texas: Romney 51%, Obama 47%
• New York: Obama 50%, Romney 45%
• Rest of Country: Obama 48%, Romney 46%

Now, Obviously some of these results are ridiculous. Romney is not ahead in California by any stretch of the imagination, much less by 8%. Romney is not ahead in Florida by 13%, and Obama is not only ahead by 5% in New York (and to be fair to Romney, he’s not up by only 4% in Texas). Given the small sample sizes – 99 for California, 66 for Florida, 68 for Texas, and 70 for New York, most of those would probably be within the margin of error if they were taken on their own.

But not California. The margin of error for a poll of 99 people is probably about 10%. Which means, at worst, to be within the margin of error the result could go out as far as Obama 53 Romney 41, which looks more reasonable. But it’s still off of the RCP average of the state by about 12%. Indeed, there were a couple of polls way back in November that showed that the state might be within 12% but nothing remotely close to that recently. So even if we took California by itself with a 10% margin of error, it’s still a horrible poll of the state.

But then I thought – California’s sample is 99 – that’s 9.9% of the total sample of 1000 people. And the result from California is 31.5% off from the RCP average for the state. That HAS to affect the top line. Add to that that Florida is 6.6% of the sample, with a Romney advantage of 13% (if one takes the RCP average of a tie), and New York, which is 7% of the sample, favors Romney by 20% vs. the RCP average. The one state that helps Romney, Texas, makes up 6.8% of the sample, but only tilts toward Obama by 3% vs. the only poll done in the state since January. That’s actually pretty damn good given that the margin of error would be something around 12% for a sample of that size.

So I thought – what would happen if I “corrected” for these numbers and made them inline with the RCP average. There are some problems with this, of course. One doesn’t know how much off the “rest of the nation” sample might be from reality, and there is really no way to tell because there is no “national poll minus California, New York, Texas, and Florida” floating out there to compare against. But I thought it was a worthy exercise anyway, if for no other reason than to see how much it affected the final result.

And affect it, it did.

First off, after figuring out the actual number of people who responded which way in each breakdown, I found out that while the poll “rounded” to 48-47 Romney, if you go down to one decimal point, it’s actually 47.5% Romney, 47.2% Obama.

Second, here is how each state individually changes the results:

• California alone added 1.4% to Obama’s percentage while removing 1.6% from Romney’s percentage, a swing of 3% toward Obama, making the average 48.6% to 45.9% Obama (49-46% Obama)
• Florida alone added 0.2% to Obama’s percentage while removing 0.7% from Romney’s, a swing of 0.9% toward Obama. this would make the average 47.4%-46.8% Obama (tied at 47%)
• Texas alone took 0.3% away from Obama while taking 0.1% from Romney, a swing of 0.2% toward Romney. This would make the average 47.4%-46.9% Romney (Tied at 47%)
• New York alone added 0.5% to Obama and took 0.8% away from Romney, a swing of 1.3% to Obama. This would make it 47.7%-46.7% Obama (or 48-47 Obama)
• All four states together added 1.8% to Obama and took away 3.2% from Romney, a swing of 5% toward Obama. It would make it 49%-44.3% Obama (or 49-44% Obama)

So as you can see, California and New York alone swung 4.3% OF THE OVERALL NATIONAL RESULT toward Romney than it should have if the sample of those states were actually correct. Florida+Texas just threw on an extra 0.7% for Obama. There is no way to know whether there is any similar bias or in which direction in other large states like Ohio, Michigan, or Pennsylvania because they simply did not break them out.

So if someone points to the Political poll as an example Romney taking the lead, just point out to them that it also has Romney ahead 51-43 in California. That should put the end to that discussion.

Oh, and by the way, these Battleground polls are one of the least reliable as rated by fivethirtyeight. It places somewhere between an organization that does internet polling (YouGov) and a polling organization that pretty much literally made their results up (Research 2000).

[Update]

I guess for transparency, I’ll add the raw numbers here:

The “base” Politico poll – that is the poll as it is laid out in the PDF came up with numbers like this. New York was the only one in which there was a question with Romney and “Undecied” both having a half person and one had to round up and one round down, but that’s a difference of 0.1% in the total. All other numbers I’m pretty comfortable being correct:

• “Rest of Country” breaks down to 334 Obama, 321 Romney, 42 undecided
• California breaks down to 43 Obama, 50 Romney, 6 Undecided
• Florida breaks down to 28 Obama, 37 Romney, 1 Undecided
• Texas breaks down to 32 Obama, 35 Romney, 1 Undecided
• New York breaks down to 35 Obama, 32 Romney, 3 Undecided.
• Total sum: 472 Obama, 475 Romney, 53 Undecided

The RCP average in these 4 states (except for Texas, where I took the one recent poll of the state) are the following, rounded to the nearest percent:

• California: Obama 58%, Romney 34%, Other/Undecided 8%
• Florida: Obama 45%, romney 45%, Other/Undecided 10%
• Texas: Romney 50%, Obama 43%, Other/Undecided 7%
• New York: Obama 58%, Romney 33%, Other/Undecided 9%

Thus the ACTUAL breakdown, as close as I could get them, for those 4 states are:

• California: Obama 57 (+14), Romney 34 (-16), Undecided 8 (+2)
• Florida: Obama 30 (+2), Romney 30 (-7), Undecided 6 (+5)
• Texas: Obama 29 (-3), Romney 34 (-1), Undecided 5 (+4)
• New York: Obama 40 (+5), Romney 24 (-8), Undecided 6 (+3)
• Total change: Obama +18, Romney -32, Undecided +14

That shifted the Overall total to: Obama 490, Romney 443, Undecided 67, resulting in the new top line of Obama 49%, Romney 44.3%.