How CNN Doctored up a “Hillary Bounce” and Got Away with it



Like many firm supporters of Bernie Sanders these days,one of my first activities after rolling out of bed is to Google Bernie’s name and click the “news” tab. I woke up today to the top story from CNN titled “Poll: Hillary Clinton’s lead over Bernie Sanders Grows.” Having become used to this type of treatment of Bernie from the corporate media, I clicked it to read and examine whether the headline actually matches the content. My first impression was that this “lead growth” isn’t really very conclusive. Then I looked more closely at the data, and I realized something I could hardly believe – they skewed the results.

The gist of the story was that Clinton’s lead over Sanders, which had been 37% to 27% in this poll, had grown to 42% to 24%. However, the story went on, when Biden is not included on the ballot, that lead balloons to a whopping 57% to 28%.  Initially I was willing to take these figures at face value because they comported with the facts that I believe to be true at the moment. The CNN poll has a margin of +/- 5% for this particular question, and both Hillary’s and Bernie’s numbers were within those margin. Secondly, other polls had demonstrated that Joe Biden’s support has significant overlap with Hillary Clinton’s and that his candidacy siphons far more votes away from Clinton than Sanders.

However, something just didn’t sit right with me. Since the previous CNN poll had been taken, Bernie Sanders didn’t have a noteworthy gaffe, and Hillary Clinton didn’t leave a particularly positive impression either at an interview or at a town hall meeting. In fact, most of the news coverage has been on the Republicans because, thanks to our illustrious DNC Chair, the Democrats are receiving a fraction of the media attention due to a limited number of debates. That’s why I decided to delve into the details of the poll, the numbers behind the numbers, and compare the old poll released September 10th and this new one with Clinton’s allegedly increased lead.

Like any respected poll, CNN lays out its methodology to tell you who they interviewed, how many of each group (Democrats, Republicans, independents and so on) and the margins of error. Both polls are prefaced by this statement. Read this one carefully because it contains the key to understanding what CNN did here.

“Crosstabs on the following pages only include results for subgroups with enough unweighted cases to produce a sampling error of +/- 8.5 percentage points or less. Some subgroups represent too small a share of the national population to produce crosstabs with an acceptable sampling error. Interviews were conducted among these subgroups, but results for groups with a sampling error larger than +/-8.5 percentage points are not displayed and instead are denoted with “N/A”.

For the uninitiated, “crosstabs” refer to subgroups within a poll. So let’s say you want to check how many black democrats support Hillary, or how many college graduates like Bernie, or how Martin O’Malley is doing with voters with a household income of over $50,000, you would refer to the various crosstabs for race, education level and income, respectively. In the passage that I have underlined, CNN is telling you that if certain crosstabs are so small in their sample size that they couldn’t possibly give you a margin of error, then they will not even bother to give you the numbers among the cross tabs because the numbers would be utterly useless and would just indicate “N/A” short for “not applicable”, or possibly in this case “not available.”

CNN pollThere is one other common cross tab that polls look at – age.  CNN’s new poll is LOADED with older voters more than any other age group. We know this because only the 50-64 and 65+ age groups was statistically large enough to give an accurate sample, whereas all the other ones are denoted with “N/A” meaning their sample size was too small.  Even when they combined all the subgroups into two, those over age 50 and those under age 50, the under 50 vote was STILL too small to give a statistically significant sample. In other words, the number of over voters over the age 50 was greater than those under 50.

When you factor in exit polling data that shows voters age 30-44 have outnumbered senior citizen voters in each of the last three presidential elections, when you factor in that the last two have seen voters ages 18-30 leapfrog 65+ voters, you have to ask yourself how representative this sample is that so heavily concentrates on the oldest segment of our population in far larger proportions than what the last three elections would justify. That undersampling becomes even harder to justify when you consider the fact that this is only a Democratic primary, an electorate which we know from other respected polling skews even younger than the general electorate.

Loading the poll with seniors is precisely what gives Clinton such an edge. We have reported that age is a significant factor in the Democratic primary based on previous polling data. For instance the recent CBS/New York times poll showed Clinton in the lead over Sanders, but when crosstabbed with age, it showed Sanders at 40% support among voters age 18-49, essentially tied with Clinton at 39%. Meanwhile, Clinton had a commanding lead among voters over the age of 50, which showed that while Clinton dropped in the previous months in support, Sanders picked up only a third of those voters, while two thirds of them migrated to Joe Biden. It is no surprise then, that in a sample of voters who skew so heavily towards the older demographics that Hillary Clinton would be doing well, and Joe Biden would be competitively running just two percent behind Sanders.

Whether CNN did this because they are in the tank for the Clinton people, or whether they are biased in favor of reporting on a more exciting horserace where each candidate takes turns having good and bad news at polling, or for any other far less siniste reason, it makes no difference that their reporting leaves a great deal to be desired. If I was able to figure that out in less than an hour of analyzing this poll, any journalist good enough to get hired by CNN should have caught it as well. Nevertheless, that didn’t happen and we spent an entire news cycle reading about how well Hillary is doing among voters over 50 masquerading as the entire Democratic electorate. Reporting which lacks credibility does not do any favors to a cable news industry that is already dying because of its failure to get traction with younger viewers. Be very careful what you read, (that goes for this piece too) and always check what’s behind the headline. Last but definitely not least, if you are a Bernie supporter, work to increase turnout of millenial age voters and Bernie will win the nomination, and subsequently the election.

Please follow and like us:

19 Comments

  1. Thank you for examining the design of this poll. I would never have figured it out – but I did wonder “where did they get their sample?”

    “The poll surveyed 392 Democrats and independents.” Now, to me, that seems like a small sample size, when you consider the wide range of demographics in a huge country like the U.S. Again, who/where/how did they conduct the poll? You seem knowledgeable, is that an adequate sample size for political polling? Is that the typical sample size used in political polling?

    • My guess is that the biggest reason for the age issue is that they keep relying far more on landlines(606) than on cell phones(400). This is obviously going to skew the results toward older demographics. Nearly anyone using a landline now is going to be older. Also, when you take into account that there a certainly a decent amount in the older demographic who use cell phones as well, the resulting age demographic is almost definitely going to be skewed. This is not the only poll to have this issue, but it seems to be the worst so far. The N/A look like it they are showing up much more in the national polls for some reason. Maybe it has something to do with cell phone demographic use in the entire country, vs Iowa and New Hampshire.

      • Well, responding to myself… Wow, I should have read my post before posting. Please excuse the grammar/wording. It is early! (I wish I could go back and edit… lol)

        • I was thinking the same thing about landlines. I’m in the “counted” group–but I do have friends and family my age who don’t have landlines any more, too. Still, most of my younger friends and family have nothng else, and my fellow supporters on social media are all ages–many in their forties. Others in their twenties have, nevertheless, never voted before because, until Bernie announced his entry into the race, they didn’t think it would matter who won! So, yes, I think if they counted cellphones, they’d find that Bernie was wiping the floor with everyone else–statistically, that is.

  2. Good detailed critique of underepresentation of younger respondents. It’s difficult to fathom why the CNN pollsters would be so obtuse. Pollsters need to state what sampling model they use so that readers and journalists can better interpret their polls.

  3. I can’t tell from this analysis whether the younger voters’ responses were totally deleted from the population analysis (which is what I think you are suggesting) OR whether thy were included in the population analysis but not singled out for separate reporting due to the smaller numbers. There’s a difference.

    • Well, I wondered about that too, but I believe there is an issue either way. As it states in the methods, they try and weight the different sub-groups to reflect national census figures. The problem is, the sub-groups for those ages were so small, whether they decided to include these subgroups or not, the only way to get anywhere near an accurate representation would be to then weight them, but with that small a sample, it seems it would likely be very inaccurate anyway. So no matter what they decided to do, the poll is obviously flawed, and it is very disappointing the majority of the news media seems to be too lazy to look at these things at all first. Seeing N/A across everything below 50 was pretty hard not to spot, which leads me to believe most of them never even looked at the poll, because that would at least deserve a comment.

  4. I understand. What I find confusing is the use of the term “unweighted” in the documentation. That suggests that the sampling was not a simple random sample but a segmented sample involving disproportionately sampled segments whose responses were weighted back up to represent the total population even though sample sizes and sample proportion may have varied from segment to segment. (I’m not trying to make any excuses for CNN I’m just hypothesizing.)

  5. So, essentially what you are saying is that even thought they weight the polls based on the census, because they under sampled the under 50 population that its incorrectly skewing Hillary? So, basically if the population sampled happen to be pro Hillary, when they weight that out it skews the results? Wouldn’t they think of that in their model? I’m not a statistician, but I’m sure they got some pretty smart people over there to build that in. I’m a Bernie supporter and would like that to be true, but I would also like to chalk this up to a blip. The overall trend is going the right way 🙂

    • Actually they are not weighting for the census and that’s the point of the article. If they are weighting based on census population, then voters over 50 are less than half of the population. http://www.census.gov/prod/cen2010/briefs/c2010br-03.pdf The poll included them them as more than half of the voting body. 50+ voters weren’t over half the voting body in the last three presidential elections. You can see the link in the article referencing exit polls.

      Last but not least, Democratic voters are younger than the general election voters which the exit poll measured because more young people and fewer older ones are sympathetic to the Democrats compared to the general population. So any way you look at it, there are too many old people in that poll.

      • This conversation has not convinced me that CNN is conspiratorially tilted the survey findings. Disproportionate sampling sometimes results in making it misleading to report individual segments even when total population results are reported. Wanting results to go one way does not make it so no matter what the client or reader wants to be the case. That said I do find amusing that the MSM are being pilloried for being “pro-Hillary” when the MSM have kept the Great Email Scandal going for so long. It’s politics folks and if we try hard enough we can make the poll numbers say anything we want them to. Time to move on.

  6. They were undersampled because 51% of respondents identified as independent and were discounted for not “leaning democratic”. 508-131=377 377)1006= 37% of respondents eliminated. Those Independents are still voting for someone. As Stalin said, it’s not about who votes, it’s about who counts the votes.

  7. I too thought the CNN polls were way wrong. Trump’s numbers should have been up and were with all the other polls except CNN’s. So either CNN tried to mislead America on both Trump and Sanders or they had a flawed poll. Perhaps it was because they had way too many land line participants. I mean who still has a land line except our grandparents?!?!

  8. They haven’t gotten away with anything yet! Those of us that are aware are pissed to say the least and many of us are willing to double down and reverse the potential negative effects it might have back into Bernie’s favor! Keep Pushing Forward! #FeeltheBern

  9. There are a few things your theory does not seem to explain. If the increase in Clinton’s lead compared to the earlier CNN poll was manufactured by undersampling younger voters, then:

    1. Why did the earlier poll also have N/A for under-50 respondents? If undersampling younger voters is not a change in methodology, it’s unclear how it produced a change in result.

    2. Why did Clinton’s share even of the 50+ respondents increase? She was supported by 42% in the old poll and 48% in the new poll.

    3. Why did Sanders’ support drop more among younger respondents than older voters? Sanders was supported by 19% of the 50+ respondents in both the old poll and the new poll (Clinton’s increase came from people who previously supported O’Malley or “other candidate”). This means that Sanders’ drop from 27% to 24% in the overall sample must have been driven by lower support among younger respondents.

    • Again, you’re relying on assumptions which are all within the margin of error and are not statistically significant, which is the case of 50+ voters with a +/- 6.0%. In the case of under 50 voters, the sample is so small that we can’t even formulate any statistically significant findings because the margin of error is over +/-8.5%. The CNN Poll explains that.

Leave a Reply

Your email address will not be published.


*