Jim Kane Explains: Possible Reason Why Presidential Polls Have Different Results
BY JIM KANE
Two national polls this past week had completely different results on the Obama/Romney race.
Gallup had Romney up by 2 points (47% to 45%) while CNN/ORC had Obama up by 9 points (52% to 43%).
Both polls were done within the same period (April 11-15) and both were large samples of registered voters with sampling errors of +/- 3% (Gallup) and +/-3.5% (CNN).
Obviously, the differences cannot be blamed on sampling error.
Curious, I decided to have my University of Florida students look at both questions to see if somehow the wording of the question may have caused the differences in results. (We call this context effects.)
I thought that unlikely since both polling firms have a long history of doing excellent surveys, but these are graduate students studying political surveys and I thought the exercise would be beneficial.
At first glance, it appeared that both “trial ballot” questions used were nearly identical.
A closer look revealed something that surprised me and my students.
The Gallop poll used the phrase “If Barack Obama were the Democratic Party’s candidate and former Massachusetts Gov. Mitt Romney were the Republican Party’s candidate, who would you vote for?”
The CNN/ORC question only identified the “Barack Obama as the Democratic Parry’s candidate, and Mitt Romney as the Republican Party’s candidate.”
In survey research when you identify a person’s office or position, it evokes in some people a sense of additional importance. This is called the “prestige effect” and virtually every experienced polling firm avoids this mistake unless they are intending to bias the outcome.
Although most folks know Romney in name, most average voters would be hard pressed to know his previous role as Massachusetts governor. Additionally, this added reference can add an untended image of Romney linked to Massachusetts more progressive (liberal) policies.
Perhaps some voters (mainly ambivalent voters) have an initial concern that Romney may be “too extreme” on social issues due to the hard right positions he took in the primary. The reference to Massachusetts could possibly mitigate this concern.
But that’s the problem with unintended effects. You don’t always know how thought processes are changed, if at all.
I do not think Gallup intended to bias the outcome, but did this inadvertently, probably thinking voters needed to know more about Romney’s past experiences. I have never, however, seen this before in any of Gallup’s previous polls.
(Browardbeat.com pollster Jim Kane has conducted polls for candidates, sponsors of referendums, state political parties, media companies such as the South Florida Sun-Sentinel and business groups. He was the first Democratic pollster to work for the Republican Party of Florida. In addition to his political work, Kane currently teaches graduate seminars on the Gainesville campus of the University of Florida in survey research, political behavior, political campaigning and political parties and interest groups. He is frequently quoted in national publications and is a commentator on national television.)
April 22nd, 2012 at 9:02 am
Jim,
Great article, but:
“Obviously, the differences cannot be blamed on sampling error.”???
You and I both know that the more correct answer is “sure they could be…. it is just incredibly unlikely.” Still, someone wins the lottery almost every week, and that’s pretty dang unlikely at the individual level.
We also know that 5 out of 100 polls, even if drawn from the same sampling distribution as the other 95 and done identically, have a 50/50 chance of being wrong (“wrong” meaning significantly off the true population mean).
With the tons and tons of polls we have these days, I wonder why we don’t point that out more often.
Kevin.
April 22nd, 2012 at 10:13 am
Jim is right and its about time that people realize it. As a political consultant who will remain anonymous here, I have seen many polls done locally which are meaningless because the questions are skewed.
April 22nd, 2012 at 10:35 am
Separate from the phrasing and semantics of the question, where was this poll you cite conducted/ limited to a small geographical area i.e. a county, tri-county, Tri-State, or a sample drawn nationwide? You mention UofF students so was it Gainesville only and the county? Was it in person, over phone, or via email? Did it include “If you were to vote today?” or did it infer November 6? Not to criticize your write up but in my opinion the knowledge, engagement and even honesty of those polled varies from where its drawn and the demographic characteristics. Separately each candidate can do much to better or worsen their chances in next 6-1/2 months so the sampling is a snapshot from the day, and polls will not really be accurate till about 3 weeks before we vote.
April 22nd, 2012 at 2:19 pm
The big mistake in both polls is that they sampled registered voters rather than likely voters. Too many “registered” voters don’t bother to show up at the voting booth. So the polling results are skewed and inaccurate.
April 22nd, 2012 at 3:05 pm
Don’t forget to add a few points to Obama’s total since there is no way to poll the hoards of deceased voters who will be casting a vote for the Democrat candidate come November.
April 22nd, 2012 at 6:32 pm
The only poll that counts is the one on election day. I know all these “Professional Pollsters” will howl when I say this, but the truth hurts. The outcome of most polls depends on who is paying for them. If a conservative pays, the conservative is shown to be in the lead. Same for Dems, if a Dem pays, then the Dem leads. BTW, I would not trust any poll from a news source like CNNCBSNBCABC or FOX.
April 23rd, 2012 at 12:14 pm
Kevin,
You are quite correct that there is a small chance that the results of the survey exceeds the sampling error. As you know, there could be other errors that can creep into almost every survey (measurement error, house effects, etc.) In this case, however, the error is likely the question wording itself and
not the small probability of errors
in excess of expected sampling error. As Robert Redford noted in his movie Havana:”I believe a butterfly over the Atlantic and can sometime start a Hurricane. I can even calculate the odds. It just isn’t very likely.”
April 26th, 2012 at 7:57 am
Polls have become a little understood and greatly influential part of the process. If Mr. Kane would regularly explore the story behind various polls it would be very helpful to the rest of us.