UC News

'Dubious' Achievements from Polling World Revealed in UC Prof's Rankings

headshot of hoffmacd By hoffmacd
6 minute read February 4, 2010

Here’s a statistic that is impossible to argue against: When shown numbers that result from a compromised or poorly-designed poll, 100 percent of those reading the poll come away misinformed.

University of Cincinnati Professor of Political Science George Bishop, one of the nation’s top experts on proper polling methods, has teamed up with the University of New Hampshire’s David Moore to again do battle against examples that came out in 2009 of just that kind of polling. Their

second annual “Dubious Polling Awards,”

which have just been published by the media ethics Web site

StinkyJournalism.com

., lists 10 of the most suspect moments in polling from the last year.

Employing a scale of 1-5 crossed fingers to indicate just how flawed each example on the list was, Bishop and Moore decided three examples rated the worst ranking of five crossed fingers:

  • The “Phantom of the Soap Opera” award went to Strategic Vision, LLC, a firm that has gained an audience for its polls with the national media, but of whom many questions are unanswered. The American Association for Public Opinion Research (AAPOR) censured Strategic Vision last fall, in response to the firm being unable to provide supporting information on polls it said it conducted in primary states in 2008.

     That was just the start of questions, though, Bishop and Moore report, as basic information as to whether Strategic Vision actually had any offices – they claimed to operate out of four cities – and where the workforce was that did their polling began to be asked.

  • The “Stonewalling/Coverup” award was given to Johns Hopkins Bloomberg School of Health and one of its professors, Gilbert Burnham, for another project that earned censure from the AAPOR. In this case, it was a survey project by the school that claimed 600,000 civilian deaths from the Iraq War between 2003-06.

     The problem was, Bishop and Moore write, that the figure claimed in the survey was  anywhere from seven to 14 times as high as any comparable study, and Burnham refused to provide any documentation to back up how his survey was conducted. When called on it, the Bloomberg School then refused to investigate.

  • The “Manufactured Out of Whole Cloth” award went to six different polling groups – Zogby, Ipsos/McClatchy, ABC/Washington Post, Associated Press/Stanford University, Pew Research Center and CNN/Opinion Research Corporation – for their heavy-handed efforts to manufacture public opinion on the subject of cap-and-trade environmental legislation.

    The problem is revealed when you learn that less than one-in-four respondents could accurately link the term cap-and-trade to the energy and environmental field, and more than 50 percent of those asked admitted they knew nothing at all about such policy. Bishop and Moore, then, are left with the conclusion that the polling organizations could only conduct polling by feeding information to the respondents, producing polling results that were all over the place. The Zogby poll on the subject showed opposition to such a policy by a two-to-one margin, while cap-and-trade when polled by CNN produced a result of two-to-one in favor of the idea. As they note: “Once pollsters give information to their respondents, their samples no longer represent the larger population, which has not been fed that same information.”

    Plenty of others were in the running for “honors,” including more of the biggest names in the polling business.  That’s just the point, though, in an exercise like the “Dubious Polling Awards,” which highlights the fact that bad polling can come from anywhere, large firms or small, political issues or otherwise. It’s up to the end user to be discerning in what any polling is actually showing.

    Establishing that standard has been a career-long quest for UC’s Bishop, whose most recent book is “The Illusion of Public Opinion: Fact and Artifact in American Public Opinion Polls.” Not all polling is created the same way, nor practice the same way.

    "The quality of measurement in public opinion surveys varies considerably between academic and commercial polling organizations,” says Bishop. “The results of mass media-sponsored polls often give the impression that a definitive public opinion exists on numerous issues of public policy when there is, in fact, widespread public ignorance and poorly informed opinions on such issues at best."

    As an example, Bishop cites the recent Pew Research Center poll showing how little most Americans know about politics and public affairs. Asked 12 relatively simple questions about news and current events, most Americans could not answer even half of them correctly.

    (You can take the political news quiz for yourself and see how you fare at the Pew Research Web site.

    Bishop’s partner on the project, David Moore, has a long and distinguished history in the public opinion field, as well. Currently a senior fellow with the Carsey Institute at the University of New Hampshire, he is a former vice president of the Gallup Organization and was a senior editor with the Gallup Poll for 13 years. His most recent work is the book “The Opinion Makers: An Insider Exposes the Truth Behind the Polls.”

    Bishop and Moore’s entire story about this year’s “Dubious Polling Awards” can be read online at the Stinky Journalism Web site.

    • Tags

    Related Stories

    3

    NIS program opens new horizons for international student

    July 17, 2024

    In his pursuit of physics and a taste for research, Akash Khanikor ventured from his hometown in India's Assam to the University of Cincinnati, drawn by the promise of hands-on exploration early in his undergraduate career as a NEXT Innovation Scholar.

    Debug Query for this