O'Neil Perspective

Pollster reflects on Polls: Why the Worst Polls Get Reported the Most

mikeoneil | 20 May, 2010 15:16

MEMO


To: My friends in the local media

 

Question:  Which news organizations refuse a matter of news policy, to run the results of robo-polls (polls conducted without the use of a real live interviewer) such as Rasmussen?

Answer:  ABC News, NBC News, CBS News, CNN, the Associated Press, the Wall Street Journal, the Washington Post.  They all have professional researchers on staff who have set standards for those organizations.

Question:  Which news organizations run these polls all the time?

Answer:  In my home town of Phoenix, that would include TV channels 3, 5, 10, 12, and 15 (not sure about KAET/Channel 8).  Also The Arizona Republic and The National Enquirer.

Ok, I made the last one up, but that is probably not much of a stretch given their journalistic standards.  Which company would you care to be in?

Why do we see a proliferation of robo-polls (such as Rasmussen and SurveyUSA)?  Because they are absolutely free to conduct.  You throw a bunch of phone numbers into a database and have a large volume of robo-calls made over an IP telephone and you can conduct the entire survey at virtually ZERO cost in 3-4 hours.  But no callbacks to those not at home (so samples over-represent those who never leave the house), no respondent selection within the household (your eight year old can answer if he can operate a touch-tone phone), no live interviewer (effects unknown, but note the reference to the eight year old).  And seldom more than a single question.  Imagine, journalists, that you were restricted in your interviews to asking a single question of your news sources.  And that they could only respond with a monosyllabic response.  How much insight would your articles have? But these polls are quick, and the price is right.  As a result, they are numerous--dwarfing the few legitimate polls that are out there.

Why do you not see very many legitimate polls?  A scientifically conducted poll has to be expertly designed and administered with real live trained and supervised interviewers.  This costs money.  Who is willing to actually pay money to conduct legitimate polls on topics of public interest?  Those with a vested interest in the outcomes (think: slanted questions designed to show that the public agrees with their positions).

Who should be interested in actually informing the public about what the public thinks about the important issues facing us?  I’d like to think this might include large multimillion dollar media organizations.  Yea, I know times are tough and news rooms are being cut back everywhere.  But I’d hate to think that the new standard (with apologies to the New York Times) is “All the News that’s Cheap to Get”.

I have been involved in designing, reporting, and analyzing public opinion research data for over 35 years.  Obviously, I share this interest in informing the public.  But my resources are just a tiny bit less than those of the combined resources of five major market TV stations and a dominant newspaper. So when you (local reporter) as me (as happens all the time): “Do we have a poll on XXX” I always think to myself, “Which of us works for a major metropolitan news organization?”  (Side note: since we are conversant with local and national trends having observed them for decades, we don’t have to have personally conducted a poll to have insight.  Indeed, the really important stuff is seldom the answer to today’s question du jour but rather insight from observing patterns in all of the available information).

But, I pose this question to every news director in town:  I know budgets are tight and journalists are being laid off all over town.  But, when was the last time we learned anything significant about the functioning of our community as the result of a helicopter picture?   And what is your budget for that helicopter?

Print this out and slip this under your news director/editor’s door.  But you might want to do so at night after everyone else has gone home.

(Note: Sad to see the demise of the Channel 8 Cronkite Poll: it was the best source of data on attitudes towards public issues that we had with any regularity.)

In the meantime, at least we don’t get polls from “Strategic Vision” a “national” firm whose poll results were widely reported in the media throughout the 2008 election.  Upon inquiry by the American Association for Public Opinion Research they were unable to produce a dataset to demonstrate that they had actually collected any data (http://www.aapor.org/ AAPOR_Raises_Objections_to_Actions_by_Strategic_Vision_LLC.htm). A little further investigation by others found that the firm’s alleged “Atlanta headquarters” were, in fact, a UPS Store mailbox and that the real “headquarters” were in a motel park in a rural community over two hours away.  They were offered a cash reward to name any field (interviewing) service they had hired to conduct actual interviews.  (See http://www.fivethirtyeight.com/). They never responded to that offer. But the numerous “polls” that had been widely reported in national media stopped coming.
 
Caveat emptor.


Michael J. O'Neil PhD

General | Next | Previous | Comments (0) | Trackbacks (0)

Comments

Add comment
 authimage
 
Accessible and Valid XHTML 1.0 Strict and CSS
Powered by LifeType - Design by BalearWeb