Opinion polls come in many shapes and forms. You can’t take any one poll as being on par with every other poll. When you consider them all, it is clear that we have a range that could perhaps best be described as the Good, the Bad and the truly Ugly.
As the election closes in you need to be armed with some questions that you can use to determine if something that a poll claims is credible. Consider for example being told that “A Poll of Fox News viewers confirms X”. You will immediately think, quite rightly, “But they don’t represent all of us, that is just one very specific political demographic“.
OK, so you have a poll, any poll, what questions should you be using to assess it?
1. Where exactly does this Poll come from?
Is the source for the poll a credible source, or has it been compiled up by an organisation that has a very specific political agenda?
It might not be too obvious.
A poll sponsored by the Union of Bread makers verifies that sliced bread is the greatest invention since unsliced bread is a poll that need not imply bias. If they simply engaged an independent and highly reputable polling organisation to go find out, then it just might be credible. Rather obviously if they themselves, with no real independence, conducted the poll, then you would be right to dismiss it as unreliable. That remains true even if you do think the results are correct.
The reputation of those that actually conducted the poll can be used to gauge it.
2. Do the results stand out in contrast to many other similar polls?
If most polls lean one specific way, and yet a poll pops up claiming something completely different, then clearly either something dramatic has happened since those other polls were conducted, or the people compiling this latest poll have either deliberately manipulated it, or simply made a bit of a hash of it.
3. How big is the poll?
If a small number of people were polled, then it may not be statistical meaningful.
If the issue is of national interest, and only people in one specific city have been asked, then it probably does not represent the diversity of opinion that often exists.
4. What are the Poll details?
When was the poll conducted?
- If it was a long time ago, and much has changed, then it most probably does not represent current opinion.
Are those polled a truly representative sample?
- Would you really trust the results if the only people expressing an opinion were old white guys who had only ever voted GOP?
How where those that were polled selected to take part?
- If for example it was a poll on gun use and those selected were people “randomly” picked as they came out of a gun store, would that truly be a fair representation of public opinion?
Are the methodology details available?
What is the published margin of error?
If you are faced with a bit of political polling, but many of those polled are not registered voters, then you might be right to question what it claims to be telling you.
In the end it is simply an expression of an opinion in response to a set of specific questions. These can be phrased to nudge those being asked in a specific direction. Are the list of questions asked available?
If no details regarding the poll are available then you should treat it as a result that is potentially highly dubious.
Bottom Line: There is a vast diversity of processes in play out there. Deploy doubt and skepticism to filter out the highly dubious.
- If the poll does not explain how it was conducted, when, or by whom, you would be wise to dismiss it.
Further Reading
- FiveThirtyEight’s Pollster Ratings – Based on the historical accuracy and methodology of each firm’s polls.
- Pew Research – A Field Guide to Polling: Election 2020 Edition
- Princeton Library – Political Opinion & Survey Data Sources
Reality check – humans have a distinct bias …
- University of Pennsylvania – Individuals find polls that favor their candidates more credible