GrrlScientist has a good pointer in her Guardian blog entitled “Baloney Detection Kit”, and yes, the title is a heck of a lot more polite than I am. I’ll call a spade a spade … we are awash with bullshit, so it is indeed vital to be armed with a good means to work out what is actually reasonable and rational, and what is pure unadulterated bovine deposit.
She also has the video by Michael Shermer (yep, our friend and hero from Skeptic magazine), so you know its going to be good. He has ten questions you always need to ask yourself when faced with a claim, here they are (with a few notes and comments added by me for each of them).
1. How reliable is the source of the claim?
All sources are simply not equal. Think about it now … “X causes cancer” … if it came from the cashier in Tesco, would you believe it? … what if you read it in the Daily Sport? … or how about the Independent?. If however it was a paper from the British Medical Journal, would that do it for you?
2. Does the source make similar claims?
There are specific media outlets that are more interested in drumming up circulation, so they will fill their pages with claims that might or might not be true, and often are simply there to attract readership. If they have a past track record of highly dubious stuff, you can take anything said with a huge grain of salt. In contrast, media sources that fact check and also link back to studies and papers tend to be credible, and also have a diverse range of news because they follow the data, not a party line.
3. Have the claims been verified by somebody else?
One paper does not make it a done deal, and one witness does not verify that a sighting is real. Have others independently verified the claim.
4. Does this fit with the way the world works?
The royal family are really all giant lizards ruling over us … and of course we know this is true because that is the way things are … right?
5. Has anyone tried to disprove the claim?
“Prayer really works”… Ah but has anybody actually done a proper clinical trial to see if anything measurable actually happens? In fact they have, I’m sure you can guess the results of the truly scientific tests, but I’ll leave it up to you to google that if curious.
6. Where does the preponderance of evidence point?
“A 600 year all chap put every species on the planet into a boat and saved them all, is it really true?” – What comes next is perhaps an observation that many cultures have flood myths, so that’s it then it is all true. Ah but wait, do we have any biological evidence, or geological evidence that any of this is true? Nope, not one jot.
7. Is the claimant playing by the rules of science?
Have they proposed a hypotheses that can actually be tested? If not, then it is not scientific. Something that cannot be measured or tested is in fact the same as something that is not real.
8. Is the claimant providing positive evidence?
Creationists will happily tell you what is wrong with evolution. If you ask them for evidence that supports the “God did it” hypotheses, you simply get more lectures on what is wrong with evolution.
9. Does the new theory account for as many phenomena as the old theory?
You know the true secret of the universe, all we know is wrong and you have a grand unification theory that replaces all that came before it. OK, fine, but can you can explain everything that all those previous theories explained?
10. Are personal beliefs driving the claim?
Indeed yes, folks often start with a belief and then work backwards looking for anything to confirm it, discarding anything that does not fit.
These are all good questions to be asking whenever you are faced with any claim at all, UFOs, Ghosts, Gods, lake monsters, miracle cures, coffee cures cancer, etc… you get the idea. This also need not be a definitive list, I’m sure you can think of lost more, for example …
– Does the claim come packaged with woo words, or new age speak? – “Well this works because of the quantum fluctuations of water memory” … the correct reply to that is ???
– Is real evidence missing and it is simply,said it, hence it must be true – that of course proves nothing at all, did they really say that?, and even if they did, so what, that is not evidence.
I’m sure you can think of lots more.
Here is the actual Video itself … and yes, its been around since 2009, so is not new, but hey, woo is still woo, so it still stands …
This is not the first such kit or list, as Shermer explains in the above video, this is all very much based upon the similar list created by Carl Sagan, his was as follows:
- Seek independent confirmation of alleged facts.
- Encourage an open debate about the issue and the available evidence.
- “In science, there are no authorities. At most, there are experts.”
- Come up with a variety of competing hypotheses explaining a given outcome. Considering many different explanations will lower the risk of confirmation bias.
- Don’t get too attached to your own ideas, lest you get reluctant to reject them even in the face of evidence to the contrary.
- Quantify whenever possible, allowing for easier comparisons between hypotheses’ relative explanatory power.
- Every step in an argument must be logically sound, a single weak link can doom the entire chain.
- When the evidence is inconclusive, use Occam’s Razor to discriminate between hypotheses.
- Pay attention to falsifiability. Science does not concern itself with unfalsifiable propositions
Now, what really is behind such sets of questions is the deployment of the scientific method. That alone has profoundly changed our world and given us a true understanding, nothing else has done so to such a degree. You see, the secret here is that “science” is not a body of knowledge, but rather is the very best means we have for working our the things that are really true. We can all deploy this mindset to any claims that land at our feet and filter out the stuff that is simply wrong.