THREE
EASY
WAYS TO PUNCH ABOVE YOUR WEIGHT
“CURIOSITY IS THE WICK IN THE CANDLE OF LEARNING.” - WILLIAM ARTHUR WARD
There are three simple questions that should help you scrutinize any opinion, fact or insight presented to you. Interestingly most children start out their lives asking the very same questions, but somehow once they grow up certainty, carelessness, and cowardice take the place of curiosity.
HAS THE CLAIM BEEN TESTED AND HOW?
Why is it that we want to know how a magician does his tricks? We investigate his sleeves, ask him to “do it again” or try and do it ourselves. But when it comes to health claims, we often stand in reverence. Has the claim been tested and if so how? There are three hallmarks to a proper test. The tester has done everything to prevent him/herself from falling prey to the psychological mechanisms described above. (In the following discussion, the term is used to refer to studies where it is beyond the scope of the investigator to interfere with the subjects by creating a control and treatment groups.)
Have the testers ensured that their own wishful thinking or the wishful thinking of the participants is not influencing the results? For example, when testing a herbal remedy, an experimenter can prevent the participants from knowing if they are taking the real thing or a placebo by a process that is called “Blinding” (neither the experimenter nor the participants should know the answer). It is uncanny how our minds can figure out and influence an outcome. Subtle clues from a knowing experimenter are sufficient to skew the results. You may remember the story of the horse that could count: The owner would ask the horse a question such as “what is 5 plus 7?” and the horse would tap its hoof an exact 12 times on the floor. Many people studied the apparently amazing intellectual quality of the horse, but in the end, it was the horse’s ability to sense the subtle satisfaction of his owner when he hit the right number.
Has the experimenter used proper randomization? For example, imagine that you would want to check whether eating seal and whale meat/blubber prolongs life. You know how long an average person lives and you would feed a group of selected people with these products. However, if you would recruit a group of athletes to test your idea, you would get very impressive results as your athletes would live longer compared to the average person irrespective of what they eat. Your conclusion would be wrong. By the way, that is probably the main reason why the Inuit (those with a traditional lifestyle rather than a western lifestyle) have such a low incidence of cardiovascular disease: They have a very high level of physical activity (hunting seals or whales), and they rarely over-eat as food is scarce in that kind of lifestyle. The conclusion that eating large amounts of animal meat and fat is not detrimental to your health, cannot be extracted from this particular observation as some books have claimed.
How substantive is the test? For example, has it been tested on a few people as is often the case for expensive medical procedures or on a large group of people, which is the case with large scale multiyear observational studies such as the Nurses’ Health Study, that is tracking over 100,000 nurses since 1976 and is the largest cohort study in women. This type of study is obviously very valuable in evaluating different lifestyle, medical, and nutritional causes of health in a relatively homogeneous group of women.
In larger studies, where multiple sub-populations may exist, has there been an effort to control for known factors and how? Sub-populations in large studies may fall into clusters having significant differences that affect the outcome being studied. For instance, in a study evaluating the effects of nutrition on serum HDL levels, it might be important to normalize, cluster, or otherwise account for varying levels of exercise.
Generally smaller interventional studies provide a better ability to control variables and more crisp results, but the results may be difficult to easily extrapolate and by virtue of probability alone may show erroneous results (remember that it is perfectly normal to get a run of heads in a short sequence (let's say 5) of coin tosses, but it does not mean that the coin is unbalanced in any way).
Larger studies (especially observational) can be generalized more readily, but often have a higher difficulty in controlling all variables.
HAS THE CLAIM BEEN VERIFIED BY SOMEONE ELSE?
Truth is so precious, it must be surrounded by a bodyguard of lies – Winston Churchill
Even the most meticulous and conscientious people will make mistakes. Therefore the quality of any claim is significantly higher if it has been tested by unrelated people (i.e., if everyone in your hunting club swears that they saw bigfoot it would carry less relevance than if bigfoot would be spotted by a group of girl scouts, some Japanese tourists as well as a few people from your hunting club).
A powerful way to see if there is some bias in the quality of the claim is to look at the full set of studies that have been conducted to date (this is called a meta-analysis). This perspective gives a few very valuable clues:
You can get a more accurate estimate of a particular effect as you get the average value of all results. This method averages out systematic errors in different experiments.
You can see if the distribution of all findings is balanced. Typically the spectrum of results should distribute evenly around the average answer (the smaller studies also tend to give a wider spread). If investigators have not published negative results or otherwise biased the outcomes, then you would see an uneven distribution.
Check the quality of trial vs. Outcome.
Remember though that the person conducting such a meta-analysis (especially when they differentiate the results by what they may exclude (for example due to quality requirements), may have a particular bias themselves, and such an analysis is one way to confirm but does not constitute proof.
While the application of Meta-analysis is becoming more common, I have gained particular respect for the Cochrane Collaboration. It is an international non-profit (they also do not accept commercial or conflicted funding) network of healthcare providers, researchers, patients, and policymakers focused on creating, independent insight that follows the highest standards of practice, and is fully open to and encourages challenge. Where available, I include the results of findings published in the Cochrane library.
IS IT PLAUSIBLE IN TERMS OF WHAT I KNOW?
“Plausible impossibilities should be preferred to unconvincing possibilities” – Aristotle
The very first filter you should apply to any new information is your own sense of plausibility. Look for inherent contradictions. For example, imagine that you are visiting a psychic who is telling you about your past and says that she is sensing the presence of “John” and if that name means anything to you. Your response, rather than searching in your mental inventory for all the Johns you know, should be the most plausible response: “can you tell me John’s last name please! That way I can tell you for sure if the person means anything to me”. Obviously, if a psychic can come up with a first name, they should equally be able to come up with a last name. Failure to come up with a last name makes the claim implausible (and will shorten your session).
The second filter you should apply is whether you actually understand the explanation. Either the person or organization making the claim can explain things in a simple way, or you can be sure that they are either confused or compromised. Remember the psychological effects described above.
The third filter is whether a claim is testable (or as the philosopher Carl Popper put it falsifiable). All this says is if I have to believe what you say, but there is no way for me to test it, then I should believe anything that you say. Unfortunately, this is what often happens in a relationship between a guru with his/her disciples and in a milder form when we accept a fact without scrutiny. For example, the claim that bad air (mal-aria) causes Malaria is difficult to test as the term is very vague and could mean stagnant air, smelly/smoky air, too dry or humid air. However, the statement “air near a swamp causes Malaria”, can be tested and would lead us closer to the true cause.
The Malaria example also points us to the next filter: Is there an alternative explanation? We often accept a claim because the explanation given is the only explanation available. Because it is the only explanation available, it is also well formulated. Alternative explanations initially are bound to be crude and can be ridiculed easily.
Finally, and very importantly, is there a physical, chemical, or biological mechanism that can support a claim? This does not mean that there has to be full consistency with our current knowledge of nature, but that your skepticism of a claim should increase with that claim’s distance from known facts. To put it differently, a claim that people can live without breathing for 10 minutes needs less of an explanation than the claim that people can survive for 10 years without breathing. The first statement does not fly in the face of what we know about human physiology and the ability to reduce oxygen consumption by, for example, reducing the body temperature, etc.; while the second statement does not have a plausible known mechanism. Extraordinary claims require extraordinary evidence.
MAY I SUGGEST THAT YOU DEVELOP AN ALLERGY TO THESE TYPES OF STATEMENTS?
“Most doctors/football players/movie stars/ agree”
This is called social proof or appealing to authority. It is a quick way for us to accept a fact, but beware of people who insist too much on this rather than providing the proof
“It is all-natural and good for you”
Deadly nightshade, the Portuguese man of war jellyfish, Poison oak, Asbestos, Ricin are among billions of perfectly natural things that are – well - not so good for you
“My best friend’s sister was a complete skeptic but went ahead with the procedure, and against all expectations, was healed...”
It is so much more convincing if a skeptic becomes a believer! But anecdotes are just that: a story for the campfire. But beware of extrapolation, compelling stories instead of facts, the survivor’s bias we discussed above, or simply the Placebo effect showing up.
Pseudo-scientific statements containing words such as “quantum”, “field”, “detox”, or even better a “quantum-field-detox”. The world is full of people who borrow words to make their argument sound more important. Just ask them to define their terms and often they just fall on their face.
Pseudo-cultural statements. While I do not want to denigrate any culture or ancient wisdom, most people using those words often lack any clue and have a hard time telling me in simple terms what that amazing wisdom really is. True wisdom is simple and understandable. Many scientists also fall prey to this and believe that by making things "sophisticated", it somehow shows their intelligence.
Vague statements (see your daily horoscope for a good selection).
"You are a person who loves to learn new things, but also sometimes enjoys doing nothing and just savors the moment. You are surprised by how you can predict what happens in the next moment and your friends rely on your foresight. Being a (fill in the star sign), you seek community and interactions, even if you have to recharge your batteries by choosing solitude and a quiet moment from time to time." I could go on and on....