Fantastic article, Anoop!
I think that most of peoples doesn’t aware about existence of scientific approach to training.
The sentence like “you must try everything until you find something that suits you” rules on many forums.
And this one: I did X and got Y, why don’t you try do the same, is simply classic.
😊
Why Anecdotes/Testimonials Are Unreliable?
March 16 2010
If you are familiar with this website, I always insist on an evidence-based approach and do not put much value in anecdotes or testimonials. Many people just say science is important without fully understanding why evidence-based approach or science evolved.
Here are some of the reasons for why anecdotes/testimonials are considered the lowest form of evidence and why an evidence-based approach evolved.
Confounding variables
There are too many variables like diet, age, trainee level, genetics, sex that can significantly affect the results of the training program.
“A muscular guy says he does this special program, so I am going to try his program.”.
The problem with this testimonial is that most muscular guys are just genetically gifted. They can do just whatever and eat whatever and still be under 8% body fat.
“That exercise program didn’t work for me”
Here is a guy who has given a decent program but didn’t make any progress because his diet was still the same.
“That program just killed me”
Here is example of 40 yr old guy who just copied a workout from a 20 year old who got great results.
In research, these confounding variables are eliminated by random selection from the specific population and randomizing people into groups, restricting, and matching. The randomization eliminates the confounding variables and hence is very important in research.
Hidden Data
In testimonials and anecdotes, you only read about positive results. The people who didn’t gain much never bother to talk about it.
“Program X worked worked for 3 of my friends”.
Does that mean X works? What if I show you 5 people (hidden data) for who the program X didn’t work or who dropped out?
In research, the negative hit and the number of people who dropped (attrition rate) out are also recorded.
Cognitive Biases
These psychological biases distort reality and judgment and has been extensively studied and confirmed in the psychology field. A few of the biases which can skew reality are memory recall biases, expectancy effect, exposure effect, bandwagon effect and so on. There are over 35 or more of them which becomes relevant depending on the specific context.
“I do high reps with light weights to tone”
This is classic case of bandwagon effect or herd effect where people just follow what others do without analyzing the underlying evidence.
“I know it works. I have been doing this for almost 20 years and there is some research to show it works”
This is an example of the confirmation bias and is one of the most common biases out there. Here people tend to prefer, seek, and interpret information which confirms their hypothesis and neglects the opposing data.This bias is so often seen among fitness guru’s who have a product or idea to sell.
“That diet doesn’t work. I didn’t eat too much ”
According to the social desirability bias, there is a tendency to provide answers which is more socially acceptable /desirable. This type of bias has now been widely accepted in nutritional studies which include dietary recalls.
“I was really excited about the program and it worked great”
This is called the hawthorne effect where just by believing in the program the person was doing all the right things that he never did with other programs. He is eating all the right foods, getting enough sleep and so on.
In research, these biases are taken into account and eliminated by having a control group and blinded designs. For example, in double blinded trials, neither the researchers nor the participants are aware of who belongs to which group.
By Chance
Hypothetically, if every bias and confounding variable is taken into account, there is still a possibility that the results were due to mere chance than the program/diet.
In science, the chance is taken care by the p value (p less than .05) as often seen in studies. If the result are statistically significant, it means there is only a 5% chance the results were due to chance or accident.
Subjective Outcome Measures
Anecdotal Outcome measures are extremely subjective. For example , judging muscle size by looking in the mirror, waist size, and so on.
“The program worked. I look a lot bigger than I was”
This could be just due to increased glycogen, fat and water accumulation than just actual muscle growth.
“My weight hasn’t changed at all”
They might be losing fat and gaining muscle or just holding water.
This can only be measured with more sensitive & accurate body composition measures.
Related Articles
Anoop | Thu March 18, 2010
Thanks, Anatoly.
I think most of them know about science in training but our brains are programmed to believe in anecdotes and story telling than science. Scientific reasoning just evolved just recently. All we had back when we were living in caves to go by were anecdotes and experiences.
It wasn’t a clever strategy to wait for reasoned decisions and further evidence when believing something false was just harmless, but not believing could just be the end of the story for you. This is why so many people still believe in all that alternative treatments and bogus treatments. Unless you clearly understand the pitfalls of anecodotes and how you avoid them in science people are just paying lip service when they say “science is important”.
The NSCA conference I went to recently was just a good example of it. There was not a single person who had a question which really questioned the science behind all those presentations an we are talking about the best trainers in the country. One of the speakers said there was no definitive link, but his “18 years of anecdotal experience..” when I questioned the link.
And don’t get me wrong, overwhelming anecodotes should be considered. But they are always preliminary. They need to stand the rigors of scientific testing before they can be accepted.This one reason I clearly tend to state that “this is just a hypothesis”, this is theory or needs more evidence and so on. At times I am guilty of all those biases I wrote earlier.
great article anoop
I think the reason anecdotes work on so many of us is because they are powerful- they are much more personal and in your face
quoting anatoly “everyone will listen to you if you’re built like an oak”
people don’t want to listen to science cause its easier to listen to stories
Anoop | Mon March 22, 2010
Watch this you tube video for why we need evidence-based medicine.Pretty funny. In medicine, an evidence based approach is pretty much known to be THE way of treatment. It is now considered unethical to offers treatment which have no scientific support.
Erik Petersen | Fri April 29, 2011
As a side note, isn’t it funny how theories are sometimes much more than that. After watching all the objective info on 9/11 how can anyone say that it was the “evil-doers” doings? C’mon, we know who, what, why, just a bit too painful for the average person to acknowledge it so they remain on auto pilot, blinders (hell, shades) pulled tight!
