How to Check the Quality of a Research Article

November 25 2016

This is one of the most important aspects of a research study, yet the most overlooked or ignored by researchers and readers. If you read studies or believe in science, keep reading:

Why is study quality important?

Truth: A scientific study is conducted to find the truth. Although we will never know for sure if we have found the truth, we know that the better the study methods, the closer we are to the truth.

how to assess quality of a research study

Photo Credit: Evandro Inetti/ZUMA and Tischenko Irina/Shutterstock

Bias: In scientific terminology, any deviation or distortion from this underlying truth is called bias. When you have a high risk of bias or deviation, we can never be confident or certain of the results.
Randomized Controlled Trial: The very reason why RCT’s or randomized controlled trials are considered to be the gold standard is because of the low risk of bias inherent to these types of designs. Rightly so, only RCT’s can show cause and effect - or if X causes Y.

However, if the study does not adhere to appropriate methods to minimize bias, even an RCT is only as good as an opinion piece in Muscle & Fitness. Or maybe I should just say an article in M & F.

How do you assess quality?

There are three major methodological characteristics that you should look for to assess the quality of an RCT.

Randomization: Randomization simply means that every participant had an equal chance to fall in either of the two groups. Randomization ensures both groups are balanced before the start of the study. This is the most critical aspect of an RCT. If they did not specify this in the study, you can safely ignore the study.
Concealed Allocation: If a randomization sequence is generated, the next question is how were the subjects allocated according to the sequence. The allocation should be performed such a way that the study personnel have no clue who goes into which group or the forthcoming assignment.
Blinding: In double blind, the participants are not aware if they are taking the drug or the placebo. Also, the testers are not aware which group the participant belongs to. Hence the name double-blind. In exercise study, participants obviously cannot be blinded, but testers can and should be blinded (called single-blinded).

There are other ones like Intention-to-Treat ( ITT) and such. But the above mentioned ones have shown in studies to over exaggerate the benefits by almost 40%. That means, if a study lacked concealed allocation, a strength increase up to 40% could be just due to bias.

What about my studies?

RCT’s: During my Ph.D, I did 3 RCT’s. Except for my first study which had no concealed allocation, all of them were blinded, randomized, and maintained concealed allocation. In fact, I explicitly stated the lack of concealment in my first study. My last one had ITT too. Being a student, it is hard to dictate the study conduct, but I am proud that I managed to do them. What I have come to realize is that unless you run a study or two of your own, you will never know the importance of these quality controls in research.
Example: Let’s take the example of blinding: Unless your outcomes measure is death or something very objective, almost all measures are subject to bias. For example, one of the popular tests to measure physical function is Get up and Go test. For this test, they have to get up from a chair and walk around a cone a few meters away and sit back as fast as they can. They get two trials usually. I can easily make them beat the their first trial by simply saying ” now try to beat the time for the first trial”. If It didn’t say, they generally don’t. Or when I measure strength, I can easily give the group which I want to improve 2 more attempts to increase their 1RM. Some of the other measures I could have done to reduce bias: trial pre-registration, and blinding of the data analysis. Compared to other trials, trials involving exercise are extremely hard since you are supposed to train subjects 3-4 days per week for 16-20 weeks. So I admit that the logistics of ensuring blinded trainers and testers are easier said than done. But that’s no excuse.

Nobody is saying researchers cheat or commit fraud, but these things happen subconsciously even if you have the best intentions. That includes myself as well. I still cannot believe journals, even top journals, accepting papers which had no blinding of testers.

Why there is a replication crisis in science

Replication: A scientific study is conducted based on the inherent assumption that the results will be the same if we repeat the exact study. What is the point of doing a study if we cannot repeat the results, right? Guess what, we are realizing that we just cannot replicate majority of our studies. Some popular examples:
- In 2012, 270 scientists tried to replicate 100 of the psychology studies published in the top 3 journals (Open Science Collaboration). And how many could they replicate? 39 out of 100. Now imagine replicating the studies in the bottom-tier journals?
- What about medical science? Amgen cancer research tried to replicate 53 ‘landmark studies’ in cancer biology. They replicated 3. Bayer tried the same and could not replicate 65% of them.
Lack of Quality: Why this crisis? There are number of reasons raised: small sample size studies, problems in duplicating the methods of the original study, conflict of interest, looking at multiple outcomes and reporting the significant ones, suppressing negative results and so forth. As many have said before, a large reason why there is a crisis is simply due to the lack of emphasis on the methodological quality of the studies. Unfortunately, the emphasis is purely on statistical significance or lack of it. In fact, 96% of the articles indexed in Medline from 1981 show a statistical significant result. That means, out of 1000 hypothesis scientists got almost 960 of them right!
Incentive for Science: In my opinion, the root cause for the lack of quality goes back to how science is rewarded.
- Journals get more citations and readership with ‘significant’ findings.
- Researchers’ chances of getting grants improve with significant findings. Mind you, for some professors and researches in big universities, salary and staying in job is dependent on if they can bring in grants.How about that for pressure?
- Graduate students are praised for their productivity. I have been called productive a number of times (a term which I dislike). I have never met anyone who spoke a good word about the quality of my studies and reporting nor emphaise the importance of quality. The more studies, the better your CV looks. So the last thing a student is worried is about is his/her study quality.
- Rankings/Impact factor are based around citations which goes back to ‘significant’ findings
- No reward for replications. No journal cares about publishing replications. It is not hot news.
Unless we change how science is rewarded, we may not change anything I am afraid. A good study can take time ( maybe years), but is well worth the time. Thank you for all the likes when I post my studies on Facebook, but I am sorry to say that any study can be published these days. There are 100’s of journals in each field.
Reviews: Reviews of scientific research perform a very important job. A review should serve as the Robin Hood for readers/practitioners in that field by rigorously analyzing and exposing the flaws that the peer review overlooked. Critical analysis is THE most crucial aspect of an evidence-based approach. And it’s especially important in a new field like exercise science, where journals hardly care about quality. I have done it too, but just explaining the study and reporting nice-looking graphs is not a review - it is just a study ‘summary’.

Conclusions

Researchers should focus on improving the quality of study conduct and reporting.
Graduate students should be educated on the importance of study quality and not on ‘productivity’.
Ranking and reward structures should be built around study quality rather than ‘significance’.
Readers should look for these quality criteria first, and then look at results.
One study can never be definite, no matter who did it or where it got published.
Remember, all this above talk about replication, quality, randomization, methodology… is all about finding that elusive truth. And the best method known to man to find the truth is SCIENCE. So none of what I wrote undermines science or scientists; it is just my honest appraisal of where and how we can improve science.

Thank you for reading Exercise Biology.