5 Simple Rules for Good Research

December 30 2021

Here are five simple rules for good research from a recent article titled, “Ten simple rules in good research practice for early career researchers”.

An important topic for both researchers and research enthusiasts. Without any further ado, here we go:

1. Register your outcomes & analysis plan:

Pre-register: This means writing the outcomes/tests and statistical analysis plan, date stamping, and uploading to a public repository well before the data analysis.
Pre-specifying your plan or registering helps prevent p-hacking: It refers to selecting outcomes or analysis that gives you statistically significant results (P value < 0.05). Or cherry picking favorable results!A simple example to illustrate the play of chance: Two archers: Archer A & B hits the bull’s eye! Who is the accurate archer? Of course, both of them.

Example: Now what if I tell you that Archer B only got 3 attempts, while Archer B gets 20 or maybe 50 more attempts. Now you clearly know who is accurate. And this is exactly why you should pre-register our outcomes & analysis plan. The more outcomes and analysis, the more likely to get a significant p-value simply by chance.

A few caveats:

If you are doing a pilot or exploratory analysis, you do not have to pre-register. BUT results will be exploratory at best.
In certain cases, if you report all the outcomes and perform a standard analysis, it should be fine. For example, if you investigate body composition changes and if you report muscle mass/fat free and body fat, you have reported all the standard outcomes there is to report!

2. Get statistical expertise

Get a statistician: Great studies will collaborate/consult with experts in their areas, hence the reason they end up great! For example, a study involving weight loss will include dietician or a nutritionist, and of course a few researchers who specialize in weight-loss interventions.

A statistician, however, is required for every study. Every study involves sample size calculation and statistical analysis. Statistics are complex. The more I learn about stats, the less confident I am doing them on my own. And yes sir, this step should come prior to registering your analysis.

3. Reduce sources of bias

Eliminate biases: When you finally conduct the study, the most important point is to reduce or if possible, eliminate all sources of bias.

Below are some of the most common biases in a randomized controlled trial (RCT’s). In my experience, this is one area that many, including researchers, are either unaware or just ignore. Here is an article that I wrote 6 years back about this issue (How to check the quality of a research article) .

Randomization & concealed allocation: You have to randomize participants into groups. Also, conceal it well so that neither you or the participant can predict and alter the sequence. For example, imagine I am investigating a new drug for COVID and a sickly person gets randomized to the drug group. I might be very tempted to move him to the placebo group because I don’t want him dying in the drug group and make my lucrative drug look bad! And if I did that, I break the randomization. In my personal experience, I never felt tempted to move participants nor participants ever requested to be put in a certain group. It is probably because none of my studies had a control group or/and exercise studies have little at stake. But this could change if I had a pet hypothesis or financial gains at stake.
Intention to treat analysis (ITT): You have to analyze all the participants in the study. For example, if you exclude participants from the analysis for adherence, it is not intention-to-treat analysis anymore. If you exclude participants for lack of adherence, often seen in exercise & nutrition studies, the study is not ITT anymore and high risk of bias.
Blinded outcome assessors: You have to make sure that people who test participants at the end of the study are not aware of which group the participant belongs to. This becomes even more important if the test is subjective. For ex, test of physical function in older adults are assessed using a stopwatch and can be easily influenced. I can easily encourage them or stop the time slightly ‘quicker’ in the groups that I want to improve! But objective measures like DEXA or death testers cannot influence the outcome. (All my RCT’s used blinded assessors)

So, the bottom line is if any of these are not controlled for, then your study ends up with “high risk of bias”. Or neither you nor the study author can be ever sure about his or her results! Also, remember these are even important in animal studies. And even pre-registering can’t help you with these biases.

Example: The most cited, recent Cochrane review on Ivermectin concluded that we are uncertain if the drug helps to treat COVID. And guess what? The main reason is that they ended up excluding trials because of ‘high risk of bias’ and was only left with 2 studies. Moreover, the 2 studies that were included still had “some concerns” which ended up downgrading the evidence to ‘very low certainity’ (means we don’t know if it helps or not!).

In summary, a drug that could have saved millions in a pandemic, but we are very uncertain because of the ‘high risk of bias’ in studies. Or a harsher take: People died in these research studies, but their deaths contributed to nothing!! I hope this tells you something about the importance of controlling for biases.

4. Follow reporting guidelines

Use reporting gudelines:There are several studies that actually did everything right, but didn’t bother reporting in the paper. Consequently, they all end up as “some concerns” or “high risk of bias”.

Reporting guideline have been developed for each study design to guide you on what to include in your paper. In my experience, reporting is probably the least work compared to all other aspects of a research study. So you have no excuse whatsoever not to report well.

5. Make data open

Share data: Upload your data once you publish the results. Or at least have it available upon request. This will enable reviewers (or the public) to check the veracity of your findings. For example, the recent retractions in NEJM & Lancet were possible because the reviewers requested the data, but the author declined, leading to further questions. Mind you, fraud is very rare in research, and most errors are honest mistakes.

A good question is if these are so critical, why aren’t these emphasized in research? In my opinion, first is the lack of knowledge on how to implement these: You either have to work with mentors/collaborators who are proficient in these or be very well read or/and up to date in research methodology. And all these takes a lot of time, effort, and even money. Second, there is little incentive for researchers to do these since you are ranked/promoted/hired/judged based on the ‘number’ of publications and not on how well you did those studies. So if you focus on quality, you will be spending lot of time reading and collaborating, which will ultimately stifle your ‘productivity’.

Conclusions

If you are a Ph.D student (or a researcher), are you sticking to these? If not, please start.
If you read research to make important health/performance decisions, look for these simple rules to identify trustworthy research.

Wishing you all a happy and healthy new year!

Ten simple rules in good research practice for early career researchers