How To Read a Study - The Pyramid of Outcomes
July 16 2011
The outcome measured in a study is the most important aspect of a study. But this aspect is the least emphasized when people try to interpret studies. So which outcomes are the most beneficial according to the pyramid?
What does the pyramid say?
Base of the pyramid: The base of the pyramid include studies with outcomes which do not have a direct impact on you. These are called surrogate outcomes. Examples for surrogate measures are protein synthesis, EMG analysis, X-rays, in-vitro models, cholesterol and so forth.
These measures are ‘stand in’ or represent meaningful outcomes higher up in the pyramid. For example, we think that protein synthesis is a stand in for muscle growth. We think x-ray is a stand in for pain outcomes. We think cholesterol levels are a stand in for deaths due to heart disease.
Top of the pyramid: As we go up the pyramid, we get to the patient or meaningful outcomes which are the most important and has a direct effect on you. Examples of these are muscle strength, muscle growth, fat loss, death, pain and so forth.
As we go up the pyramid, the importance of the study and the outcome goes up too.
What about some examples about surrogate measures?
Muscle growth: The most popular surrogate measure for muscle growth is protein synthesis. What people care about is muscle growth. We are not really sure if an increase in protein synthesis will lead to an increase in muscle growth. It may - or it may not. The molecular makers like AKT, MAPK, mTOR are surrogate measures for muscle growth which are way below the pyramid.
Even if synthesis goes up, protein breakdown could go up too. Or the fatigue generated can go up and cause the person to lower the weights or reduce their training. All these can affect muscle growth in a negative manner.
Performance: One of the usual performance measures for an athlete is the increase in squat strength. But can the increase in squats increase how many goals you scored, decrease the time taken to run or make you throw or hit longer? There are a host of factors that can influence performance, and squat strength may or may not be one among them.
Bone density: Bone density is a classic example of surrogate measure. We do not care about bone density. What people care about is does it lower the chance of a hip fracture, the most disabling kind of fracture.
People could fall due to poor vision, lack of strength, balance, their environment and so forth. So an increase in bone density may not always translate to fewer fractures.
Cholesterol: Drug companies usually measure cholesterol levels to show the efficacy of a certain drug. But the question is do they make the person live longer or do they feel better?
Clofibrate, a drug for cholesterol, lowered cholesterol really well, but inexplicably increased the number of deaths. A recent example is the popular diabetic drug -Avandia - for lowering blood glucose. The drug worked really well for lowering glucose levels in diabetes, but people suffered from heart attacks, stroke and even deaths. These are good examples of drugs that got approved from surrogate end point studies.
Core strength: In the fitness field, there are lot of people who quote studies showing increase in EMG, core strength, pig spine models, shear /compressive stress values as evidence for prescribing or not prescribing certain exercises for the core. These are just surrogate end points. Until they can show the intervention can increase performance or prevent injury or decrease low back pain, it is just a beautiful hypothesis.
Exceptions: There are cases where just lowering the symptoms might be really important for you. For example, people in end-stage cancer treatments, improving the quality of life or the symptoms can be very important. There are also surrogate measures which are highly correlated with the end point.
Why do people use surrogate measures?
- Cheap & quick: These surrogate measures only need a small sample size and are relatively short-term studies. For example, to study mortality you will need a study with 100’s of participants for a period 5-10 years.
- Preliminary: Most studies use these designs as preliminary evidence to justify long-term studies with meaningful outcomes.
Practical Application
- The outcome measure is the most important aspect of a study. Next time, when you read a study or hear someone quoting a study, look to see if it is a surrogate measure or not.
- If it is a surrogate measure, be skeptical about the conclusions.
- If you know about other surrogate measures used to prove a hypothesis, please comment below
Related Articles
| Sun July 17, 2011
Anoop, I’m having trouble understanding. I have osteoporosis, so studies that show X food improves bone density are of interest to me. In fact, I will start including more of that food in my diet, particularly if there are several studies which show the same outcome.
Other osteoporosis recommendations are to avoid exercises that involve flexing the spine and that metabolic acidosis results in poor BMD. (Can I assume these recommendations come from studies?)
A case of surrogate measures highly correlated with the end point (QOL)?
Anoop | Sun July 17, 2011
Hi Kathleen,
Thanks for the comment. Good question too.
Bone mineral density is a surrogate measure for fracture risk. An increase in bone mineral density do not change how you feel, function or survives. So it is not a meaningful clinical outcome. What is meaningful for you is the risk of fractures which is what you and everyone cares about. You get hurt when you get fractures and you are disabled.
Have studies shown that BMD is highly correlated with fracture risk, and increasing bone density decreases fractures or increasing BMD plays a substantial role in lowering fractures? If we can show this, we can say BMD is a good surrogate measure for the risk of fractures. If not, we are just hypothesizing that it may help.
There are many drugs in the market that have shown to increase bone density, but few have been shown to reduce fractures.
Another common example is tumor suppression. Drug companies will show tumor shrinkage with a particular drug. But what we care about is if the person will live longer or not. The tumor shrinkage may or may not increase survival. The tumor may start growing back aggressively after a period of shrinkage, it may not be the size but the spread of the tumor to other parts that is more important, and so forth. The same can be said some of screening for breast cancer and such.
My point is do consider surrogate measures, but be skeptical about it. There are lots of examples in the past where we were convinced of certain surrogate measures and got it completely wrong.
| Sun July 17, 2011
Good article. I hope many people will read and apply.
The part about bone density interested me too. Is bone density the same as bone ‘mineral’ density? Are, for instance, also amino acids involved in measuring BMD?
PS: One extra surrogate measure: Testosterone and muscle growth.
It’s hard to believe how many things cause Testosterone to raise a little and how often these little raises are directly linked to supposed muscle growth without considering all the other hormones, enzyms, receptors and so on that are involved in muscle growth.
Anoop | Sun July 17, 2011
Hi Fulldepelex,
Yes, they are the same.
That’s a really good one. High reps and short rest period increases testosterone and GH, so it increases muscle growth. That’s a classic example of a surrogate measure.
| Mon July 18, 2011
Hence the reason so many of us love your site Anoop; you make the confusing information easy to understand and can give us the “meaningful outcomes”.
Thanks again for keeping the site going,
Steve
| Tue July 19, 2011
Boom! Dropping some knowledge bombz! You should post more often.
Anoop | Wed July 20, 2011
Thanks Steve and Mark for the nice comments.
I just think this is a really important topic which not many people write or talk about. If people knew this, we could have eliminated a lot of debates in the fitness and the health field.
If you haven’t read Mark Young’s website check it out. He is one of the very few fitness professionals who can understand and read research.
Thanks again!
| Mon July 25, 2011
Good stuff, Anoop. FYI, the research I’ve seen shows a high correlation between BMD and risk of fracture. Kanis et al. (2005) performed a large-scale study that found a ~threefold increase in risk ratio for each standard deviation decrease in BMD. The confidence intervals were pretty narrow, strengthening the validity of results.
Brad
Johnell O, Kanis JA, Oden A, Johansson H, De Laet C, Delmas P, Eisman JA, Fujiwara S, Kroger H, Mellstrom D, Meunier PJ, Melton LJ 3rd, O’Neill T, Pols H, Reeve J, Silman A, Tenenhouse A. Predictive value of BMD for hip and other fractures. J Bone Miner Res. 2005 Jul;20(7):1185-94.
Anoop | Tue July 26, 2011
Hi Brad,
Thanks for the good study! You are right. It looks like there is quite a few which shows a consistent relationship with BMD and fracture risk. From what I have read, it is still not in the same league as blood pressure for cardiovascular events and forced expiratory volume in COPD.
There is your answer Kathleen. If you like some solid science based info, check out Brad’s blog at http://www.workout911.com
>
