How science really works, in 11 minutes
Why is it that health recommendations often seem to contradict each other? I’m going to explain to you why that happens. But first, let’s look at some examples. You may have heard studies say we should cut cholesterol out of our diets. But then later research suggests that actually no, dietary cholesterol isn’t bad for us after all. But wait, even newer research suggests that the truth is actually more complicated than either of those stories. You may also have heard that any amount of alcohol is
Bad for us. But later research suggested that actually one drink a day may be good for us. And then even newer research says actually maybe that’s not right. Maybe any alcohol at all is bad for us. This sort of issue isn’t just limited to food or nutrition. The same kind of problem happens with research into supplements. You may have heard for instance that vitamin C helps cure colds. Lionus Pauling a two-time Nobel Prize winner promoted this perspective
And products like emergency that come loaded with vitamin C have become popular. But then, oh wait, newer research says that vitamin C actually doesn’t work if we take it after a cold has begun. This kind of flip-flopping about the truth presents two big problems. First, it makes it challenging to decide how to live our lives. Should we try to avoid cholesterol in our diets or not? If we want to be as healthy as possible, should we avoid all alcohol or
Is a little alcohol fine and maybe even good for us? It’s easy to end up confused. The second problem is that when the public perceives scientific research as contradicting itself, they can easily end up not trusting science. Even worse, they may end up trusting YouTubers instead. I’m going to explain to you why this flip-flopping happens in research and how it can be avoided. I’ll also talk about the underlying drivers of it. Is it a problem with science
Itself or a problem caused by scientists or maybe just science communication? Or are none of those to blame? Suppose you want to tell if taking vitamin C supplements helps prevent colds. How would you do that? Well, suppose that a survey had already been conducted where people were asked whether they took vitamin C supplements over the past 12 months and they were also asked how many colds they’d gotten in that period. Figuring out if vitamin C supplements
Help prevent colds is then simple. All you have to do is look at this data to see whether those who were taking vitamin C supplements got fewer colds on average than those who weren’t. Right? I recommend that you pause this video now for a moment to see if you can explain what’s wrong with this approach. Why can’t you reliably tell if vitamin C helps prevent colds by checking whether those who are taking vitamin C over the
Past 12 months got fewer colds than those who weren’t? The general issue here is we can’t rule out the possibility of what are called confounders. We want to show that A, which in this case is taking the supplement, has a causal effect on B, which in this case is getting sick with colds. But all we’ve shown using this method is that A and B are associated or correlated, not that A causes B. When we prove that A and B are associated, there
Are three common possibilities. One, it could be that A causes B. In this case, that would mean that vitamin C prevents colds. Two, it could be that B causes A. In this case, that would mean that people who get more colds go take vitamin C more often, presumably because they have heard it helps with colds and are eager to stop them. Or three, it could be there’s some confounder, call it X, that causes both A and B. In this
Case, that confounder could be that healthy people are more likely to take vitamin C and they’re less likely to get colds due to their healthy behaviors. So, we’ve seen that showing that A and B are associated or correlated doesn’t prove that A causes B. But does that mean that associations are useless? Actually, they can be quite useful for two reasons. One, if no association is found, that is moderately strong evidence that A doesn’t cause B. So, if
We found no association between taking vitamin C and fewer colds, that suggests that vitamin C is not a promising way to prevent colds. And two, associations are a good way to get hypothesis. For instance, if vitamin C and getting fewer colds are found to be associated, that suggests a hypothesis to consider that perhaps taking vitamin C causes fewer colds. As Randall Monroe, creator of XKCD, put it, correlation doesn’t imply causation, but it does waggle its eyebrows suggestively and gesturively
While mouththing look over there. But if we find the two things are associated, we still need further evidence to tell which way the causality goes. Is it that A causes B, B causes A, or some other variable X causes both A and B? Associations can be very interesting to look at. They’re a starting point for research. They can help generate hypotheses and they can even help shoot hypotheses down. With this in mind, we gathered 1 million correlations about
Humans and made them publicly available. If you’re interested, you can search them here for free at personalitymap.io. For instance, you can use it to look at the correlation between anxiety and depression or to look up common traits of narcissists. Getting back to the subject at hand, if we can’t use associations to prove that taking vitamin C prevents colds, what should we do instead? Well, here’s an idea for an experiment to fix the problem. We can simply pick 10 people, find out how
Often they get colds, and then give them vitamin C for 12 months to see how many colds they get. If they have fewer colds than normal, that proves that vitamin C prevents colds, right? Unfortunately, that’s not right. What if those 12 months were simply a period when the cold season was less bad than normal? That would make it seem like vitamin C worked when in fact, most people, whether they took vitamin C or not, got fewer colds than they usually do. Or
What if there’s a placebo effect? Perhaps believing you’re getting treated for having a cold makes it so you have less stress, which causes you to get fewer colds. In that case, it would seem like the vitamin C was working. But actually, a sugar pill would work just as well. This time, instead of just picking 10 people to give vitamin C to, say 10 college students, we can also pick 10 people to monitor without giving them vitamin C, say 10 people from our
Nearby community who we will give a placebo sugar pill to instead. None of the participants will know whether they are given vitamin C or the sugar pill. Then for the next 12 months, we’ll monitor any colds in the two groups. Finally, we’ll compare them to see which group got fewer colds. This successfully solves the problem we had before of addressing changes that impact everybody. And it also handles the placebo effect. But unfortunately, it introduces another important problem.
Can you tell what it is? The issue is the 10 college students who we selected to give vitamin C to may differ in important ways from the 10 people in our community who we gave the sugar pills to. If the college students have stronger immune systems on account of being younger, they may get fewer colds than the other group. Not because vitamin C works for them, but simply because they’re younger. Thankfully, there’s a simple fix for this. Can you
Guess what it is? It’s one of the most powerful tools that studies use. Randomization. All we have to do to solve this problem is to randomize which people get the placebo and which get vitamin C. This prevents there from being any differences on average between the placebo group and the vitamin C group. But wait, we’re still not quite done. There’s still a big problem with this experiment. The last issue is that with just 10 people in each group, even
If vitamin C truly does help reduce colds, there’s a pretty good chance that just due to random noise, we’ll actually end up with more colds in the vitamin C group. The reason for that is that the number of colds we get each year fluctuates randomly. So just by chance, the 10 people getting vitamin C might happen to get more colds even if vitamin C is helpful. Instead of recruiting 20 people and randomizing 10 of them to get
The placebo and 10 to get vitamin C, we can recruit 400 people and randomize 200 of them to get the placebo and 200 to get vitamin C. With enough people in our study, then if vitamin C really does meaningfully reduce colds, we’ll be able to tell by comparing the average number of colds in the vitamin C group to the average number in the placebo group. We can even use statistics to calculate the probability of getting a difference in
The averages as big as we got if in fact vitamin C doesn’t work. This is known as the p value. Notice how in trying to answer the simple question, does vitamin C prevent colds? We had to correct error after error after error to converge on a study design that actually tells us if one thing causes another. The study design that we converged on is what’s known as a well-powered randomized control trial, sometimes abbreviated as an RCT. These studies are controlled,
Meaning that we don’t just give people vitamin C. We also have a control group that gets a placebo. It’s randomized, meaning that we don’t just manually pick some people to get the vitamin C and others to get the placebo. We randomize which one each person gets to avoid any systematic differences between the two groups. And it’s wellpowered, meaning that there are enough participants in the study to reliably detect the effect if vitamin C actually does work.
Wellpowered randomized control trials are powerful. Unlike surveys, studies with no control group, and studies with only a few participants, well-powered randomized control trials can reliably tell us if one thing causes another. This brings us back to the original question we set out to answer. Why does it seem like nutrition research and supplement research keep flip-flopping what they tell us? The biggest reason is that most studies don’t allow us to tell if one thing reliably causes another.
Unfortunately, when people see these weaker studies come out, they often prematurely jump to the conclusion that X causes Y. So, are scientists to blame for this? Often no, but sometimes yes. Many scientists are cautious in their claims, and they don’t make it seem like they found strong evidence for X causing Y, unless they really have. But there are some scientists who write their papers in a way to get more attention, which makes it seem like they’ve shown X
Causes Y, even when they really haven’t. The news media contributes a great deal to this problem by latching on to the latest study without understanding what it really did and didn’t show. And influencers, who are typically not scientifically trained, make matters worse by talking about very weak studies without understanding their limitations. These forces create bizarre situations, as depicted in this chart, where it seems like everything both prevents and causes cancer. A major reason why a chart like this shows so many
Contradictory studies is that many of those studies are not of the appropriate design to show whether something actually causes or prevents cancer. Fundamentally, the problem is not that science lacks the tools to figure out what works in health supplements and nutrition, but there are actually three problems happening simultaneously. First, lots of studies that aren’t capable of showing whether X causes Y are used as though they can answer questions like that. So, studies appear to contradict each other when in fact
The studies are just being misused. Second, the actual studies we need to figure out answers are often very timeconuming and very expensive to conduct. So lower quality studies get done instead. These don’t actually answer the questions we care about, but at least we can pump them out. Third, the world is actually very complex and human health is very complicated, which means that even the highest quality studies occasionally disagree for very good reasons. One wellpowered randomized control trial might correctly conclude
That a supplement is helpful in one population, whereas another well-conducted trial might conclude that supplement doesn’t help in another population. But both can be correct. The reason for the disagreement may simply be that the first population is deficient in something the supplement provided. The second population wasn’t deficient, so the supplement didn’t actually help. The world is a complicated place. I hope you found this discussion today informative. If you did, I hope you’ll take a moment now to
Subscribe and also hit the bell icon to get updates whenever we have new videos. You can also deep dive on many interesting topics on our website at clearerthinking.org.