Reading a Study
Whether a study may claim that one thing causes another depends on whether it merely observed the world or actively assigned a treatment, and on which competing explanations it ruled out. · 13 min
Every week a headline tells you that some habit lengthens your life or some food shortens it. Behind each is a study, and studies are not all built to answer the same question. Some only watch the world as it already is; others step in and change something on purpose. That single difference — did the researchers merely observe, or did they act — decides whether the study is entitled to say one thing causes another. This lesson gives you the test to apply before you believe the headline.
Guess before you learn
Researchers follow thousands of adults for a decade and find that regular coffee drinkers live longer than non-drinkers. The study never assigned anyone to drink coffee. What may it fairly claim?
Because no one was assigned to drink coffee, the drinkers and non-drinkers may differ in dozens of ways — wealth, exercise, stress, other habits. Any of those could be the real cause. The study honestly shows an association; it cannot, by its design, promote that to cause. If you reached for the first choice, you are in good company — and this lesson is about resisting it.
9–12
3–5
There are two big kinds of study. In an observational study you only watch: you compare people who already do a thing with people who do not. In an experiment you act: you decide who gets the treatment, ideally by a coin flip, so the groups start out alike. Only the experiment can really show that the treatment is what made the difference.
6–8
An observational study measures variables without intervening: it records who already drinks coffee, smokes, or exercises, and compares outcomes. Its weakness is confounding — the groups differ in other ways, so an association can never be pinned to one cause.
An experiment assigns the treatment, and a good one assigns it by random assignment — a coin flip decides who gets it. Randomizing makes the groups alike, on average, in every respect except the treatment, so a difference in outcome points to the treatment itself. That is why only a randomized experiment earns a causal claim.
9–12
The pivotal question is who decided the treatment. In an observational study, people (or nature) sorted themselves into groups, so the groups differ in countless measured and unmeasured ways — any of which may confound the result. In a randomized experiment, the researcher assigns the treatment by chance, and chance does not consult wealth, health, or motivation.
Randomization balances confounders you thought of and confounders you never imagined, which is its unique power. A control group that receives no treatment (or a placebo) supplies the comparison. Even then, ask what a good study still controlled for and what it could not: a clean experiment can establish cause within its sample yet still generalize poorly to other people.
K–2
To learn if a medicine works, do not just watch who takes it. Give it to some people and not others, chosen by chance. Then compare. Watching alone can fool you, because the people who chose it may be different.
Undergrad
Separate two validities. Internal validity asks whether the observed effect is caused by the treatment within the study; external validity asks whether it transfers beyond the sample. Confounding is a backdoor path from treatment to outcome through a common cause; observational designs must block it by measuring and adjusting for every confounder — an untestable assumption of no unmeasured confounding.
Randomization severs the backdoor paths in expectation, which is why the randomized controlled trial is the reference standard for internal validity. Blinding and placebo control guard against expectation effects. Where randomization is impossible, quasi-experimental designs — matching, instrumental variables, regression discontinuity, difference-in-differences — approximate it, each purchasing a causal claim with a different, explicit, and falsifiable assumption.
Postgrad
In the potential-outcomes model each unit has outcomes Y(1), Y(0) under treatment and control; the causal effect is their contrast, and the fundamental problem is that only one is ever observed. The naive difference in group means equals the average treatment effect plus a selection-bias term, which vanishes only under (conditional) ignorability: assignment independent of potential outcomes given covariates.
Physical randomization guarantees ignorability by construction, identifying the ATE without modeling assumptions; observational identification instead invokes ignorability as an assumption, or leverages exogenous variation — an instrument satisfying relevance and exclusion, a running-variable threshold, a parallel-trends counterfactual. Reading a study critically is auditing which assumption is doing the causal work and whether the design makes it credible.
random assignment
Deciding by chance — effectively a coin flip — which subjects receive the treatment and which do not. It makes the groups alike on average in every respect except the treatment, which is what lets an experiment support a causal claim.
Not all evidence sits at the same height. A single person's story is the weakest rung: it might be true, but it controls for nothing. An observational study comparing groups is stronger, because it uses real data at scale — yet confounding still limits it. A controlled experiment with a comparison group is stronger still. And a randomized controlled experiment, where chance assigns the treatment, sits at the top, because randomizing is the one move that neutralizes the confounders no one anticipated. When you read a claim, place it on this ladder before you weigh it.
Note
Want to practise separating a claim from its evidence in ordinary reading? The Atelier of Mind (the college's study-skills workshop) runs drills on exactly this.
Practice — new ink and old, interleaved
1.In one sentence: what is confounding, and which study design defends against it best?
Confounding is when the groups differ in some other variable that affects the outcome; a randomized experiment defends against it by balancing those variables through chance assignment.
How close were you? Grade yourself honestly — it sets your review date.
2.Match each correlation to the kind of explanation most likely behind it.
3.A study randomly assigns volunteers to a new exercise plan or to their usual routine, then compares fitness after twelve weeks. This design is:
4.A correlation of r = −0.1 between two variables tells you:
5.Three points have z-score products of 0.8, 0.2, and 0.5. With n − 1 = 3, what is r?
6.Towns with more churches also have more bars. The best explanation is:
7.A website poll asks its own visitors whether they trust that website. The results will most likely be:
8.A scatterplot of hours studied (across) and test score (up) has points rising from lower left to upper right. What is its direction?
9.The same right-skewed wait times have a mean of 22 minutes and a median of 14. Which is the more honest headline figure for a typical wait?