Reading A Psychology Study Carefully

Most of the distance between a psychological finding and its popular version is created by a small number of recurring moves. Checking for them takes a few minutes and does not require statistical training.

Who was measured

Start with the sample: how many, recruited from where, and how similar to the population the headline describes. A great deal of the literature rests on convenience samples drawn from a narrow slice of the world, which does not invalidate the work but does bound the claim. Small samples also produce unstable estimates, so a striking result from a small sample is a reason to wait rather than a reason to update.

What was measured

Then find the actual measure. Constructs with familiar names are operationalised in particular ways, and the operationalisation is often much narrower than the name. A study of wellbeing may be a study of answers to five questions on one afternoon. That is legitimate and worth knowing about; it is not the same as the word in the title.

What was compared

Next, the design. Was anything manipulated, or were two things observed together? Observation supports statements about association only. If something was manipulated, what was the comparison condition, and did it match the treatment on everything except the variable of interest?

How large, not just whether

A result reported as significant means the pattern is unlikely under a specific null assumption. It does not mean the effect is large or important. Look for the effect size and, where given, the interval around it. A wide interval that comes close to zero is telling you the study did not settle the question.

How it was reported

Finally, compare the abstract's language with the discussion's, and the discussion's with the press summary's. Hedges tend to disappear at each step, and correlations tend to become causes somewhere between the paper and the headline. Reading backwards from the headline to the method is usually the fastest way to find where that happened.

Permalink: /notes/reading-a-psychology-study-carefully