“Remember that study that found that most psychology studies were wrong?” asks Slate writer Rachel Gross. Not even, “did you hear”, but “do you remember”. There is now no longer a question as to if this matter needs explained first. It’s already common knowledge. Everybody’s read this study. If you haven’t, you’re behind. Just play along and wing it. You know how you didn’t always do the reading assignment before class? Same deal. The study. You know the one. It was in the New York Times. They do studies now, apparently. Pardon me? The Times didn’t actually do the study? Oh. Who did? Are any of the authors cute and single, perchance?
-Observations: Girls on the left and right are unmarried (no wedding ring). Apple computer. Bitly link written down on a board, like how our cavemen ancestors used to do it. Also, the link is defunct (try it).
The actual “study” was done by an organization called the Open Science Collaboration (OSC), and involved dozens of authors and even more volunteers–some of whom were even credited for their work, which is not always the case in the business, as I can attest from firsthand knowledge. Dr. Nosek (man in photo), not to be confused with the late philosopher Dr. Robert Nozick, may have been the project lead, but he and his team of plucky young ladies did not actually do the whole of the work. And I say “study” because it was actually an analysis of the results of retesting nearly a hundred studies.
The real study may be interesting, but Slate doesn’t give a damn about the study, it cares about what we think the study says. That’s why she links to the Times, not PubMed, let alone another Slate article where they talk about it. Because that’s not the point. “Yeah, that study was wrong. That’s the conclusion of four researchers who recently interrogated the methods of that study, which itself interrogated the methods of 100 psychology studies to find that very few could be replicated. (Whoa.) Their damning commentary will be published Friday in the journal Science. (The scientific body that publishes the journal sent Slate an early copy.)” continues the author. The translation: “people who think that psychology has a validity problem are wrong”.
“Damning”? Really? The arguments that some of the methodologies used to reproduce the studies were flawed is an excellent one, but it was one of several used by Gilbert et al. And when you factor in the rebuttal by authors of the OSC that Slate appended to the article the next day, it hardly seems like insurmountable criticism. For the most part, both the original authors and the commentators are focusing on the statistics of reproduciblity. To what extent are results reproducible, how do we determine this, what does this mean, etc. What the OSC seems to be noticing is something that looks suspiciously similar to the “decline effect“, which is a tendency for some scientific claims to decline in acceptance over time. Originally coined to describe experiments testing paranormal abilities like ESP, it resurfaced in a 2010 New Yorker article about psychiatric medication. Like with the OSC findings, various explanations for statistical results are all well and good, but it misses the larger point: that the studies were bad to begin with. As one of my favorite writers put it in a fantastic piece: the decline effect is stupid. What the OSC, Gilbert and friends, and Slate and the rest of the media miss is “so what?” Reproducibility is an important aspect of science. But so is experimental design. If I reproduce a terribly designed study, that doesn’t corroborate the study; if I can’t reproduce a terribly designed study, that doesn’t necessarily tell me the study is bad.
Let’s take an example. I end up as a psychology professor because I have made very bad life choices, and I’m eager to churn out more publications so my university will up my job security. So I get all my students in Psych 101 to do some survey charting some indicators for personality types. I also ask them what they had for lunch. I cram the results through a stats program and mention everything that has a hit. Turns out the kids who had bologna sandwiches for lunch scored higher on the sociopathy scale. It’s statistically significant, too. I throw in an introduction about processed meats being all the norm in America today, and the media runs wild with it. By the end of the year, “diet influences personality” will be accepted as a thing, albeit one that “needs more further research”. So what happens when somebody reproduces the study? Well, the odds are low that it’ll work out again. Maybe there won’t be a relationship with those variables. Maybe the bologna sandwich folks will score high on anxiety this time. Maybe the sociopaths will have had Reubens this go-around. All of these statistical things can be debated. What will not be discussed is the fact that my experiment was stupid, because even if there were some causal link between diet and personality type, in no scenario would it be detectable with a survey I give to a few kids about what they had for lunch and how many people they’d kill with a trolley, or whatever.
While we get a brief flirtation with this idea in the Comments by Gilbert and associates, it’s lumped in as an argument in the debate about statistics and reproducibility. It should be it’s own argument, the lead argument, really. If the methodology doesn’t make sense, the results shouldn’t be considered. That the materials and methods do not seem to be easily accessible from the papers is unfortunate. That would be interesting to look at. Is the system even set up to deal with such a thing? Are retractions going to be issued? Probably not. What we get is at least 3 new papers. More information to wade through and no real filtering. The experts in the field aren’t auditing old work, they’re arguing over whether or not all of it is good or bad in summation, and they’re letting the media pick a side. When you look up the old studies they reviewed, are the journals going to link to the new study if it shows a contradiction in results? No. Publications are not in the science business, they are in the business business; their product is science. “We’ve got papers! You can cite ’em for free, but it’s $39.99 to read ’em.” Unless you ask the Russians.