Another worthy addition to our coverage of all things human fallibility-related from Jonah Lehrer, published a few weeks ago in The New Yorker. Specifically, on the increasingly complicated nature of epistemology in the modern scientific community. It would appear that “hard” scientists run into more problems replicating significant results than they would care to admit. And the phenomenon, known as the Decline Effect, reveals some disturbing (though unsurprising) truths about the scientific method. It all adds up to a pretty serious reality check for all the empiricists out there. Read the whole thing here, or for the more relevant sections, see below. Just be sure not to glaze over before Lehrer’s stunning conclusion:
Before the effectiveness of a drug can be confirmed, it must be tested and tested again. Different scientists in different labs need to repeat the protocols and publish their results. The test of replicability, as it’s known, is the foundation of modern research. Replicability is how the community enforces itself. It’s a safeguard for the creep of subjectivity. Most of the time, scientists know what results they want, and that can influence the results they get. The premise of replicability is that the scientific community can correct for these flaws… But now all sorts of well-established, multiply confirmed findings have started to look increasingly uncertain. It’s as if our facts were losing their truth: claims that have been enshrined in textbooks are suddenly unprovable. This phenomenon doesn’t yet have an official name, but it’s occurring across a wide range of fields, from psychology to ecology.
—————
For many scientists, the effect is especially troubling because of what it exposes about the scientific process. If replication is what separates the rigor of science from the squishiness of pseudoscience, where do we put all these rigorously validated findings that can no longer be proved? Which results should we believe? Francis Bacon, the early-modern philosopher and pioneer of the scientific method, once declared that experiments were essential, because they allowed us to “put nature to the question.” But it appears that nature often gives us different answers.
—————
[Noted Australian biologist Michael] Jennions, similarly, argues that the decline effect is largely a product of publication bias, or the tendency of scientists and scientific journals to prefer positive data over null results, which is what happens when no effect is found. The bias was first identified by the statistician Theodore Sterling, in 1959, after he noticed that ninety-seven per cent of all published psychological studies with statistically significant data found the effect they were looking for. A “significant” result is defined as any data point that would be produced by chance less than five per cent of the time. This ubiquitous test was invented in 1922 by the English mathematician Ronald Fisher, who picked five per cent as the boundary line, somewhat arbitrarily, because it made pencil and slide-rule calculations easier. Sterling saw that if ninety-seven per cent of psychology studies were proving their hypotheses, either psychologists were extraordinarily lucky or they published only the outcomes of successful experiments. In recent years, publication bias has mostly been seen as a problem for clinical trials, since pharmaceutical companies are less interested in publishing results that aren’t favorable. But it’s becoming increasingly clear that publication bias also produces major distortions in fields without large corporate incentives, such as psychology and ecology.
—————
“Once I realized that selective reporting is everywhere in science, I got quite depressed,” [biologist Richard] Palmer told me. “As a researcher, you’re always aware that there might be some nonrandom patterns, but I had no idea how widespread it is.” In a recent review article, Palmer summarized the impact of selective reporting on his field: “We cannot escape the troubling conclusion that some—perhaps many—cherished generalities are at best exaggerated in their biological significance and at worst a collective illusion nurtured by strong a-priori beliefs often repeated.”
—————
“A lot of scientific measurement is really hard,” [biologist Leigh] Simmons told me. “If you’re talking about fluctuating asymmetry, then it’s a matter of minuscule differences between the right and left sides of an animal. It’s millimetres of a tail feather. And so maybe a researcher knows that he’s measuring a good male”—an animal that has successfully mated—“and he knows that it’s supposed to be symmetrical. Well, that act of measurement is going to be vulnerable to all sorts of perception biases. That’s not a cynical statement. That’s just the way human beings work.”
—————
The problem of selective reporting is rooted in a fundamental cognitive flaw, which is that we like proving ourselves right and hate being wrong. “It feels good to validate a hypothesis,” Ioannidis said. “It feels even better when you’ve got a financial interest in the idea or your career depends upon it. And that’s why, even after a claim has been systematically disproven”—he cites, for instance, the early work on hormone replacement therapy, or claims involving various vitamins—“you still see some stubborn researchers citing the first few studies that show a strong effect. They really want to believe that it’s true.”
—————
Such anomalies demonstrate the slipperiness of empiricism. Although many scientific ideas generate conflicting results and suffer from falling effect sizes, they continue to get cited in the textbooks and drive standard medical practice. Why? Because these ideas seem true. Because they make sense. Because we can’t bear to let them go. And this is why the decline effect is so troubling. Not because it reveals the human fallibility of science, in which data are tweaked and beliefs shape perceptions… And not because it reveals that many of our most exciting theories are fleeting fads and will soon be rejected… The decline effect is troubling because it reminds us how difficult it is to prove anything. We like to pretend that our experiments define the truth for us. But that’s often not the case. Just because an idea is true doesn’t mean it can be proved. And just because an idea can be proved doesn’t mean it’s true. When the experiments are done, we still have to choose what to believe.
COMMENTS
6 responses to “Irreplicability, Publication Bias, The Decline Effect and The Scientific Method”
Leave a Reply
Great Scott!!! That's heavy, Doc…
A friend of mine who works in chemistry once told me that if you ever want to work in science you have to recognize that 90% of the work is literally working through failure. You do things wrong, you get null results, you rethink what you thought you were looking for, and be willing to concede that most of the time you followed a blind alley. She said that if you're doing your science well you can still learn a lot even from your failures.
This article is shocking due to its honesty,not its conclusions. All trial lawyers know that, for a price, one can hire a highly credentialed expert to come to just about any given desired conclusion, under oath, applying accepted scientific methodologies. Many complex cases, and particularly products liability cases, come down to a "battle of experts" in which a jury of generally ignorant "peers" decides with which of the highly paid "experts" they, in their collective ignorance, agree. "Follow the money" may be one of a very few universal and empirically verifiable, truths. But "the money" is usually grant money, not expert witness fees. The beauty of the legal profession is that it knows full well what it is up to and, except for a few "public interest" "non-profit" types, still has a firm understanding of the effects of original sin on even the "scientific method."
Wow, Dave, this was one of the most fascinating things I have read in a while.
I admit to having some skepticism about the author of the piece since he treats ESP, precognition, and other "phenomena" studied by psi researchers as real. The problem he seems to think is in finding a mechanism to explain them, not whether they actually exist in the first place.
Still, the piece on the whole was fascinating and raises important questions. It'll be interesting to see what kind of reaction it gets.
I especially enjoyed the title of one of the cited papers: “Why Most Published Research Findings Are False.”
Now you're talking about something right up my alley. It deserves to be mentioned which scientific fields are especially susceptible to this problem: psychology, ecology, biology, and the like. Scientists face an epistemological challenge in these fields that they have yet to really come to grips with. They have yet to really develop and adhere to a theory of complex phenomena. Basically, scientists in these fields are trying to emulate physics, which studies simple objects in order to obtain rigorous results. Living things are so complex that the methods of physics simply can't apply to them, even in principle (or so I would argue). The alternative, I propose, would be a more "humble" epistemology, one in which we can get at general principles explaining complex phenomena (which can have beneficial applications) without presuming to obtain strict "cause-effect" relationships.
"When the experiments are done, we still have to choose what to believe." The scary question lurking behind this is, how much are we willing to risk? When you're talking about these kinds of sciences, especially medical science, the most important question is, how much damage are you willing to do in an attempt to care for living things, based on general trends which may not prove to be universally true?
As a Ph.D. student in psychology, all I can say is, "sigh."