What to do when the crops are failing because of a drought? Why, we persuade the Gods to send rain of course! I'll let the fourth Roman Emperor, Claudius, explain:
 |
Derek Jacobi stuttering away as
Claudius in the TV series I Claudius
|
It sounds an awful lot as if Claudius is weighing in on the replication debate, coming down squarely on the side of replication critics, researchers who raise the specter of hidden moderators as soon as a non-replication materializes. Obviously, when a replication attempt omits a component that is integral to the original study (and was explicitly mentioned in the original paper), omission of that component borders on scientific malpractice. But hidden moderators are only invoked after the fact--they are "hidden" after all and so could by definition not have been omitted. Hidden moderators are slight mistakes or imperfections in the ritual that are only detected when the ritual does not produce the desired outcome. As Claudius would have us believe, if the ritual is performed correctly, then rain always follows. Similarly, if there are no hidden moderators, then the effect will always occur, so if the effect does not occur, there must have been a hidden moderator.**
And of course nobody bothers to look for small errors in the ritual when it is raining cats and dogs, or for hidden moderators when p<.05
I call this the Dripping Stone Fallacy.
Reviewers (and readers) of scientific manuscripts fall prey to a mild(er) version of the Dripping Stone Fallacy. They scrutinize the method and results sections of a paper if they disagree with its conclusions and tend to give these same sections a more cursory treatment if they agree with the conclusions. Someone surely must have investigated this already. If not, it would be rather straightforward to design an experiment and test the hypothesis. One could measure the amount of time spent reading the method section and memory for it in subjects who are known to agree or disagree with the conclusions of an empirical study.
Even the greatest minds fall prey to the Dripping Stone Fallacy. As Raymond Nickerson describes: Louis Pasteur refused to accept or publish results of his experiments that seemed to tell against his position that life did not generate spontaneously, being sufficiently convinced of his hypothesis to consider any experiment that produced counterindicative evidence to be necessarily flawed.
Confirmation bias comes in many guises and the Dripping Stone Fallacy is one of them. It makes a frequent appearance in the replication debate. Granted, the Dripping Stone Fallacy didn't prevent the Romans from conquering half the world but it is likely to be more debilitating to the replication debate.
Footnotes
* Robert Graves, Claudius the God, Penguin Books, 2006, p. 172.
** This is and informal fallacy; it is formally correct (modus tollens) but is based on a false premise.
"Someone surely must have investigated this already." Mahoney's 1977 'Publication prejudices' comes pretty close: http://link.springer.com/article/10.1007/BF01173636 (paywall but googleable)
BeantwoordenVerwijderenI don't think there is anything wrong with confirmation bias as such. It serves a useful function in science, strict falsificationism is crippling, as Lakatos showed. There would be little progress in science if any experiment, however inept, could bring down any theory, however promising. Pasteur was right to stick to his hypothesis. I think 'hidden moderators' is a perfectly good argument, as long as you accept the responsibility to find out what they are. The real problem is the idea that any single study can tell us very much about how good a theory is
BeantwoordenVerwijderenAn excellent reply.
VerwijderenAlso the classic Lord, Ross, & Lepper (1979).
BeantwoordenVerwijderenI think that if hidden moderators are really such a threat in our experimental designs (as some seem to argue), then we might as well call in to question all published results that are out there. And I am not just talking about failures to replicate, but also positive findings! If hidden moderators are out there, lurking in the dark, waiting to strike and distort, then why should studies reporting positive findings be spared? Maybe we just need some kind of standard moderator disclosure statement to be added as a footnote in all future published work. I propose something like this:
BeantwoordenVerwijderen"Please note that the results from the present experiment might have been distorted by numerous hidden moderators (e.g., lunar cycle, leprechauns, smelly feet). The actual effect might not exist at all, or the effect might even be in the direction opposite to the one reported here. This is the standard moderator disclosure statement endorsed by the centre for open science. I include in every manuscript."