One of my last actions as Editor-in-Chief of Acta Psychologica was to accept a manuscript for publication that is very timely given the current “crisis of confidence” in
psychology. One of the paper's key points is that it is crucial to distinguish
between hypothesis testing and exploratory research.
The paper chimes in with many critics of current practices in psychology when it asserts that [i]t is
essential that these hypotheses have been precisely formulated and that the
details of the testing procedure (which should be as objective as possible)
have been registered in advance. Several journals, such as Cortex,
as well as special
issues of journals already require authors to preregister their submissions
and the Open Science Framework offers an
extremely user-friendly platform to do just this.
The paper makes a clear distinction between hypothesis-generating and
hypothesis-testing research and argues that researchers regularly conflate the
two, passing off exploratory research as confirmatory—a very apt description of
current research practices. In exploring the data, the paper continues, researchers try to extract from the material what is in it
but necessarily also what is accidentally in it. And thus the
researcher proceeds by trying and
selecting, whereby the selection is based on whether it promises to produce
interesting (i.e., significant) results. By operating in this fashion the researcher is capitalizing
on coincidences. The paper then goes on to explain what the problem with
this practice is by using the example of rolling a die (hmm, where have we seen
this
example before?).
The paper ends with a rather stern conclusion: If the processing of empirically obtained
material has in any way an “exploratory character”, i.e. if the attempt to let
the material speak leads to ad hoc decisions in terms of processing, as
described above, then this precludes the exact interpretability of possible
outcomes of statistical tests. This conclusion resonates well with comments made
by other researchers in the current debate. Perhaps it is not surprising then that the paper modestly
acknowledges: this conclusion is not new.
Adriaan Dingemans de Groot (1914-2006) |
So if the conclusion is not new, then what’s so special
about this paper? Well, it was published 1956, the year that the Soviets
invaded Hungary, Elvis Presley entered the music charts for the first time, and
Dwight D. Eisenhower was re-elected as President of the United States. The
current one, Barack Obama, hadn’t been born yet, John Lennon and Paul McCartney
had not even met, and Donald Trump (probably) still had normal hair. The
internet was decades into the future and so was the “crisis of confidence” in
psychology.
The article that I'm talking about here was written by Adriaan de Groot, a
Dutch psychologist, who became internationally famous for his research on thought
in chess, which had a major influence on Nobel Prize winner Herb Simon (as well as on
my former colleagues Anders Ericsson and Neil Charness). De Groot also
developed an intelligence test that all Dutch children are required
to take at the end of elementary school and which selects them for various
tracks of higher education (I vividly remember taking that test when I was eleven).
A group of researchers from the University of Amsterdam, led
by Eric-Jan Wagenmakers, has now provided an annotated English translation of De
Groot’s article. This is fitting because De Groot held a professorship in
research methods in the Department of Psychology at the University of
Amsterdam. As I said at the beginning of this post, this article is currently
in press in Acta Psychologica, which
is also apt given the Dutch origin of that journal, but the author version can
be downloaded here
legally and for free from Wagenmakers’ site (also appropriate, given the Dutch stereotypical
thriftiness).
In their annotations Wagenmakers and his colleagues make the
sad observation that De Groot’s original article has been cited only twice (!)
to date. I expect that this translated
version will receive the number citations that the original already deserved.
I thank E.J. Wagenmakers for comments on a previous version.
I thank E.J. Wagenmakers for comments on a previous version.
I hope this paper will convince researchers that preregistration of analysis plans is nothing more than Methodology 101.
BeantwoordenVerwijderenE.J.
Indeed. I'm trying to convince all my collaborators of this and OSF makes preregistration a breeze.
VerwijderenThis blog is a very nice tribute to a psychologist who really deserves it: A.D. de Groot. 'Modern' psychology can still learn a lot from him, and in my opinion, his Methodology (1961) should still be obliged literature for psychology students in general and also for psychologists who haven't read it. One thing: De Groot did not develop an intelligence test for children, but you can consider him as the intellectual father of the Cito, after a visit in the late fifties to the Educational Testing Service in Princeton, during which De Groot developed a kind of blueprint for the Cito. This Dutch testing and assessment company developed the famous 'Cito-toets' (Cito-test) to which you are referring, which is more a scholastic aptitude test than an intelligence test (although you could argue how it is used nowadays).
VerwijderenVittorio Busato