Some years ago I served as an outside member on a
dissertation committee in a linguistics department. The graduate student in
question wanted to conduct experiments, an unusual idea for a linguist. When
the idea was discussed during the initial committee meeting, a colleague from the
linguistics department sighed and said dismissively Ah, experiments. Psychologists always want to do experiments because
they don’t know what’s going on. The years had not yet mellowed me (ahem),
so I had to bite back a snide comment.
But now I’m starting to wonder if that linguist didn’t have
a point after all. Isn’t our field suffering from premature experimentation? Don’t
we all have a tendency to design and run experiments before research questions
have been really thought through?
I see four major sets of reasons why this might be the case.
- Institutional. Empirical articles are the principal currency of our field, so there exists an incentive structure to design and run experiments. Graduate students need experiments to be awarded their degree, postdocs need them to secure a tenure-track job, junior faculty need them to receive tenure, senior faculty need them to procure grants.
- Educational. Students need to learn the trade. Designing and running experiments is a complex set of skills that take years to master.
- Cognitive. Experiments are used as mental crutches. It’s hard to mentally manipulate abstract concepts. It’s easier to think about designs, variables, conditions, counterbalancing, and randomization because much of this can be grounded in perception and action (e.g., in the form of an imaginary 2 X 2 table), or offloaded on the environment (e.g., sketched out on the back of an envelope).
- Temperamental. We are eager to “get our hands dirty” and curious to see results early on. When I get a new piece of software or some home appliance, I’m usually too impatient to carefully read the manual and/or do the tutorial. I take a quick look and then play it by ear. Running experiments before having things thought through is like starting without having perused the manual.
The last two reasons may not apply to everyone (they all
apply to me, I’m afraid) but it is clear that there is a pressure and a drive to produce
experiments.
How do we counter this pressure? I believe we should re-evaluate the
importance of speculative articles, essays in which authors try out and develop
thoughts about a specific topic. The essay is reviewed much like a philosophy
or linguistic article would be reviewed (e.g., based on theoretical relevance, soundness
of argumentation and clarity of exposition) and is then introduced to the
field, whereupon it may receive post-publication feedback, feedback that might give
rise to further theory development. And then, at some point, the moment for
experimentation has arrived.
Some topics are just not (yet) amenable to empirical
research. But this does not mean they aren't interesting or worthwhile to discuss... (I had drafted this post up to this point about two weeks ago. Today someone on Twitter referred to an article that illustrates my point and helps me conclude the post.)
In an article
that is currently in press in Perspectives
on Psychological Science, the social psychologist Joseph Cesario
takes a critical look at what he calls the literature on behavioral priming,
which has been the focus of several posts in this blog. Cesario observes that
the field is lacking theories that specify the precise contingencies under
which particular priming effects can be obtained. He then asserts that therefore failures [to replicate priming effects] are uninformative at the current time.
I concur with Cesario’s theoretical criticism of priming
research but I disagree with his statement that replications are uninformative. Moreover,
if a researcher cannot state the conditions under which an effect is expected
to replicate, then the study itself is uninformative. The experiment was
conducted prematurely.
It is much better in such cases to just be upfront and present
the idea as a thought experiment. The Gedankenexperiment
has a venerable history in science. Moreover, it does not carry with it the
pretense that there is empirical support for the idea.
There is more to say about the Cesario article but I’ll
limit myself here to the conclusion that it nicely illustrates my
main point: we should elevate essays to a higher status in our field and at the
same time become warier of premature experimentation.
Just imagine what would (not) have happened if Diederik
Stapel had not felt the need to produce “evidence” and had just described
thought experiments in a series of essays.