Bruce Springsteen and Lazy Susan: The Logic of Experimentation

Many papers have more than one experiment. How do researchers string together experiments?

When I was a beginning assistant professor, I would tell my students The difficult part is not Experiment 1, it is Experiment 2 (when saying this, I would always picture myself as a grizzled war veteran, smoking a pipe and using its stem to point out locations in the field of battle to my crop of young recruits.)

Why was Experiment 2 so difficult? Well, it couldn’t stray too far from Experiment 1 or reviewers might expose a gap in the causal chain that leads from experiment to experiment. It also couldn’t be too close because then you’d have a direct replication of Experiment 1. (Nowadays we would see this as a virtue but back then it would probably have been viewed as unnecessary padding.)

For many years I thought having a tight chain of experiments was normal for experimental research. But then I started reading articles in social psychology. I did this partly because I was asked as a reviewer (due to my interest in embodied cognition there was some overlap with some work in social cognition, at least in the perception of some editors) and partly because I became chairman of a committee investigating alleged fraud in the work of Dirk Smeesters.

It struck me that quite a few social psychology articles (though most certainly not all) don’t follow the logic of experimentation I just described. The experiments in those papers all relate to a general theme but there is no causal chain. I’ve tried to depict the difference between cognitive and those social experiments below.

Eagles

Youth soccer

In cognitive psychology there tend to be strong causal connections between the individual experiments (the blue boxes). Usually, the first experiment is the main one, relating to the research question (the red box), and the others serve to rule out potential alternative explanations for the results of the main experiment. It’s the Bruce Springsteen model. One experiment is the boss and the others are important but act in a supportive role—the E Street Band. However, sometimes all experiments are equally important to the main research question and then we have the Eagles model. The Bruce Springsteen and Eagles models describe pretty much all of cognitive research.

In the social psychology articles I’m talking about, there does not always appear to be a causal connection between the experiments. They all relate somewhat (hence the thinner arrows in the social model) to the central research question (which is pitched at a higher level than in cognitive articles) but in very different ways. You could pretty much put the experiments in any order and the paper would be the same. I cannot think of a musical analogy but I'm reminded of youth soccer. The experiments are like a bunch of six-year-olds playing soccer. They’re all very cute but they’re not playing as a team.

I don't know how much of social psychology can be explained by the youth-soccer model but I think it's a sizable subset (I also think virtually no cognitive studies conform to this model).

Here is a concrete example. I’m not picking on the authors. I could easily have selected many others and I think the experiments in this paper are very clever. I’m also not going to discuss the research in detail. I’m only interested in how the experiments hang together.

The authors examined whether merely executing and seeing clockwise (vs. counterclockwise) movements would induce psychological states of temporal progression (vs. regression) and according to motivational orientations toward the future and novelty (vs. the past and familiarity).

Here is how they summarize the support for their hypothesis:

Participants who turned cranks counterclockwise preferred familiar over novel stimuli, but participants who turned cranks clockwise preferred novel over old stimuli… (Experiment 1). Also, participants rotating a cylinder clockwise reported higher scores in the personality measure openness to experience than participants rotating counterclockwise (Experiment 2). Merely passively watching a rotating square had similar but weaker effects on exposure and openness (Experiment 3). Finally, participants chose more unconventional candies from a clockwise than from a counterclockwise Lazy Susan, that is, a turntable (Experiment 4).

So Experiment 2 used a different task than Experiment 1 (what happened to the cranks?) and had a different dependent measure. Why not change one thing at a time? Do we really already know what’s going on with the crank task? And why use a self-report task when you had a nice implicit task in Experiment 1? Is indicating agreement with statements like Poetry does not deeply impress me really comparable to judging Chinese ideographs (which is what the subjects did in Experiment 1)?

In Experiment 3, the subjects watched a square whereas they rotated a cylinder (which turns out to be a paper towel roll; I’ll take the cranks anytime) in Experiment 2. The researchers have good reasons for using a square (it doesn’t look like a clock) and the experiment makes sense in light of the previous one. They wanted to disentangle visual and manual rotation. Also good about Experiment 3 is that it uses the dependent measures of Experiments 1 and 2. Still, how do we know that those tasks are measuring the same mechanism? One is implicit and the other requires introspection. One is visual and the other verbal.

But then we get to Experiment 4. Here Lazy Susan makes an entrance and the dependent measure is candy selection (leave it to Lazy Susan to get the party going). Now I am seriously confused, and it’s not because of Lazy Susan’s pretty blue eyes.

The advantage of the Springsteen/Eagles model is that it results in a tightly interwoven set of experiments. Each experiment is often a partial replication of the previous one. But there is the risk of vacuous experimentation; researchers lose sight of the bigger question that motivated the research in the first place. But that is just an experiment about an experiment I have heard myself say during many lab meetings (waving my imaginary pipe). I could mention whole areas of cognitive psychology that are mired in vacuous experimentation, but that’s for a later post.

The advantage of the youth-soccer model is that researchers keep their eye on the bigger question. The disadvantage is that it does not allow them to gather a coherent body of evidence to address this question. And there is another disadvantage: false positives. Suppose the Lazy Susan experiment had “failed.” The researchers could have simply concluded that the task didn’t “work” (damn you, Lazy Susan!) and tried something else. Maybe they would have had people watch a hamster in a treadmill and then judge melodies played by a harpsichord or by a synthesizer.

I don’t know if there are courses on the logic of experimentation. If one were to be developed, it should teach the Bruce Springsteen model and the Eagles model. But it should also tell students not to lose sight of the red box.