Replicating pencils and eagles but not steaks

What happens when we understand language?  What kind of mental representations do we form? Are they word-like or more perception-like? Traditionally, cognitive psychology has been leaning toward the first option but a little more than a decade ago, people started thinking that the second option might actually be closer to the mark.

Intuitively, the second idea makes a great deal of sense. I recently saw The Hobbit and it looked a lot the way I had imagined it during reading. Okay, perhaps I hadn’t pictured the wizard Radagast’s hair being plastered with bird shit when I read the book but otherwise the resemblance was pretty close. It sure looks like we create perception-like representations when we read.

How to test this intuition?

In my lab we came up with the sentence-picture verification task. Subjects read a sentence like He put the pencil in the cup and then a picture of a pencil. The pencil could either be oriented vertically or horizontally. A vertical pencil is more consistent with the sentence than a horizontal one, but this is irrelevant to the task we asked the subjects to perform. They merely decided whether or not the pictured object was mentioned in the sentence.

If people create word-like representations, the pencil’s orientation should be irrelevant but if they create perception-like representations, they should be faster in responding to the matching pencil than to the mismatching one.

What did the data tell us?

In the first experiment we ever did along these lines, my graduate student Rob Stanfield and I found support for perception-like representations. Subjects were faster on matching trials than on mismatching ones. We then tried to explore the limits of this effect. If it works for orientation, would it also work for shape? For example, if subjects read The ranger saw the eagle in the sky, would the respond faster to a flying eagle than a perched one? They did.

Others started getting into the game as well. One study wondered about color. Subjects read John looked at the steak in the butcher's window and then judged a picture of a steak that was either raw or cooked. And here something interesting happened. In contrast to the earlier studies, subjects now responded faster to the mismatching pictures than to the matching ones. Evidently, the sentences still influenced responses to the pictures, but in the opposite way.

These are the three experiments that we set out to replicate. The goal was to get a better understanding of the role of perception-like mental representations in language comprehension as viewed through the lens of sentence-picture verification.

Penciling in the results

The original orientation study found a small match effect. In both of our replications, we also found a small, but significant effect. But as I mentioned in my previous post, this doesn’t mean much if you have a large sample. And, in fact, the Bayes factor suggested the evidence was ambiguous at best for both replication attempts. However, when we combined the two experiments, the Bayes factor clearly pointed towards the alternative hypothesis. The evidence that people responded faster to the matching picture was about 25 times stronger than the evidence for the Null hypothesis.

It required a lot of power to replicate the effect. Yet, we didn’t have this much power in the original study. Did we fudge the data? Did we use “researcher degrees of freedom”? No we didn’t. In fact, we didn’t throw out any response times at all because we used median response times. We also didn’t run several experiments and just reported the “best” one (which seems a pretty silly thing to do anyway, if you ask me). We were just lucky. Had we not stumbled upon the effect, I’m pretty sure we would have abandoned the entire enterprise, especially because at the time I was still fully on board with the traditional cognitivist notion of word-like mental representations.

Shaping up to be a solid effect

 The original shape experiment produced a convincing effect. We replicated this effect twice. The effect for the alternative hypothesis was at least 100 times stronger than for the Null hypothesis. A resounding victory? Maybe. But one puzzling thing is that the effect size of the replications was about half that of the original experiment. We’re not sure why. There was less variability in the original lab-based experiment than in our Mechanical Turk replications. Also, the original experiment had a smaller sample and might thus have shown a more extreme effect.

Turning colors

The most interesting effect occurred in the color studies. Our replications yielded an almost perfect mirror image of the original results. Those results showed that mismatching pictures were responded to faster than matching pictures. Our replication results were in line with the orientation and shape studies.

Throughout the process, we have been in contact with Louise Connell, the author of the original color study, who has been very helpful by providing her original stimulus materials and by giving useful feedback on our results and manuscript. Did someone make a labeling error? We went back to our data and couldn’t find anything wrong. Louise couldn’t find anything wrong with the labeling in her data either. Do Mechanical Turk experiments for some weird reason show the opposite of lab experiments? The orientation and shape experiments suggest this is not the case. The original experiment was run with English subjects and the replications with Americans (at least, people residing in the US). So is it just a You say "potato," I say "patattah" thing? Unlikely.

The color study has considerably fewer items than the shape and orientation experiments. It also yielded longer response times. It is possible that this makes the color data less stable.  Might this explain the reversal of the pattern? We don’t know. We talked with Louise Connell about a collaborative effort to further investigate this question.

And in the end

This walk down replication lane has proven useful. We now have more confidence that when language implies features of objects, such as their orientation, shape, or color responses to subsequent pictures are facilitated (at least in the context of our experimental tasks). This is consistent with the idea that we form perception-like mental representations while reading. Perhaps a far more complex form of this is at work when we read novels like The Hobbit. This would explain why we experience a déjà vu when the movie adaptation shows us the lands of Middle Earth.

We also have developed a powerful and relatively fast way of doing replication studies that removes experimenter effects, has safeguards against false positives, and uses a broader sample of the population than the typical lab-based experiment. We are currently performing other replication studies with this method. And of course we’re also using it to study novel ideas. I will describe those in a later post.