Thursday, June 25, 2015

Diederik Stapel and the Effort After Meaning

Sir Frederic, back when professors still
looked like professors.
Take a look at these sentences:

A burning cigarette was carelessly discarded.
Several acres of virgin forest were destroyed.

You could let them stand as two unrelated utterances. But that’s not what you did, right? You inferred that the cigarette caused a fire, which destroyed the forest. We interpret new information based on what we know (that burning cigarettes can cause fires) to form a coherent representation of a situation. Rather than leaving the sentences unconnected, we impose a causal connection between the events described by the sentences.

George W. Bush exploited this tendency to create coherence by continuously juxtaposing Saddam and 9-11, thus fooling three-quarters of the American public into believing that Saddam was behind the attacks, without stating this explicitly.

Sir Frederic Bartlett proposed that we are continuously engaged in an effort after meaning. This is what remembering, imagining, thinking, reasoning, and understanding are: efforts to establish coherence. We try to forge connections between what we see and what we know. Often, we encounter obstacles to coherence and we strive mightily to overcome them. 


Take for example the last episode of Game of Thrones. One of the characters, Stannis Baratheon, barely survives a battle and is shown wounded and slumped against a tree. Another character strikes at him with a sword. But right before the sword hits, there is a cut to a different scene. So is Stannis dead or not? This question is hotly debated in news groups (e.g., in this thread). The vigor of the debate is testament to people's intolerance for ambiguity and their effort after meaning.

Stannis Baratheon, will he make it or not?
The arguments pro or contra Stannis being dead are made at different levels. Some people try to resolve the ambiguity at the level of the scene. No, Stannis could not have been killed: the positioning of the characters and the tree suggests that the sword would have struck the tree rather than Stannis. Other people jump up to the level of the story world. No, Stannis cannot be dead because his arc is not complete yet. Or: yes, he is dead because there is nothing anymore for him to accomplish in the story—let’s face it, he even sacrificed his own daughter, so what does he have left to live for! Yet other people take the perspective of the show. No, he is not dead because so far every major character on the show that is dead has been shown to have been killed; there are no off-screen deaths. Finally, some people take a very practical view. No Stannis cannot be dead because the actor, Stephen Dillane, is still under contract at HBO.

The internet is replete with discussions of this type, on countless topics, from interpretations of Beatles lyrics to conspiracy theories about 9-11. All are manifestations of the effort after meaning.

Science is another case in point. In a recent interview in the Chronicle for Higher Education, Diederik Stapel tries to shed light on his own fraud by appealing to the effort after meaning:

I think the problem with some scientists […], is you’re really genuinely interested. You really want to understand what’s going on. Understanding means I want to understand, I want an answer. When reality gives you back something that’s chaos and is not easy to understand, the idea of being a scientist is that you need to dig deeper, you need to find an answer. Karl Popper says that’s what you need to be happy with — uncertainty — maybe that’s the answer. Yet we’re trained, and society expects us to give an answer.

You don’t have to sympathize with Stapel to see that he has a point here. Questionable research practices are ways to establish coherence between hypothesis and data, between different experiments, and between data and hypothesis. Omitting nonsignificant findings is a way to establish coherence between hypothesis and data and among experiments. You can also establish coherence between data and hypothesis simply by inventing a new hypothesis in light of the data and pretending it was your hypothesis all along (HARKing). And if you don’t do any of these things and submit a paper with data that don’t allow you to tell a completely coherent story, your manuscript is likely to get rejected.

So the effort after meaning is systemic in science. As Stapel says, when nature does not cooperate, there is a perception that we have failed as scientists. We have failed to come up with a coherent story and we feel the need to rectify this. Because if we don't, our work may never see the light of day.

Granted, data fabrication is taking the effort after meaning to the extreme--let’s call it the scientific equivalent of sacrificing your own daughter. Nevertheless, we would do well to acknowledge that as scientists we are beholden to the effort after meaning. The simple solution is to arrange our science such that we let the effort after meaning roam free where it is needed—in theorizing and in exploratory research—and curb it where it has no place, in confirmatory research. Preregistration is an important step toward accomplishing this.

Meanwhile, if you want to give your effort after meaning a workout, don’t hesitate to weigh in on the Stannis debate.





6 comments:

  1. But how do you create meaning by fabricating data?
    Do you create more meaning for yourself? How can you ignore the fact that you just made up the data when you look at them as meaningful information?
    Surely, you created meaning for readers of your articles who falsely assume that the data are real data, but how can you get a sense of meaning out of lying to others.
    The search for meaning seems a strange explanation for data faking to me.

    It would make more sense for questionable probing of real data until they reveal the desired outcome, but I do not understand how faking data can satisfy a meaning-making motive. I see how it can satisfy other motives like fame, status, and power.

    ReplyDelete
    Replies
    1. I'm not sure where the line is. I can see how changing a single 3 into a 5 is not much different than deleting an "outlier" when you know that 95% of people wouldn't call them an outlier. But when you make up your entire study, it suggests that at some level, you don't really believe that what other people are doing is meaningful. Stapel hints at the idea that others must have been cheating, because how else did they get those amazing results? I suspect that there is some truth in that. People in the right-hand tail of the scientific achievement curve might be there due to skill, or just luck, or they might just be "making their own luck".

      Delete
    2. Let's not forget that in the same interview, Stapel admits that status was more important to him than truth. Effort after status is a very good reason for data fabrication in my view.

      Delete
    3. My take on this is similar. I suspect people committing fraud are creating coherence by assuming that something went wrong with the experiment whereas the hypothesis is essentially correct. The misbehaving data are then made to conform the hypothesis, which is the truth anyway. The "data" are more or less a formality required to obtain access to a top journal.

      Delete
    4. I didn't see Malte's comment until I posted mine. I meant to say that my take on it is similar to Nick's. However, I also agree with Malte. But in order to achieve status, you need to achieve coherence, is what Stapel must have thought. The post is arguing that there is pressure towards coherence in science.

      Delete
    5. I didn't see Malte's comment until I posted mine. I meant to say that my take on it is similar to Nick's. However, I also agree with Malte. But in order to achieve status, you need to achieve coherence, is what Stapel must have thought. The post is arguing that there is pressure towards coherence in science.

      Delete