Update February 3, 2013, I removed the figure for reasons explained in my next post. However, I wanted to keep the text of this post to provide a documentation of the process. I'm trying to be open as well as accurate.
Holy cow! The data are in already!
Holy cow! The data are in already!
As I explained in my previous
post, I was interested to see what the effects of amusing titles are on the
perception of the results of studies published over the years in Psychological Science. Subjects read the titles
and the associated abstracts of twelve articles and indicated their
confidence and interest in the results of these studies as presented in the abstracts. They used an 11-point scale to do so, with 0 being extremely low and 10
extremely high confidence and interest in the findings.
The preliminary results are in! It only took about 5 hours to collect the data. I was worried that the Mechanical Turk subjects would find the task too difficult and/or boring. Instead, they seemed to like it quite a bit!
I haven’t had time to do very detailed analyses yet but I wanted to present the initial findings now.
My prediction was that amusing titles would lead to lower
confidence but higher interest in the findings. I was partly right and partly
wrong, as the initial results show.
Indeed, amusing titles lead to lower confidence in the associated findings. Surprisingly (to me at least), they also lead to lower interest in the findings. Overall, the subjects had moderate amounts of confidence and interest in the findings of the twelve studies.
The differences between amusing and non-amusing titles may be small (at least, a half point seems small to me) but they are highly significant.
The error bars indicate 95% confidence intervals. I also computed Bayes factors. They show that there is very strong evidence (but see my next post) that amusing titles lead to lower confidence in the results of the study than do non-amusing titles. They also show that there is very strong evidence for the surprising result that amusing titles lead to lower interest in the associated results than do non-amusing titles. In other words, I did not just find significant effects because of the large sample size. The Bayes factor guards against false positives like this.
The error bars indicate 95% confidence intervals. I also computed Bayes factors. They show that there is very strong evidence (but see my next post) that amusing titles lead to lower confidence in the results of the study than do non-amusing titles. They also show that there is very strong evidence for the surprising result that amusing titles lead to lower interest in the associated results than do non-amusing titles. In other words, I did not just find significant effects because of the large sample size. The Bayes factor guards against false positives like this.
So clearly (but see my next post), amusing titles affect the perception of research in a negative way!
I guess it’s unusual and a little risky to design an experiment, pre-register it on the same day, run it a few days later, analyze the data
the same day that they were collected, write an initial report a few hours later, and post it online immediately afterwards (no kidding!). If only I were younger, I'd attribute it to youthful enthusiasm. Now I don't have an excuse.
I will provide a more detailed report later.
Cool stuff. Thanks for satisfing our curiosity so quickly. Now I can't wait for the juicy more detailed report - please check if any moderating/mediating effects appear on the actual knowledge about the studies read or time spent on reading the abstract...
BeantwoordenVerwijderenThanks. Will do!
VerwijderenCool. Very nice and informative.
BeantwoordenVerwijderenI have added a link to this from my blog post about how to draft paper titles. http://wp.me/p11WgN-6Y
Thanks, very useful tips. I agree that being smart about your titles will enhance the impact of your work.
VerwijderenWere these the same abstracts, but with different titles, or actual abstracts and their titles? If the latter, it may simply be that lower-quality abstracts are more likely to have amusing titles. Interesting topic to study, though.
BeantwoordenVerwijderenDeze reactie is verwijderd door de auteur.
VerwijderenThe same abstract was used in both conditions. They were actual abstracts. The articles all had "amusing" titles. The amusing part appeared before the colon and could be eliminated without loss of information to create a non-amusing title.
VerwijderenThis is neat for two reasons. For one it's great to see the question put forth, developed, and resolved so quickly. I'm sure like many the plague of cute titles has become almost gag inducing. I went the opposite way with my first paper and used what my supervisor called "might just be the most boring title" he'd seen. Now i'm glad I did :)
BeantwoordenVerwijderenAlso kudos to you for doing something I've wanted to for a long time- straight up self-publishing something that most would save for an academic publication. In the greater debate about OA and publishing I think it's an awesome step forward to just throw stuff out there like this.
It isn't like we can just ignore your findings - and users can and already have question you about your methodology. You could imagine a future where specially formatted blog-posts are automatically picked up by google scholar and dumped into PDFs with annotated comments. Now if I want to refute or add to your findings, I could self-publish as well and directly cite your post. For now it's just neat to see folks going from idea to completion totally within the public domain.
I think there might be differences between different types of amusing titles. Some will be more effective than others. This is a topic for the next experiment.
VerwijderenThanks for the kudos. I wouldn't call this a full-fledged publication, though. The results section would have to be a lot more detailed but this is not necessarily what you want in a blog. You're right though that it is citable and that people can respond to it. And even though it hasn't been peer-reviewed, it has been "peer pre-reviewed."
Again, though, this is just the first in a series of experiments. It takes more than a single experiment to really figure out how amusing titles work (or do not work).
Very nice results, with a very neat design!
BeantwoordenVerwijderenWould you predict it goes the same way for conference talks? In this context, it might be more difficult to give up on listening, for politeness reasons (although this assumption is probably not evidence-based). But if a paper stops interesting you, you can leave it, definitely or to come back to it later. Or you can skip parts. That is, you have the control of your pace should interest get lower, and being bored is not so much a concern.
Thanks! I stole the design from my dissertation research where I manipulated genre expectations while using the same texts in both conditions (and I stole this idea from Stanley Fish).
VerwijderenI don't know what to predict about conference talks. I also don't think at all that an amusing title is necessarily a harbinger of something bad. In fact, I thought that several of the studies that I used were very interesting and clever.
Interesting! I was planning on doing a similar analysis to this but instead of looking at the confidence of average joe on mechanical turk (who might just skip the abstract and be too distracted by the title?), look at the reactions of academics by comparing how often papers with puns in the titles are cited compared to papers without puns in the titles. Data easily found on google scholar. Might be worth a look?
BeantwoordenVerwijderenGood idea. There already is a study along these lines though. I cite it in my first post on amusing titles http://rolfzwaan.blogspot.nl/2013/01/overly-amusing-article-titles.html.
VerwijderenSimilar comment to Shlomo about control conditions. I seem to recall seeing evidence that papers with shorter titles are cited more (I'm afraid I have no idea if my memory is correct, or where the original source might be). By removing the 'funny' bit you obviously made the title shorter.
BeantwoordenVerwijderenTo control for this you could rate the abstracts with either no title, or the 'funny' section of the title only. Obviously this would lead to a loss of information but would resolve the length of title issue.
Great analyses though, really interesting. I'm still going to generate pun titles, but probably not try to put them into papers now!
That's true, the titles were shorter. I think this is better than inventing a long title, which would be hard to do in a believable way. It is also better than only using the amusing part because by itself it usually isn't very informative.
VerwijderenI might include this condition in a future experiment, though. I thought that subjects would hate the experiment and thought that using 12 abstracts was already stretching it. Now I think I could probably use a few more, which might allow me to include more conditions.
I wouldn't want to change the way I generate titles based just on these initial results.
Love it! Provides preliminary facts to support what has been the experience of many researchers...
BeantwoordenVerwijderenThanks! Like you say, these are preliminary facts.
VerwijderenI love that you have publicly documented the entire experimental process both before and immediately after collecting the data, and presumably with further updates to be reported in the near future. This is how all scientific communication ought to take place.
BeantwoordenVerwijderenThanks. I'm working on the update. I think it would be better to take a little bit more time between collecting the data and reporting them. The update will make clear why.
VerwijderenPerhaps some titles are more amusing than others? Reaction might differ depending on whether the person is genuinely amused versus if the reader thinks the author is attempting and failing to be clever.
BeantwoordenVerwijderenPerhaps try improving titles in various ways?
I do think there are differences among amusing titles. Some are genuinely enlightening whereas others seem gratuitous. I plan to write more about this in the future.
Verwijderen'Amusing' titles have been a pet peeve for a while. Thanks for supporting its sustenance :) Did or will you check whether they also get cited less often?
BeantwoordenVerwijderenOther researchers have done this. I cite them here: http://rolfzwaan.blogspot.nl/2013/01/overly-amusing-article-titles.html.
BeantwoordenVerwijderenDeze reactie is verwijderd door een blogbeheerder.
BeantwoordenVerwijderen