tag:blogger.com,1999:blog-6322739827777311964.post6870659574122544759..comments2023-11-30T13:15:01.893+01:00Comments on Drang naar Samenhang: Coming out of the file drawerRolf Zwaanhttp://www.blogger.com/profile/07617143491249303266noreply@blogger.comBlogger6125tag:blogger.com,1999:blog-6322739827777311964.post-7303548701830337972013-01-03T09:42:01.212+01:002013-01-03T09:42:01.212+01:00I don't think it's an either-or-situation....I don't think it's an either-or-situation. If the standards for analysis go up, the standards for publication in regard to effects don't necessarily have to go down. Maybe the publication pressure should go down instead so that researchers can take more time to run studies. However, I'm the first one to admit that I'm a very impatient guy, so I'm glad that I can run online experiments.<br /><br />Everybody always says it's hard to get non-replications published but I wonder how much of this is self-handicapping. In my three years as Editor-in-Chief, I've handled well over a 1000 manuscripts. Practically none of them featured non-replications.<br /><br />There should be a place for both chronologically accurate and plot-based accounts. Aristotle (if I may throw a Greek philosopher back at you) already argued that historians should use the former and dramatists and epicists the latter. Maybe the chronological accounts should go into archival journals and maybe the plot-based ones in blogs. The former would be part of the scientific record and the latter would be ways to inform a broader audience. <br /><br />In my next post, I'll describe a chronologically-based 14-experiment Behemoth we are writing at the moment.<br />Rolf Zwaanhttps://www.blogger.com/profile/07617143491249303266noreply@blogger.comtag:blogger.com,1999:blog-6322739827777311964.post-69468339746559307872013-01-02T21:31:31.177+01:002013-01-02T21:31:31.177+01:00Ah but what I wanted to ask is whether the new sta...Ah but what I wanted to ask is whether the new standards could put a lot of social/cog questions out of reach for many people in these fields using p-values now as sufficient evidence. How many people are getting effects that would stand up to Bayesian analysis? Or non-overlapping CIs? The standards for publication in regard to effects must, in a way, come down if the standards for analysis go up. A theoretically sound design should be publishable regardless of the results as long as the results are informative. <br /><br />It's remarkable how often I've found people talking about non-replications in their labs (usually of other people's work) as if those efforts provided valuable information yet journals look with deep suspicion on null results. I asked one editor about this and he replied that there could be many reasons for not seeing a difference between conditions! Annnnnnnnd?<br /><br />I think we must also abandon the Platonic mode of writing and publishing in which studies are reported in ways that fit schematic views of scientific research. At SPSP a couple years ago the editor of one of the top social journals was asked whether he liked to see a chronology of the work with various hypotheses considered or a clean story, and he dismissed chronologies as mere historical records. I kind of snorted and looked around to expect similar reactions but if anyone else had a problem with his response, I didn't see it. If we constantly reframe the research process in publications with a fictional narrative that obscures the exploratory nature of much psychological research, the clean story that makes for easy bedtime reading only fuels the conceptual and statistical misunderstandings that have made so many "findings" in print highly doubtful now in retrospect. Not every notion and every lab debate needs to be reported but somehow our publications should reflect the actual process more than they obscure it. This would reflect the humility of science so much discussed in introductory textbooks.Dr. Mhttps://www.blogger.com/profile/00312008793887210079noreply@blogger.comtag:blogger.com,1999:blog-6322739827777311964.post-85955461804660850502013-01-02T21:23:21.719+01:002013-01-02T21:23:21.719+01:00Let me preface this by repeating I'm not at a...Let me preface this by repeating I'm not at all an expert in Bayesian statistics. However, I believe this article, http://pps.sagepub.com/content/6/3/291.full.pdf., might be helpful to you. It addresses a lot of your questions, including the relation between p-values and Bayes factors and how to interpret the latter. I believe yours is already nothing to sneeze at. http://pps.sagepub.com/content/6/3/291.full.pdf.<br /><br />Thanks for reading the blog!Rolf Zwaanhttps://www.blogger.com/profile/07617143491249303266noreply@blogger.comtag:blogger.com,1999:blog-6322739827777311964.post-19998576138029448202013-01-02T20:56:06.459+01:002013-01-02T20:56:06.459+01:00Thanks for the link to the Bayesian calculator. A...Thanks for the link to the Bayesian calculator. Are there guidelines yet for what Bayesian ratios we should expect in most psychology experiments?<br /><br />I'm running a study now involving motor priming and the initial results look good from the old school perspective: p = 0.05, Cohen's d = .55, N=50, between subjects design. But I plug the numbers into the Bayesian calculator and the ratio is only 0.28, far less than the ratio you cited in your Turk replication.<br /><br />The 95% CIs for each condition still overlap, to the extent that I imagine I'd need at least 50 more subjects to show a clear separation, even though the p-value could be driven well below .05 much sooner than the CIs diverge.<br /><br />These are the cleanest results I've ever obtained in a first attempt at a new concept. I'm not sure my study could be run through Turk at all, and I'm pretty sure I couldn't access participants with the required demographic backgrounds. I have a large subject pool, so I can add 50 more in February/March without difficulty, but as you indicate, I think most researchers face severe limitations with studies that can't be executed online. Effect sizes in social/cognitive psychology aren't usually high enough to be captured reliability with small samples (not that ES depends on N mathematically, but that there's a relationship between p values, sample sizes, and effect sizes in conducting studies).<br /><br />And through all of this, I'm not sure I even understand how p values and Bayesian ratios systematically relate to each other, if they do. My impression is that Bayesian analysis tests the specific alternative that's being used in the study, as opposed to merely indicating the probablity of getting these results when the conditional populations are theoretically identical. It shifts our statistical attention toward the likelihood of the alternative hypothesis rather than the probability of our data existing in a null world? This distinction seems so subtle if you're used to (mis-)using p values as statements about your hypothesis.<br /><br />(Why the hell is my ID here listed as Dr M.?? This is Mark, btw).Dr. Mhttps://www.blogger.com/profile/00312008793887210079noreply@blogger.comtag:blogger.com,1999:blog-6322739827777311964.post-31109474565649981062013-01-01T19:58:09.695+01:002013-01-01T19:58:09.695+01:00Thanks, Andrew!Thanks, Andrew!Rolf Zwaanhttps://www.blogger.com/profile/07617143491249303266noreply@blogger.comtag:blogger.com,1999:blog-6322739827777311964.post-34226385263973822792013-01-01T19:30:44.201+01:002013-01-01T19:30:44.201+01:00Great start to the blog -- the metaphors are great...Great start to the blog -- the metaphors are great!Anonymoushttps://www.blogger.com/profile/04898698566426392639noreply@blogger.com