You’ve dutifully conducted a power analysis, defined your
sample size, and conducted your experiment. Alas, p=.20. What now? Let’s find
out.
The Good Ship DataPoint* 
One of my students wanted to extend (or conceptually
replicate) the verbal overshadowing effect for her master’s thesis by using
different stimuli and a different distractor task. I’m not going to talk about
the contents of the research here. I simply want to address the question that’s
posed in the title of this post: p=.20,
what now? Because p=.20 is what we found after having run 148 subjects, obtaining a verbal overshadowing effect of 9% rather than RRR1's 16%.**
Option 1. The effect is not significant, so this
conceptual replication “did not work,” let’s file drawer the sucker. This response
is probably still very common but it contributes to publication bias.
Option 2. We consider this a pilot study and now
perform a power analysis based on it and run a new (and much larger) batch of
subjects. The old data are now meaningless for hypothesis testing. This is
better than option 1 but is rather wasteful. Why throw away a perfectly good
data set?
Option 3. Our method wasn’t sensitive enough. Let’s
improve it and then run a new study. Probably a very common response. But it may be
premature and is not guaranteed to lead to a more decisive result. And you’re
still throwing away the old data (see option 1).
Liverpool FC, victorious in the 2005 Champions League final
in Istanbul after overcoming a 30 deficit against AC Milan

Option 5. The
effect is not significant but we’ll keep on testing until it is. Simmons et
al. have provided a memorable
illustration of how problematic optional stopping is. In his blog, Ryne Sherman describes a Monte Carlo simulation of phacking, showing that it can inflate
the false positive rate from 5% to 20%. Still, the intuition that it would be
useful to test more subjects is a good one. And that leads us to…
Option 6. The
result is ambiguous, so let’s continue testing—in a way that does not inflate the Type I error rate—until we have decisive
information or we've run out of resources. Researchers have proposed several ways of sequential testing that
does preserve the normal error rate. EricJan
Wagenmakers and colleagues show how repeated testing can be performed in a Bayesian
framework and DaniĆ«l
Lakens has described sequential testing as it is performed in the medical
sciences. My main focus will be on a littleknown method proposed in psychology by Frick (1998),
which to date has been cited only 17 times in Google Scholar. I will report
Bayes factors as well. The method described by Lakens could not be used in this
case because it requires one to specify the number of looks a priori.
Frick’s method is called COAST (composite open adaptive sequential test). The idea is appealingly
simple: if your effect is >.01 and <.36, keep on testing until the
pvalue crosses one of these limits.*** Frick’s simulations show that this procedure keeps the overall alpha level
under .05. Given that after the first test our p was between the lower and upper limits, our Good Ship DataPoint was in deep waters. Therefore, we continued testing. We
decided to add subjects in batches of 60 (barring exclusions) so as to not
overshoot and yet make our additions substantive. If DataPoint failed to reach shore before we'd reached 500 subjects, we would abandon ship.
Batch 2: N_{total}=202,
p=.047. People who use optional
testing would stop here and declare victory: p<.05! (Of course, they wouldn’t mention that they’d already
peeked.) We’re using COAST, however, and although the good ship DataPoint is
in the shallows of the Rectangular Sea of Probability, it has not reached the
coast. And BF10=0.6, still leaning toward H0.
Batch 3: N_{total}
= 258, p=.013, BF10=1.95. We’re
getting encouraging reports from the crow’s nest. The DataPoint crew will likely
not succumb to scurvy after all! And the BF10 now favors H1.
Batch 4: N_{total}
=306, p=.058, BF10=.40. What’s
this??? The wind has taken a treacherous turn and we’ve drifted away from shore. Rations are getting lowmutiny
looms. And if that wasn’t bad enough, BF is
<1 again. Discouraged but not defeated, DataPoint sails on.
Batch 5: N_{total}
=359, p=.016, BF10=1.10. Heading back
in the right direction again.
Batch 6: N_{total}
=421, p=.015, BF=1.17. Barely closer.
Will we reach shore before we all die? We have to ration the food.
Batch 7: N_{total}
=479, p=.003, BF10=4.11. Made it! Just before supplies ran out and the captain
would have been keelhauled. The taverns will be busy tonight.
Some lessons from this nautical exercise:
(1) More data=better.
(2) We now have successfully extended the verbal overshadowing
effect, although we found a smaller effect, 9% after 148 subjects and 10% at the end of the experiment.
(3) Although COAST gave us an exit strategy, BF10=4.11 is
encouraging but not very strong. And who knows if it will hold up? Up to this
point it has been quite volatile.
(4) Our use of COAST worked because we were using Mechanical Turk.
Adding batches of 60 subjects would be impractical in the lab.
(5) Using COAST is simple and straightforward. It preserves an overall alpha level of .05. I prefer to use it in conjunction with Bayes factors.
(6) It is puzzling that methodological solutions to a lot of our problems are right there in the psychological literature but that so few people are aware of them.
Coda
In this post, I have focused on the application of COAST and largely ignored, for didactical purposes, that this study was a conceptual replication. More about this in the next post.
Footnotes
Acknowledgements: I thank Samantha Bouwmeester, Peter Verkoeijen, and Anita Eerland for helpful comments on an earlier version of this post. They don't necessarily agree with me on all of the points raised in the post.
*Starring in the role of DataPoint is the Batavia, a replica of a 17th century Dutch East Indies ship, well worth a visit.
** The original study, Schooler and EngstlerSchooler (1990), has a sample of 37 subjects and the RRR1 studies typically had 5080 subjects. We used chisquare tests to compute pvalues. Unlike the replication studies, we did not collapse the conditions in which subjects made a false identification and in which they claimed the suspect was not in the lineup because we thought these were two different kinds of responses. I computed Bayes factors using the BayesFactor package in R. I used the contingencyTableBF function with sampleType = "indepMulti", fixedMargin = "rows", priorConcentration= 1. In this analysis, we separated false alarms from misses, unlike in the replication experiments. This precluded us, however, from using onesided tests.
*** For this to work, you need to decide a priori to use COAST. This means, for example, that when your pvalue is >.01 and <.05 after the first batch, you need to continue testing rather than conclude that you've obtained a significant effect.
(5) Using COAST is simple and straightforward. It preserves an overall alpha level of .05. I prefer to use it in conjunction with Bayes factors.
(6) It is puzzling that methodological solutions to a lot of our problems are right there in the psychological literature but that so few people are aware of them.
Coda
In this post, I have focused on the application of COAST and largely ignored, for didactical purposes, that this study was a conceptual replication. More about this in the next post.
Footnotes
Acknowledgements: I thank Samantha Bouwmeester, Peter Verkoeijen, and Anita Eerland for helpful comments on an earlier version of this post. They don't necessarily agree with me on all of the points raised in the post.
*Starring in the role of DataPoint is the Batavia, a replica of a 17th century Dutch East Indies ship, well worth a visit.
** The original study, Schooler and EngstlerSchooler (1990), has a sample of 37 subjects and the RRR1 studies typically had 5080 subjects. We used chisquare tests to compute pvalues. Unlike the replication studies, we did not collapse the conditions in which subjects made a false identification and in which they claimed the suspect was not in the lineup because we thought these were two different kinds of responses. I computed Bayes factors using the BayesFactor package in R. I used the contingencyTableBF function with sampleType = "indepMulti", fixedMargin = "rows", priorConcentration= 1. In this analysis, we separated false alarms from misses, unlike in the replication experiments. This precluded us, however, from using onesided tests.
*** For this to work, you need to decide a priori to use COAST. This means, for example, that when your pvalue is >.01 and <.05 after the first batch, you need to continue testing rather than conclude that you've obtained a significant effect.
I'm curious whether the COAST approach would be invalidated by stopping with N=500 if you hadn't surpassed the thresholds. That is, do the thresholds depend on continuing to test until you pass either the upper or lower one. If so, then this wouldn't be a particularly useful approach  you'd have no way to know how many participants you might need to test, and you would be obligated to continue testing indefinitely. Presumably simulations could show whether or not stopping with a fixed N changes the overall false positive rate. For the sake of argument, let's assume that stopping at some fixed N doesn't invalidate the approach. Had you reached N=500 without surpassing one of the p cutoffs, what could you conclude?
ReplyDelete(Note: I tried posting a variant of this comment a few minutes ago, but the site seemed to lose it. Sorry if it shows up twice.)
I agree that this is a potential problem. I believe the same is true about the Bayesian sequential analysis, though. The Bayes factor could in principle remain between 1/3 and 3 indefinitely (I think). So you need to also have a practical exit strategy (a maximum N). So if we'd reached N=500, we'd be forced to conclude that we failed to reject H0 and provide an effect size estimate, stating that this was accomplished with a sample size about 7 times those in RRR1. We'd also report the Bayes factor, which would have probably suggested that there was more support for H1 than for H0 but not convincingly so.
Delete"The Bayes factor could in principle remain between 1/3 and 3 indefinitely (I think)"
DeleteNope: The BF converges to 0 or infinity with increasing n. So it is guaranteed to leave the inconclusive area. It can take long, but not indefinitely. In a forthcoming paper, we simulate such sequential BFs. The longest journeys happen at d=0, and there 95% of all studies reach a boundary of 3 (resp. 1/3) with n<120 (in a twogroup ttest setting).
But: We do not recommend a BF boundary of 3; rather at least 5.
COAST sounds interesting, but users should be aware of two properties:
 At the end you report p<.01; but you should emphasize very clearly that the actual level is p<.05. Furthermore, it is hard to compute the actual (exact) pvalue, but it is definitely *not* .003. So the sequential p can be seductive for the casual reader.
 Effect size estimates can be very biased in sequential designs, conditional on whether you have an early stop or a late stop. If you stop early, ES is overestimated, if you stop late, it can be underestimated. How much bias happens depends on the properties of the COAST procedure.
Frick does not discuss this issue in his paper, but in the literature of clinical sequential designs this conditional bias is well known. These biased effect sizes do not invalidate the hypothesis test; but the empirical (sample) ES estimates should be treated with caution.
(1) You're right. Let's correct this to "the Bayes factor could in principle remain between 1/5 and 5 until the researcher runs out of resources and/or patience." It is also true of course that p will eventually leave the inconclusive area. (2) You're also right about the actual level of p. This is not reported in the post because I'm just concerned with where we stop. (3) Yes, that's why it is recommended that for metaanalyses, the effect size from the initial sample is used: http://link.springer.com/article/10.3758/s134280130433z#page1.
DeleteBTW: I directly dived into technical issues, but most important: Great post!
Deletead 3) Thanks for the link. It seems to me that there's no real consensus yet about how to handle sequential designs in metaanalyses. These authors (http://www.ncbi.nlm.nih.gov/pubmed/23824994) conclude that: `early stopping of clinical trials for apparent benefit is not a substantive source of bias in metaanalyses whereas exclusion of truncated studies from metaanalyses would introduce bias. Evidence synthesis should be based on results from all studies, both truncated and nontruncated' (p. 4873).
Thanks! And interesting reference. This is clearly an issue that is going to deserve greater attention if sequential analyses become more common.
DeletePvalue corrections and adjustments for effect sizes have been developed (see Lakens, 2014, or Proschan et al., 2006). Alternatively, metaanalytic adjustments for publication bias might work here (PETPEESE). So sure, something to keep in mind, but not too problematic.
Delete"in a way that does not inflate the Type I error rate"
ReplyDeleteIt has to inflate the Type I error rate. The Type I error rate is 0.05 once you finished your first batch. If you got significant results after the first batch you would not use COAST. Above the 0.05, you have to add the Type I error rate added by using COAST. It may be small, but it is necessarily bigger than 0. You cannot hold Type I at 0.05 using sequential testing unless you decide to use it in advance and adjust alpha for the first batch accordingly.
"You cannot hold Type I at 0.05 using sequential testing unless you decide to use it in advance and adjust alpha for the first batch accordingly." This is true.
DeleteYes, so Frick adjusts the criterion value so that p < .05 remains valid.
DeleteSure, but you have to adjust the criterion value even for the first batch. That is, if you decide to use COAST only after getting nonsignificant results, the error rate will be inflated.
DeleteYes, you cannot decide beforehand you're going to use COAST and then drop that plan when your initial sample has <.01 p <.05 and call it a day.
DeleteWell, I am interested in the possibility that you seem to recommend in the post – if you don't get significant results, use COAST. That is what increases the Type I error rate. I mean, it may be better than the other options, but it should not be described as holding the Type I error rate at 0.05.
DeleteThe way to imagine this: Assume that the null is true. You run the first batch. You find a significant result with probability 0.05 and if you do, you publish the result as significant. If you do not find a significant result (with probability of 0.95), you use COAST. COAST can than give you with certain probability (say 0.04 for the sake of the example) a signficant result. If it does, you describe it as a significant result. The total Type I error rate is than ~0.09 (i.e. 0.05 + 0.95*0.04) and not 0.05.
I see what you're saying. But it is not what I recommend. Rather, it is my attempt to engage the reader by appealing to a situation that is probably familiar to them. I agree that it would have been better to say under option 6: "You could have chosen to use COAST" or something like that.
DeleteI have added footnote *** to address your point. Thanks.
DeleteGreat post. And Frick deserves more attention and citations.
ReplyDeleteIt is important to stress that typeI error probabilities (pvalues) increase with the number of tests that are being conducted. Moreover, biases in effect sizes will be greater if testing stops after each new data point is collected and p reaches criterion value. To avoid these problems it makes sense to increase in meaningful batches. A fixed N = 60 may seem reasonable, but sampling error decreases in a nonlinear fashion. So it would be more efficient to increase sample sizes in an exponential function (50, 100, 200, 400). The limit of data collection can be determined with a priori power analysis. Would you give up when the effect size is small (d = .2) and needs N = 800 participants (betweensubject design).
Interesting, intuitively I'd think that exponential increases would be less efficient. The idea to use the sample size based on a power analysis for a small effect as the final exit strategy makes a lot of sense.
DeleteHi Rolf,
ReplyDeleteI liked this post. I'm curious after having thought about it a little bit.
You say that the Bayes factors aren't really interesting to you or meaningful after batch1 because surely things might change with the next additions of new data. 60 more minutes of play and all that. And on twitter you said the bayes factor would be meaningful once it stabilizes. But had the results come in more conveniently for you (read: ev+ for the alternative in batch1), wouldn't you have stopped the experiment and simply taken the bayes factor as is? There would not have been 60 more minutes of play in that case, and you would not have thought your data were too few. So are they only uninteresting or not meaningful when they are not convenient for you?
Perhaps I missed something subtle but that is my perception of your comment in the post and on twitter. Of course interest level is wholly subjective and personal, so I'm interested to hear your thoughts.
 Clearly this assumes you would have stopped the experiment after batch1 if the results were different. But as you say, you did plan to initially stop after batch1 so I don't think it is too strong of an assumption on my part.
Glad you liked the post. I haven't seen many sequential Bayesian analyses yet but the ones I've seen, mostly from E.J., show a lot of fluctuation at the beginning (as you might expect) and then some form of stabilization. So this is what I was also expecting for our data. One of E.J.'s stopping criteria is when BF <.10 or >10. This makes a lot of sense to me. So if I'd had a BF of 13 after 148 subjects, yes I probably would have stopped. I get the sense that people are viewing my post as critical of BF but that is not at all how it was intended. I'm merely being critical of people who run an underpowered experiment (e.g., in a replication), fail to reject H0 and then simply tack on a BF claiming to have found evidence for H0 and leave it at that. E.J.'s analyses clearly show that this is a mistake and our data do as well.
DeleteI didn't think it was overly critical of BFs, just an interesting thought process. I see what you mean now, and it makes sense.
DeleteI agree with you about tacking on BFs to null results thoughtlessly. Model comparison only makes sense if the models are interesting!
I think it is hard for many who were trained in traditional stats to get the idea of absolute evidence out of their head. There is no such thing as "evidence against H0, full stop". It's always relative!
This is exactly what I wanted to convey.
DeleteI'm glad Frick's work is getting the attention it deserves. I covered it in my book and have mentioned it reviews of a couple papers that cover similar ground but didn't seem aware of COAST and CLAST.
ReplyDeleteFrick presents several simulations of the COAST strategy and it is surprisingly hard to break  so it is easy to get Type I error rates a little above .05 using COAST, but under reasonable conditions it seems hard to get COAST to misbehave badly. It will also tend to be more efficient than using a fixed stopping rule.
Thanks, Thom. I tweaked Ryne Sherman's phack program to simulate COAST. It kept alpha <.05 (<.04, actually), so I also find it quite robust.
DeleteIt seems to me that your question remains unanswered, it is just rephrased as: p=.003, what now? You could also rephrase it as: BF10=4.11, what now?
ReplyDeleteIf there is a consequence to your investigation, then it should be clear what to do next. (e.g., you run the followup study, you decide the effect is too weak to be the basis of a master's thesis, you change your theory, or you advise the justice system to discount eyewitness testimony.) You could have answered any of those questions after batch 1 with the data that was available then. If there was no answer after batch 1, then I cannot see how there can be an answer after batch 7.
If there is no consequence to your investigation, then you simply report what you found and move on; perhaps with the hope that the data will help someone else in the future. Those future researchers might thank you for running the additional subjects, or they might feel it was overkill.
These are fair points. I'm using the data to illustrate some points. As I say in the post, I haven't talked (yet) about the contents of the study. This is the level at which consequences come in. These consequences are mostly theoretical in nature. There are aspects to the conceptual replication that have likely led to a reduction of the effect. As I said above, this is the topic of a different post.
DeleteI think these are really the same issues. If the experiment provides data to help you understand a theoretical issue, then I think you have to accept the answer that is given by the data you have. If the answer is insufficiently definitive, you should feel free to gather more data until the answer becomes sufficiently definitive (whatever your criterion). So, the answer after batch 1 might tentatively be that a model assuming no effect better accounts for the data than a model assuming an effect. It's a tentative conclusion because you realize that more data might change it and you might develop a superior model (e.g., some interaction effect) in the future.
DeleteMore or less the same attitude applies after batch 7. You should tentatively conclude that a model assuming an effect better accounts for the data than a model assuming no effect. It's a tentative conclusion because it is (still) the case that future data might change it and you might come up with a better model.
Various criteria such as p<.05 or p<.01for COAST, or BF10<1/3 or BF10>3, really do not enter into these conclusions at all. They are just rules of thumb that seem to be useful for some common situations. At the end of each day, you are still just drawing tentative conclusions with an appreciation that they might (hopefully will, as new models are developed) change in the future.
When does it end? When practical constraints get in the way. If you can easily get large data sets (e.g., using MTurk or lots of funding), then maybe Ntotal=500 (or 5000) is doable. If you are constrained to Ntotal=50, then you give up on this line of research because you do not have the resources to investigate it (or you do a small study and convince others to do the same and then pool the data together).
I think it is great that you discuss these issues on your blog.
I think we are in agreement. The assessment after batch 1 was that we had inconclusive evidence. Our tentative conclusion after batch 7 is that we have evidence for VO. In a future post I want to address the conceptual replication aspect of this study. Bayesian statistics will be helpful here.
DeleteI'm glad that you like the discussion. I'm learning a lot from this comments section.
There is a crucial difference between pvalues and Bayes Factors in this context: if you keep running more subjects with BF, you are either going to converge towards 'relative evidence for H0' or to 'relative evidence for H1' (see Felix SchĆ¶nbrodt's reply). However, if you keep running more subject with NHST, at some point it will be significant, unless your subjects are collectively fooling you by using a very good random generator to generate their responses. So while the BF gives you a clear guideline on what to do, the pvalue leaves a large part of the decision on whether to stop and claim some effect (or the lack thereof) to you. I admit that I am not familiar with COAST, but my instinctive questions are: a) does the COAST method solves this paradox? and b) how does this method justify claiming support for H0 on the basis of a pvalue?
ReplyDeleteI had the same instinctive questions you have. My answers would be: a) If your pvalue is >.36, you're out (you're TOAST;)); this could happen early on in the proceedings. b) This method does not allow you to claim support for H0, just that you've failed to reject it.
DeleteThis is a nice post. I just ran another simulation of this method. The simulation is of a basic betweensubjects ttest with a d of 0 (no effect). I generated 2 groups of 20 subjects then ran a ttest. If the p value was below .01 or above .36 the simulation would end, else, I'd get another 40 subjects (20 per cell) and run another ttest. This continued until one of the 2 thresholds was achieved (<.01 or >.36). I repeated this process 100,000 times.
ReplyDeleteThe mean n (per cell) was just under 49, which seems logistically reasonable. The max n (per cell) was 48,240. Which means that this procedure would require us, sometimes, to run 100,000 participants. Logistically, this is not so reasonable.
Interesting results. I agree that running 100,000 subjects is usually not feasible. What do you think of the solution proposed by replicationindex above?
DeleteWell, it looks like replicationindex's concerns about alpha inflation turn out to not be problematic (in the situations that I simulated: a single betweensubjects ttest). As for the biased effect size concern, I modified my script so that I could modify the actual difference between the groups I sampled from (i.e., so I could change the population effect size). I then ran your algorithm using a max N of 100 per condition. Here's a graph of the results: https://goo.gl/Ae1Vmf
DeleteThe black line is the observed d (the mean difference between groups for all 100,000 simulations) for each population d. The red dashed line is what the observed d would look like if it perfectly mirrored the actual d. So, it looks like theres some bias (underestimating with a small d and overestimating with a large d), but it's not dramatic (~.05). Here are the actual values:
population_d = .3
observed_d =.251
population_d = .4
observed_d = .374
population_d = .5
observed_d = .502
population_d = .6
observed_d = .630
population_d = .7
observed_d = .747
population_d = .8
observed_d = .857
population_d = .9
observed_d = .959
population_d = 1
observed_d = 1.05
Interesting. I just ran a few more simulations. It was the same procedure as in my earlier post, with one more stopping rule: if max n (per cell) was greater than x, the data collection would cease. Here are the FA rates for various values of X:
ReplyDeletemax n = 50 per cell: FA rate = .0475
max n = 100 per cell: FA rate = .0391
max n = 200 per cell: FA rate = .0369
max n = 500 per cell: FA rate = .0352
This is surprisingly good news. We're appropriately controlling our FA rate even with only 50 people per cell, which is logistically doable.
Here's a link to my R code: https://www.dropbox.com/s/u4b63mrqnuqj7i4/Frick.R?dl=0
Great! l'll take a look at the R code.
Delete