These are troubled times in our little frontier town called
Psytown. The priest keeps telling us that deep down we’re all p-hackers and
that we must atone for our sins.
If you go out on the streets, you face arrest by any number
of unregulated police forces and vigilantes.
If you venture out with a p-value of .065, you should count
yourself lucky if you run into deputy Matt Analysis. He’s a kind man and will
let you off with a warning if you promise to run a few more studies, conduct a
meta-analysis, and remember never to use the phrase “approaching significance”
ever again.
It could be worse.
You could be pulled over by a Bayes Trooper. “Please step
out of the vehicle, sir.” You comply. “But I haven’t done anything wrong,
officer, my p equals .04.” He lets out a derisive snort “You reckon that’s doin’ nothin’ wrong? Well, let me tell
you somethin’, son. Around these parts we don’t care about p. We care about Bayes factors. And yours is way below the legal
limit. Your evidence is only anecdotal, so I’m gonna have to book you.”
Or you could run into the Replication Watch. “Can we see
your self-replication?” “Sorry, I don’t have one on me but I do have a p<.01.”
“That’s nice but without a self-replication we cannot allow you on the
streets.” “But I have to go to work.” “Sorry, can’t do, buddy.” “Just sit tight
while we try to replicate you.”
Or you could be at a party when suddenly two sinister people
in black show up and grab you by the arms. Agents from the Federal Bureau of
Pre-registration. “Sir, you need to come with us. We have no information in our
system that you’ve pre-registered with us.” “But I have p<.01 and I replicated
it” you exclaim while they put you in a black van and drive off.
Is it any wonder that the citizens of Psytown stay in most
of the day, fretting about their evil tendency to p-hack, obsessively stepping
on the scale worried about excess significance, and standing in front of the
mirror checking their p-curves?
And then when they are finally about to fall asleep, there
is a loud noise. The village idiot has gotten his hands on the bullhorn again.
“SHAMELESS LITTLE BULLIES” he shouts into the night. “SHAMELESS LITTLE
BULLIES.”
Something needs to change in Psytown. The people need to know
what’s right and what’s wrong. Maybe they need to get together to devise a
system of rules. Or maybe a new sheriff needs to ride into town and lay down
the law.
