Authors can't be trusted to run their own robustness checks.
In 17 AER papers, only 12/211 robustness checks "fail" with p > 0.05 (white).
In robustness checks chosen by 3rd parties, almost *half* of them fail (blue).
1/
I think this paper is really important. Favoring unbiased estimators can actually increase bias when coupled with selective reporting, and we have lots of reasons to think that economists are doing plenty of this.
Excited to share a new paper with @jfischman, just accepted at JEL.
We argue that empirical research tends to be biased and overconfident due to a weakness in the dominant econometric framework: insufficient attention paid to humans “in the loop” with the research process. 1/
Test score gains fade.
But long-term outcomes like graduation and earnings often persist.
So what does that mean for judging school success?
@drewhalbailey joins The Education Gadfly Show to discuss the fadeout effect.
Listen: fordhaminstitute.org/nationa…
📢 #EdWorkingPapers: @tw_watts, @EmmaRoseHart, & @drewhalbailey analyze 87 RCTs to assess persistence of educational intervention effects & find that fadeout is common. Intervention characteristics explain a small share of differences in persistence.
📄 bit.ly/4aGrXY4
Why do educational intervention impacts fade? Isn't catch-up a good thing? Are sleeper effects real? Does fadeout mean failure?
@drewhalbailey, @tw_watts, and I address these questions & more in an EdNext piece & 4 new working papers!
educationnext.org/why-do-mos…
🚨 New working paper!
How well do people predict the results of studies?
@sdellavi and I leverage data from the first 100 studies to have been posted on the SSPP, containing 1,482 key questions, on which over 50,000 forecasts were placed. Some surprising results below.... 🧵👇
A clear and compelling read on IES. I hope policymakers pay attention to this. There is a very strong bipartisan case to be made for continuing to fund the development, evaluation, and syntheses of evaluations of educational programs and policies.
Helping teachers learn what works in the classroom − and what doesn’t − will get a lot harder without the Department of Education’s Institute of Education Sciences theconversation.com/helping-… via @ConversationUS
One of my favorite IES resources are the practice guides. There are 5 guides with the research about teaching math. Given the two stop work orders I received last night (to stop work on 2 upcoming IES STEM practice guides), I'd download these now.
buff.ly/4hyHeeI
Our new review, with @tw_watts, @EmmaRoseHart, and April Yu, summarizing some recent work on "Learning about Development from Interventions" is published open access at Annual Review of Developmental Psychology. Comments, questions, insults welcome.
annualreviews.org/content/jo…
Last year, Killingsworth, Kahneman, and Mellers published a paper reporting that, for a group of unhappy people, money does not improve happiness.
@dingding_peng, @sewenz, and I wrote separate critical comments of it, which were published today.
This is just to say
I don't believe
your diff-in-diff
Forgive me
the effects
are way too large
plus forking paths
And so many
other things happened
during that period
Little is known, although much is theorized, about the long-run benefits of boosting children's social-emotional skills.
Across 450 post-test & follow-up impacts from 86 RCTs, we found similar fadeout rates for intervention impacts on social-emotional & cognitive skills.
1/11
Spending lots of time in interdisciplinary schools, it is striking how different the “cultures” are in writing and formatting of articles. Each discipline has a more than slightly different way to write the front-end. And not doing it “right” surely leads to a rejection...
Very happy to see our paper published in the @JPubEcon!
We do not find significant reductions in maternal employment after four years of receiving an unconditional cash transfer.
I am excited to share our new paper on the effects of Baby’s First Years unconditional cash transfers on maternal reports of child language & socioemotional development across the first three years of life! Check it out in Developmental Psychology: doi.org/10.1037/dev0001824
Underpowered designs and p hacking in econ should be treated as a scandal. The field has otherwise does a pretty nice job enforcing high methodological standards for the social sciences. But if a field's norms produce this pattern, they're probably the wrong norms.
6/ P-Hacking: They then manually code 150 JMPs to document the relationship between p-hacking and academic placement. JMCs with marginally significant results were more likely to secure academic jobs, especially during the job market of the COVID-19 pandemic.
I suspect economists are over-incentivized to produce sexy findings (like social psychologists, especially 10-15 years ago) and very much under-incentivized to preregister and replicate.