What can “finding nothing” – statistically insignificant results – tell us in economics? In his
breezy personal essay, MIT economist Alberto Abadie makes the case that statistically insignificant results are at least as interesting as significant ones. You can see excerpts of his piece below. In case it’s not obvious from the above, one of Abadie’s key points (in a deeply reductive nutshell) is that results are interesting if they change what we believe (or “update our priors”). With most public policy interventions, there is no reason that the expected impact would be zero. So there is no reason that the only finding that should change our beliefs is a non-zero finding. Indeed, a quick review of popular papers
(crowdsourced from Twitter) with key results that are statistically insignificantly different from zero showed that the vast majority showed an insignificant result in a context where many readers would expect a positive result.
and even
It also doesn’t hurt if people’s expectations are fomented by active political debate.
In cases where you wouldn’t expect readers to have a strong prior, papers sometimes play up a methodological angle.
Sometimes, when reporting a statistically insignificant result, authors take special care to highlight what they can rule out.
Of course, not all insignificant results are created equal. In the design of a research project, data that illuminates what kind of statistically insignificant result you have can help. Consider five (non-exhaustive) potential reasons for an insignificant result proposed by Glewwe and Muralidharan (and summarized in my blog post on their paper, which I adapt below).
Here are two papers that – just in the abstract – demonstrate detective work to understand what’s going on behind their insignificant results. For example #1, in Atkin et al. (QJE, 2017), few soccer ball producing firms in Pakistan take up a technology that reduces waste. Why? "We hypothesize that an important reason for the lack of adoption is a misalignment of incentives within firms: the key employees (cutters and printers) are typically paid piece rates, with no incentive to reduce waste, and the new technology slows them down, at least initially. Fearing reductions in their effective wage, employees resist adoption in various ways, including by misinforming owners about the value of the technology." And then, they implemented a second experiment to test the hypothesis. "To investigate this hypothesis, we implemented a second experiment among the firms that originally received the technology: we offered one cutter and one printer per firm a lump-sum payment, approximately a month’s earnings, conditional on demonstrating competence in using the technology in the presence of the owner. This incentive payment, small from the point of view of the firm, had a significant positive effect on adoption." Wow! You thought we had a null result, but by the end of the abstract, we produced a statistically significant result! For example #2, Michalopoulos and Papaioannou (QJE, 2014) can’t run a follow-up experiment because they’re looking at the partition of African ethnic groups by political boundaries imposed half a century ago. “We show that differences in countrywide institutional structures across the national border do not explain within-ethnicity differences in economic performance.” What? Do institutions not matter? Need we rethink everything we learned from Why Nations Fail? Oh ho, the “average noneffect…masks considerable heterogeneity.” This is a version of Reason 4 from Glewwe and Muralidharan above. These papers remind us that economists need to be detectives as well as plumbers, especially in the context of insignificant results. Towards the end of the paper that began this post, Abadie writes that “we advocate a visible reporting and discussion of non-significant results in empirical practice.” I agree. Non-significant results can change our minds. They can teach us. But authors have to do the work to show readers what they should learn. And editors and reviewers need to be open to it. What else can you read about this topic?
AuthorsDavid EvansSenior Fellow, Center for Global Development What does a nonsignificant test result imply?'Non-significance' (p>0.05) is often taken to mean that there is no difference between the treatments, or that the intervention is not effective.
What does nonsignificant mean in statistics?Non-significance in statistics means that the null hypothesis cannot be rejected. In laymen's terms, this usually means that we do not have statistical evidence that the difference in groups is not due to chance.
Does a nonsignificant test result implies that the null hypothesis is true?Interpreting Non-Significant Results. When a significance test results in a high probability value, it means that the data provide little or no evidence that the null hypothesis is false. However, the high probability value is not evidence that the null hypothesis is true.
What does the result is not significant at p .05 mean?A p-value > 0.05 would be interpreted by many as "not statistically significant," meaning that there was not sufficiently strong evidence to reject the null hypothesis and conclude that the groups are different.
|