I just finished reading An estimate of the science-wise false discovery rate and application to the top medical literature. The authors ask how many of what we read is scientific journals is actually incorrect, or false.
Why should anything be false you ask?
I claim a coin is not ‘fair’, and I show how when I toss it 10 times in a row I get all heads, this looks very convincing to back my claim. It becomes less convincing when 20 other teams of researchers submit results from their toss sequences and show the coin is actually fair. But then, they do not submit their results since they are not interesting, see where am I going with this?
In the paper there is an elaborate scraping exercise from published abstracts and some robust statistical methodology, the answer comes out: 14% of what you read is false. In a discussion Benjamini and Hechtlinger argue that we should up this number to about 30%, I am persuaded. This means that out of every three papers you took the time to read, on average, one is just plain wrong, we do not know which one of course. This is done for the medical literature, my personal opinion is that in Economics, it may be even higher.
Well, depressing but what can we do about it? As individuals, we can give more value to working papers, in the past I used to be more judgemental towards those since they still did not pass the ‘judges’, fact a paper is not published yet seems less important now. As a society, we should better promote the exposure of insignificant results. Think of the quality men-power and mind-power that is wasted on testing uninteresting hypothesis only because people have little chance to see someone already tried this and it just does not work. Not to flood editors with “I checked A-Z and nothing comes out”, uninteresting results can be reported in a much briefer format, and should have solid intuition behind both the try and the fail.
