Over at Nature they’ve just announced new guidelines for submitted papers. They want results to be more reproducible. I’m guessing where Nature leads other journals will follow.
From next month, Nature and the Nature research journals will introduce editorial measures to address the problem by improving the consistency and quality of reporting in life-sciences articles. To ease the interpretation and improve the reliability of published results we will more systematically ensure that key methodological details are reported, and we will give more space to methods sections. We will examine statistics more closely and encourage authors to be transparent, for example by including their raw data.
I guess this makes sense. For a lot of the genomics papers that get into Nature the article itself is just really a lot of results with a few highlight pictures. The Supplementary then contains extended Methods. Even then it is sometimes difficult from that to really see what has been done. Ensuring that people provide raw data is a no-brainer. They have been supposed to do this for years but some groups are backsliding. A re-emphasis is welcome.
They also provide a checklist which amongst other things has a rudimentary statistical list. This I agree with. The analysis sections of many big papers often seem to elide over far too much. As an R person I’d like to actually see the code that did the calculations – the software settings and parameters even. I actually think the problems with most papers are not in the p-values and error bars – but generally in the normalisation of big datasets, the estimation of null in resampling stats, or just not really having an appreciation of whether you might get different results with slightly different analysis choices.
Then again… it’s getting harder and harder to write/publish papers in big journals. Each year as competition is fierce – more and more “stuff” is requested. Sometimes this is a request for more and better work – sometimes it just feels like more and more box-ticking. So if we are going to have more thoroughness in Methods, Stats, and Data sharing – It would be nice to see a relaxation elsewhere so we can genuinely concentrate on the important stuff. I doubt that will happen though.