Storing Reproducible Results from Computational Experiments using Scientific Python Packages

We had a presentation at the 2016 Scientific Computing with Python (SciPy) conference on making computational experiments in Python reproducible. This is an attempt at advancing reproducibility of computational results by storing metadata describing the computational setup along with the results. An accompanying paper has now been published in the conference proceedings. SciPy 2016 was held in Austin, Texas, USA, July 11 - 17, 2016.

Abstract

Computational methods have become a prime branch of modern science. Unfortunately, retractions of papers in high-ranked journals due to erroneous computations as well as a general lack of reproducibility of results have led to a so-called credibility crisis. The answer from the scientific community has been an increased focus on implementing reproducible research in the computational sciences. Researchers and scientists have addressed this increasingly important problem by proposing best practices as well as making available tools for aiding in implementing them. We discuss and give an example of how to implement such best practices using scientific Python packages. Our focus is on how to store the relevant metadata along with the results of a computational experiment. We propose the use of JSON and the HDF5 database and detail a reference implementation in the Magni Python package. Further, we discuss the focuses and purposes of the broad range of available tools for making scientific computations reproducible. We pinpoint the particular use cases that we believe are better solved by storing metadata along with results the same HDF5 database. Storing metadata along with results is important in implementing reproducible research and it is readily achievable using scientific Python packages.

Download

• The original paper is available open access (CC-BY) here, URL: http://conference.scipy.org/proceedings/scipy2016/christian_oxvig.html
• The reference implementation is part of the Magni Python software package which can be downloaded here, DOI: 10.5278/VBN/MISC/Magni
• An extensive example is available here, DOI: 10.5278/VBN/MISC/MagniRE
• Additional supplementary material is available here, URL: https://github.com/SIP-AAU/SciPy2016_material