Background
Reproducibility in Science
How do we define Reproducibility in Science? According to The National Academies of Sciences, Engineering, and Medicine, reproducibility in science can be defined as “obtaining consistent results using the same data and code as the original study”. [Source]
In essence, this means that a scientist at minimum should be able to access an experiment’s
- methods,
- data (if provided),
- and code (if provided),
Generally, when we refer to reproducibility, we are referring to the ability of one group to reproduce the work of another research group. However, it is possible and encouraged for scientists to perform internal reproducibility checks to see if their methods, code, and data (where appropriate) are reproducible.
Replicability
Replicability is a concept that is adjacent to reproducibility. In our context, it means “obtaining consistent results across studies aimed at answering the same scientific question using new data or other new computational methods,” according to The National Academies of Sciences, Engineering, and Medicine.
In essence, this means that a scientist with access to an experiment’s methods should be able to run existing methods or analyses on a new dataset.
Computational Reproducibility
Sometimes reproducibility can be further specified. For example, computational reproducibility was generally
defined in a 2009 article by Dr. David Donoho et al. as the provision of data and code to allow others to
reproduce results.
This definition has since become more refined and many tools, guides, and organizations have arisen to support the practice of computational reproducibility and its many challenges; we will continue to learn more about this in this module.