Overview
As we consider reproducibility and rigor to be expected standards for scientific research, we can also introduce other "stress tests" for our work to enhance public trust and validation. Some terms that are used for these additional checks on scientific work include:
- Generalizability
- Reliability
- Robustness
- Bias/Fairness
- ...and more!
Stress Tests on Science
This is not necessarily a call to add in a number of external and internal checks and balances on a project. However, awareness that the "spirit" or intent of the scientific method implies our research to be rigorous and reproducible helps us design projects that fit those criteria. To enhance awareness of this intent, in this module we introduce additional concepts that should inform your design, implementation, and dissemination of your research work.
If you think of your experiment or project as a building, consider these addtional metrics as stress tests on your project. In our context, we can think of a stress test is an examination or metric to determine how well the findings of a project hold up to interrogation. Not every project will be perfectly reproducible, replicable, or generalizable - but you as the researcher can help spread awareness by noting weaknesses you identify as you share your research.
We commonly find notes on potential weaknesses in a "Limitations", "Considerations", or even the "Discussion" section of a paper or talk. Not every publication requires the inclusion of one of these sections, but you can always choose to include your own as a part of your manuscript. By highlighting for others the limitations of your work - or where you think the "stress tests" of science may find issues - you help the scientific community with your intimate knowledge of your work.
Relevance to Bioinformatics
Being a bioinformatician goes beyond knowing how to problem solve with technology and code in the biomedical sciences. It does require knowledge of the processes and ecosystem of science, and how to pratice the scientific method. As a part of this, we are expected to hold ourselves to standards that are not easily communicated.
Adjacent fields often do have pathways to indicate competence - for example, the American Medical Informatics Association (AMIA) provides certification for Health Informatics Professionals (AHIC). The American Health Information Management Association (AHIMA) provides mulitple credentials based on experience and education. As of this writing, there are no professional licensures or certifications for bioinformaticians that signal their competence on an agreed-upon body of work. There are many reasons that underlie this lack of options for bioinformaticians. We should note that the International Society for Computational Biology (ISCB) does include recommendations on competencies for bioinformatics professionals, and some certifications do exist, but largely the knowledge and training required to be familiar with rigor and reproducibility standards is expected to come through research training performed at the undergraduate or graduate level (i.e. in academia).
The Bottom Line
It is up to scientists themselves to uphold standards for rigor and reproducibility through many angles, including but not limited to:
- Reading the acaemic literature with a critical eye
- Participating in peer-review with intention
- Communicating your work clearly with others
- Communicating these expectations with trainees
- Engaging in scientific discourse through conferences, publications, seminars, and workshops
- Sharing transparent and well documented code, data, and methods
- "Stress-testing" your project for strengths and weaknesses