Rigor and reproducibility has been a major topic in biomedical, physical, and social science research as scientists have recognized the importance of thoroughness and independent reproducibility. Beginning January 2019, the NIH began including rigor and reproducibility in grant application instructions and review criteria. There are four main areas of focus:
- Rigor of the prior research
Applicants must describe the strengths and weaknesses of their preliminary data. Elements that should be described include experimental design, biological variables, and authentication of key resources.
- Rigorous experimental design for robust and unbiased results
Applicants must utilize the scientific method to ensure robust and unbiased experimental design, methodology, analysis and interpretation of results. The NIH provides some brief excerpts of scientific rigor here: Scientific Rigor Examples.
The following resources provide excellent tools and guidance for preparing a rigorous experimental design (i.e., sample size calculations, reducing bias, biomedical statistics, etc.):
XLSTAT (Addinsoft) has a useful grid to help you determine which statistical test you should use to analyze your data.
The Experimental Design Assistant (EDA) is a free online tool designed to guide researchers through the design of their experiments, helping to ensure the appropriate animal cohort sizes for their specific scientific objectives, methods to reduce subjective bias, and appropriate statistical analysis.
R For Biomedical Statistics is a simple introduction manual to biomedical statistics using the R statistics software.
StatPages provides links to statistical calculators, online statistics books, tutorials, downloadable software, and related resources.
Many universities with biostatistics departments also provide free consultations for power and data analyses. Utilize your connections and collaborators to find expert statistics advice. The North Carolina State University (NCSU) Department of Statistics provides fee-for-service support.
- Consideration of relevant biological variables
Applicants must consider biological variables, including age, sex, weight, and underlying health conditions. Biological variables must be factored into the research design, analysis, and interpretation of vertebrate animal and human studies. Applicants may propose studies with only one sex if appropriately justified with scientific literature and preliminary data. FASEB Journal has an excellent article on how to incorporate sex as a biological variable.
- Authentication of key biological and/or chemical resources
Applicants must demonstrate validation of key biological and chemical resources, including cell lines, specialty chemicals, antibodies, and other biologics. Since these resources may differ between laboratories and over time and may have qualities that influence the results, the applicant must ensure their research material is appropriately validated. NIH provides several examples of authentication plans for reference.
More information on NIH’s Rigor and Reproducibility requirements can be found on the NIH Central Resource for Grants and Funding Information. Additionally, NIGMS has made NIH Rigor and Reproducibility Training Modules available. These modules, developed by NIH, focus on integral aspects of rigor and reproducibility in the research endeavor, such as bias, blinding and exclusion criteria. The modules are not meant to be comprehensive, but rather are intended as a foundation to build on and a way to stimulate conversations, which may be facilitated by the accompanying discussion materials.