Simmons, Nelson, and Simonsohn (2011) demonstrated how researchers can omit inconvenient details from research reports. For example, researchers may have omitted to mention a manipulation that failed to produce a theoretically predicted effect. Such questionable practices have the undesirable consequence that reported results are difficult to replicate. Simons et al. (2011, 2012) proposed a simple solution to this problem. Researchers who are not engaging in questionable research practices could report that they did not engage in these practices. In contrast, researchers who used questionable research practices would have to lie or honestly report that they engaged in these practices. Simons et al. (2012) proposed a simple 21 statement and encouraged researchers to include it in their manuscripts.
“We report how we determined our sample size, all data exclusions (if any), all manipulations, and all measures in the study.”
A search in WebofScience in June 2014 retrieved 326 articles that cited Simons et al. (2011). To examine the effectiveness of this solution to the replication crisis, a set of articles was selected that reported original research results and claimed that they adhered to Simons et al.’s standards. The sample size was determined by the rules to sample a minimum of 10 articles and a minimum of 20 studies. The R-Index is based on 11 articles with 21 studies.
The average R-Index for the set of 11 articles is 75%. There are 6 articles with an R-Index greater than 90%, suggesting that these studies had very high statistical power to produce statistically significant results.
To interpret this outcome it is helpful to use the following comparison standards.
When true power is 50% and all non-significant results are deleted to inflate the success rate to 100%, the R-Index is 50%.
A set of 18 multiple study articles in the prestigious journal science had only 1 article with an R-Index over 90% and 13 articles with an R-Index below 50%.
Conclusion
The average R-Index of original research articles that cite Simmons et al.’s (2011) article is fairly high and close to the ideal of 80%. This shows that some researchers are reporting results that are likely to replicate and that these researchers use the Simmons et al. reference to signal their research integrity. It is notable that the average number of studies in these 11 articles is about two studies. None of these articles reported four or more studies and six articles reported a single study. This observation highlights the fact that it is easier to produce replicable results when resources are used for a single study with high statistical power rather than wasting resources on several underpowered studies that either fail or require luck and questionable research practices to produce statistically significant results (Schimmack, 2012).
Although it is encouraging that some researchers are now including a statement that they did not engage in questionable research practices, the number of articles that contain these statements is still low. Only 10 articles in the journal Psychological Science that published Simmons et al.’s article make a reference to Simmons et al. and none of these cited it for the purpose of declaring that the authors complied with Simmons et al.’s recommendations. At present, it is therefore unclear how much researchers have changed their practices or not.
The R-Index provides an alternative approach to examine whether reported results are credible and replicable. Studies with high statistical power and honest reporting of non-significant results are more likely to replicate. The R-Index is easy to compute. Editors could ask authors to compute the R-Index for submitted manuscript. Reviewers can compute the R-Index during their review. Editors can use the R-Index to decide, which manuscripts gets accepted and ask authors to include the R-Index in publications. Most important, readers can compute the R-Index to examine whether they can trust a set of published results.