APBN New Site

APBN Developing Site

Reproducibility Issues with Psychological Meta-Analyses

Researchers analyse 33 meta-analysis articles in the field of psychology to understand the difficulty in reproducibility of psychology studies.

According to a new study published on 27 May 2020 in the open access journal PLoS ONE by Esther Maassen of Tilburg University, the Netherlands, and colleagues found that studies in psychology aren’t always reproducible due to the lack of transparency of reporting.

The meta-analytical studies included were published in 2011 and 2012, all had data tables with primary studies, and all included at least ten primary studies. For each meta-analysis, the team searched for the corresponding primary study articles, followed any methods detailed in the meta-analysis article, and recomputed a total of 500 effect sizes reported in the meta-analyses.

Out of 500 primary study effect sizes, the researchers were able to reproduce 276 (55%) without any problems. (In this case, reproducibility was defined as arriving at the same result after reanalyzing the same data following the reported procedures.) However, in some cases, the meta-analyses did not contain enough information to reproduce the study effect size, while in others a different effect than stated was calculated. 114 effect sizes (23%) showed discrepancies compared to what was reported in the meta-analytical article. 30 of the 33 meta-analyses contained at least one effect size that could not be easily reproduced.

When the erroneous or unreproducible effect sizes were integrated into each meta-analysis itself, the team found that 13 of the 33 (39%) meta-analyses had discrepancies in their results, although many were negligible. The researchers recommend adding to existing guidelines for the publication of psychological meta-analyses to make them more reproducible.

The authors shared that Individual effect sizes from meta-analyses in psychology are difficult to reproduce due to inaccurate and incomplete reporting in the meta-analysis. To increase the trustworthiness of meta-analytic results, it is essential that researchers explicitly document their data handling practices and workflow, as well as publish their data and code online. [APBN]