The objective of the study “Assessing the reproducibility of research results in EU Framework Programmes for Research” was to assist the European Commission in understanding, testing and monitoring the progress of reproducibility over time and across the programmes. The study primarily focused on the state of reproducibility during the implementation of Horizon 2020.
To test the extent to which reproducibility-related practices are addressed in EU Framework programmes, the study was built on three key methodological blocks:
- Qualitative/manual assessment of 50 H2020 projects, which analysed how their proposals, DMPs, publications and datasets addressed reproducibility-related factors, as well as how did these outputs referred to the reuse of data.
- Quantitative assessment of 300 H2020 projects, which analysed three key aspects: (1) Assessment of FAIRness: how the datasets produced by these 300 H2020 projects adhered to FAIR principles of Findability, Accessibility, Interoperability and Reusability; (2) Reusability assessment: extent to which publications produced by 300 H2020 projects reused the datasets vs. produced their own datasets; (3) Citation semantic analysis: assessment of the extent to which other publications reused the results produced by publications of 300 H2020 projects and in which context their results were reused.
- Surveys of researchers (H2020 beneficiaries), journal editors and research funders.
- Finally, to further follow up and interpret the opinions gathered through the surveys, and to obtain input for developing the study’s recommendations, two workshops with researchers and one workshop with research funders were organised
The data collected indicates that while reproducibility of research is an important principle to stakeholders overall, the level of implementation of reproducibility-related practices is very mixed depending on the specific reproducibility-related factor. Our findings suggest that the key barriers for reproducibility are cultural factors in the scientific field (e.g. pressure to publish, lack of recognition of reproducibility-related research practices) and the perception that reproducibility is costly in terms of time and resources. Based on these findings, this study provides a comprehensive set of recommendations for research funders to overcome the existing barriers and increase reproducibility of EU-funded research and innovation.