No results...

Evaluation study on Excellent Science in the European Framework Programmes for Research and Innovation

The purpose of this study is to feed into the ex-post evaluation of Horizon 2020, and into the interim evaluation of Horizon Europe. Accordingly, the study will be implemented in two phases: 1) supporting the ex-post evaluation of Horizon 2020, and 2) supporting the interim evaluation of Horizon Europe. Scope-wise, this study covers the area of excellent science under Horizon 2020 and Horizon Europe:

- Phase 1:

  • programme parts: European Research Council (ERC), Marie Skłodowska-Curie Actions (MSCA), European research infrastructures (INFRA), and Future and Emerging Technologies (including Flagships) as well as Spreading Excellence and Widening Participation (SEWP) and Science with and for Society (SwafS)
  • contribution to excellence science across the whole Horizon 2020 programme (including JRC and open science aspects)

- Phase 2:

  • programme parts: European Research Council (ERC), Marie Skłodowska-Curie Actions (MSCA), Research Infrastructures (INFRA), including the European Open Science Could (EOSC) Partnership
  • activities under the horizontal pillar on ‘Widening Participation and strengthening the European Research Area’
  • contribution to excellent science across the whole Horizon Europe programme (including JRC and open science aspects)

The core methods applied to deliver this back-to-back evaluation study include desk research, 200+ interviews, 30 case studies, 8 international benchmark cases, bibliometric analysis, survey and counterfactual analysis:

  • Desk research will be carried out to map and synthesise the evidence and findings of already existing information sources, such as previous evaluations, studies, databases and similar material on the topic of excellence science under Horizon 2020/Horizon Europe. This includes analysing the EC administrative and monitoring data on relevant Horizon 2020/Horizon Europe projects and applications that come in the form of both structured and unstructured data mined from applications and project reports.
  • Interviews will be conducted to (i) map and clarify the policy-making needs, (ii) identify the topics for in-depth inquiries during the case study interviews, (iii) collect the stakeholder opinions and perceptions of intervention results and impacts for both the case study analysis and general reporting/answering the evaluation questions, (iv) identify the gaps in data for assessing the efficiency, and (v) to cross-analyse, validate and supplement the quantitative data.
  • Case studies will complement the existing studies and evaluations and their findings on programme- or action-specific impacts of Horizon 2020/Horizon Europe parts supporting excellent science. The proposed case study programme also includes a set of cross-cutting case studies designed to build on study findings drawn from quantitative data (bibliometric analysis, econometric modelling, network analysis, and survey data) and intended to inform the cross-cutting assessment of excellent science under Horizon 2020/Horizon Europe.
  • Benchmarking will be implemented primarily to provide lessons learnt for future improvements of the FP participation and other best practices of supporting the excellence science. 
  • Bibliometric analysis will establish the profile of the published research funded by the Framework Programme overall and by its different parts (ensured by proposing the same indicators for measuring the short, medium, and long-term scientific impacts across multiple studies feeding into ex-post evaluation of Horizon 2020), and identifying the strengths and weaknesses of EU science compared to the world level. These indicators are an integral part of the Key Impact Pathways (KIP) framework, and specifically those indicators developed under KIP1: “Creating high-quality new knowledge”.
  • Survey and counter-factual analysis will be used to collect and analyse data on the structuring effects and the respective added value of participation in Horizon 2020/Horizon Europe on the participating organisations (compared to organisations that unsuccessfully applied for Horizon 2020/Horizon Europe funding).

In addition, a set of complementary methods and research techniques will be employed to enrich/elaborate and expand the analysis: co-publication and co-patent network analysis, network analysis of the participating organisations, mining of unstructured data, patent analysis and econometric modelling.