The study was designed to assist the European Commission in carrying out a thorough evaluation of the European Union Agency for Cybersecurity (ENISA), providing evidence-based conclusions and lessons learned on the Agency’s performance, the Cybersecurity Certification Framework (ECCF) and the potential need for changes to the existing Regulation. Established in 2004, ENISA is a European decentralised agency with a mission to ensure a high level of network and information security within the Union and to promote a culture of network and information security among stakeholders. With the adoption of the NIS Directive, the Cybersecurity Act 2019 and the NIS2 Directive, ENISA’s responsibilities were expanded, and it was given a permanent status within the EU’s institutional framework. The 2019 mandate focused on improving the EU’s ability to manage cybersecurity incidents, supporting cybersecurity cooperation within the Union and internationally, raising awareness of cybersecurity risks and best practices, and promoting cybersecurity certification. The ECCF, established by Regulation (EU) 2019/881, aims to achieve a high common level of cybersecurity across the EU and to reduce fragmentation in the internal market through the creation of harmonised EU-wide certification schemes, as set out in Article 46. This framework gave ENISA the role of developing and coordinating candidate certification schemes, facilitating cross-border trade and improving cooperation on cybersecurity between stakeholders. Its objective is to raise the level of cybersecurity across the Union by ensuring that ICT products, services and processes meet specific security requirements to protect data throughout their lifecycle.
The evaluation objectives can be broadly divided into two main categories:
- Objective A: A comprehensive evaluation of ENISA’s performance, including an assessment of its effectiveness, efficiency, coherence, relevance and added value within the EU. This evaluation considers the Agency’s performance, governance, organisational structure and working practices, as well as positioning in the cybersecurity landscape and relative strengths, weaknesses, opportunities, and threats.
- Objective B: An evaluation of the ECCF, including an assessment of its effectiveness, efficiency, coherence, relevance and added value within the EU. This evaluation assesses the framework’s purpose, roles and procedures.
The evaluation used a mixed-method approach that included a review of administrative and monitoring data, including internal documents, a literature review, 65 interviews, of which 52 focused more on ENISA and 13 concentrated most of the questions on the ECCF, a survey programme with 209 responses (of which 70 were on the ECCF), the results of the public consultation, and two workshops on SWOT analysis and recommendations with 26 and 70 participants respectively.
- The interview programme for the evaluation of ENISA and the ECCF involved contacting 182 individuals, resulting in 49 interviews for ENISA and 13 for the ECCF. All interviews included questions on both topics, and 19 stakeholders were selected from survey respondents willing to participate. Four scoping interviews and a focus group were conducted with Agency and Commission participants to refine the questionnaires. For ENISA, 52 interviews were arranged with various stakeholders, including ENISA staff, EU representatives, industry, academia, and Member State officials. For the ECCF, 13 interviews were conducted with 31 interviewees from EU institutions and ECCG/SCCG members, despite difficulties in engaging private stakeholders. Public consultation analysis mitigated any potential limitations from this challenge.
-
The study team identified relevant stakeholders and utilised publicly available contact information, sending personal survey invitations to 856 individuals. This effort resulted in the collection of 209 survey responses. Of these respondents, 70 (33%) were involved with the ECCF and received a specific set of questions to provide feedback on its implementation.
-
The SWOT workshop aimed to conduct a comprehensive analysis of ENISA and the ECCF, based on data collected by the study team. It brought together stakeholders including academic experts, ENISA representatives, and DG CNECT. The workshop focused on understanding the strengths, weaknesses, opportunities, and threats facing the Agency and ECCF, and facilitated the exchange of ideas for potential improvements to the Cybersecurity Act. Key outcomes included verifying preliminary SWOT analysis results through expert discussion, gaining detailed insights into the capabilities and limitations of ENISA and the ECCF, and identifying areas for future enhancements.
-
The recommendations workshop aimed to gather ideas for improving the performance of ENISA and the ECCF, leveraging information collected by the evaluation team and inputs from stakeholders. The study team presented preliminary evaluation results through an interactive presentation and facilitated two parallel collaborative discussions for in-depth analysis and validation. The intended outcomes included familiarising Commission officials and EU Member State representatives with the main preliminary results and validating these through participant engagement. Additionally, the workshop sought to collect ideas on lessons learned and suggestions for future performance improvements of ENISA and the ECCF within the evolving cybersecurity framework.
The study team came across a couple of key challenges:
- Poor performance of the PwC FWC partners. The inputs were underwhelming with grammatical mistakes, AI-generated "empty" arguments. PPMI escalated it to the PWC management who took over some of the tasks but have not completely resolved the problem.
- DG CNECT appeared to be a centralised DG with a strong involvement of the management level. This slowed down the feedback from the Client, which they acknowledged. The project was delayed for 2 months because of this.
- Workshop participants showed rather limited participation. Next time, preventative measures should be applied in advance.
TBC upon the finalisation of the contract.