Measuring Impact: the role of the Impact+ Exercise in changing professional and organisational practice

By Kevin Robinson and Steven Murray

The Impact+ exercise has been developed by the UK National Agency for Erasmus+. The purpose is to support applicants and projects funded through the Erasmus+ programme to achieve maximum impact for their project and thereby the programme more widely. The Impact+ exercise is part of an on-going Transnational Cooperation Activity (TCA) and has been evaluated alongside its continued evolution.

Introduction

Early in the Erasmus+ programme, the UK National Agency (NA) identified weaknesses in the applications they were receiving. Among Adult Education and VET applications between a quarter and a fifth of applications were failing the minimum assessment criteria (20–25%). As a result, they were ineligible for funding. The UK NA found that in many of these cases, a poor assessment score for “Impact & Dissemination” was often a contributing factor.

This was perceived as a problem because (i) it left us with a smaller pool of applications to select from and (ii) it made it more difficult to allocate all the programme funding. This second point would become especially important with the increase in funding scheduled in the 2017 Call. The NA felt that some preventative action was required.

Discussions with other NAs indicated that this was a common problem and a common concern. We found that none of the Erasmus+ NAs consulted offered specific, detailed pre-application or post-selection support for impact assessment. A rapid review of materials available showed that while there were lots of materials, these were generally for those with some previous experience of impact or evaluation or they focused on specific techniques or data collection methods (e.g. survey questionnaires). Therefore, we felt that there was a need for some specific entry-level impact materials.

The solution of the UK NA was to design the Impact+ Exercise to help applicants and beneficiaries explore their project impact. What is the exercise? It’s a guided workshop exercise designed to help project partners

Kevin Robinson currently works as Team Leader – Stakeholder Engagement for the UK Erasmus+ National Agency, which is a partnership between Ecorys UK and the British Council. For over 20 years he has worked in the field of education and training, with a particular focus on European initiatives and funding programmes that aim to improve the quality of provision through transnational cooperation. He has been with Ecorys UK for 10 years and is an Associate Director within

KEYWORDS Erasmus+, Impact+,

Q uality

the Programme Management and Communications Division. He works closely with a range of stakeholders as well as counterparts from other countries to ensure the effective implementation of Erasmus+, including overseeing key areas of activity for the UK NA, which include Transnational Cooperation Activities (TCA) and impact assessment.

Steven Murr

ay is a Senior Consultant in the United Kingdom Erasmus+ National Agency (NA) based at Ecorys UK. He is involved in qualitative and quantitative research and evaluation that aims to understand the impact of the Erasmus+ programme in the UK. His research interests include education, culture, social statistics and survey methods.

or staff work through the stages of identifying their impact, working out how to measure it and then how to capture the data. It intends to start the process off showing where the focus for further work should be, rather than to take the user on a complete impact assessment journey from start to finish. Before official launch, we tested the Exercise with an international group of beneficiaries.

The Impact+ Exercise

The Exercise is designed to be: a. Flexible: you can do it as a group, alone or with participants; b. Generic: it’s not Erasmus+ specific – you can adapt it for other projects or activities; c. Low-tech: you don’t need any expensive equipment to do it; d. Simple: impact assessment is really a very straightforward process – designing indicators and data collection strategies can get complicated but in essence, it’s a simple concept.

Within the NA, we’ve tried to integrate the Exercise into our work. We promote it as one source of help, not the only source. There are other materials available, like Logic Model and Theory of Change guidance, and these will be more suitable for some people.

There are several specific activities we’ve done: → Presented the Exercise in leaflets at our Erasmus+ Information Days (pre-application); → Included it in our guidance for applicants (pre-application) and guidance for projects (post-selection); → Included it on the agenda at our Start-Up Seminars (post-selection); → Included workshop sessions at our events, e.g. Annual Conference,

Learning Networks (pre-application & post-selection); → Included it in our communications activities – social media, blogs (pre-application & post-selection).

We’ve promoted the translation of the materials into other languages and have supported other NAs where we can. The Impact+ Tool is available in 8 languages with 2 more expected. The video guide is subtitled in English, French and German. This helps partners of UK-led applications and projects as well as other NAs. We’ve trained almost 20 NAs to use the Exercise and we know that at least 7 NAs have already run their own Impact+ Exercise workshops with beneficiaries.

Measuring success

How are the UK NA assessing the impact of Impact+ Exercise? We’re using the Impact+ Exercise itself. We’ve been applying it to our Transnational Cooperation Activities (TCA) work and it has proved useful there too.

→ Impact: improved quality of Erasmus+ applications and projects. → Indicators: - Year on year change in Impact & Dissemination score at application assessment stage; - Year on year change in no. / % of applicants passing the minimum Impact & Dissemination criteria at application assessment stage; - Quality assessment rating of interim / final project reports; - Quality of project impact assessment activity / materials; - No. of good practice case studies for impact; - Changes in organisational and staff approaches to impact assessment. → Data sources: - EPlusLink (assessment scores); - Interim & final project reports; - Project evaluation / impact reports; - Training event feedback (where Impact+ Exercise is used); - Qualitative research with training participants.

Preliminary results

Is it working? For many of the indicators it is too early to say. There are a limited number of projects that have finished and provided their final reports and evaluation reports to the NA. But there are some indicators that are still to be reported on. Specifically the changes in application quality.

So far the UK NA have looked at the difference in KA2 VET and Adult Education projects between 2016 and 2017. The Impact+ Exercise has only been promoted to the 2017 Call applicants. At the moment, this shows no difference in the quality of applications or in the proportion passing the minimum quality criteria.

Indicator

Average “impact” score (out of 30)

% passing minimum “impact” criteria Field

VET AE VET AE

2016

17.4 18.3 84% 85%

2017

17.5 18.0 83% 81%

However, the qualitative research with beneficiaries who attended the pre-launch test event shows that there are some benefits and improvements from using the Exercise. → The majority of respondents felt that the workshop had been helpful or very helpful in assisting them in considering the impact of their project;

→ A third of respondents had already used the materials in their current Erasmus+ project either at the start or during implementation; → Almost all respondents would recommend the materials to colleagues or partners and many already had done so; and → Some respondents reported observable positive changes in their personal and organisational understanding of and approach to impact assessment.

“I used the Impact+ methodology for writing the project to clarify for myself and the group I’m working with, the staff. We used this tool to clarify what was our position towards public local policies and partners as well to potential partners that we didn’t consider at the beginning; and also what we would like to develop in terms of internal competencies, what we are lacking now and what we would like to develop internally.” – Impact+ Exercise workshop participant.

Conclusions

The UK NA believe that it is too early to say whether the Impact+ Exercise is having the desired impact or not. It is expected that it will be possible to say something more conclusive after the 2018 Call applications have been assessed. In the meantime there are some actions for the UK NA to try and maximise impact:

→ Expand our analysis of assessment scores to all Key Actions and programme fields. We will also continue a text analysis of the impact and evaluation activities proposed in applications; → Continue to embed the Impact+ Exercise in NA support materials and practice. Evidence from the 2017 KA2 Start-Up seminar shows that few projects were aware of the materials; → Continue organising beneficiary training events and research to understand the impact.