Sweden Evaluation Profile - Evaluation Systems Review 2016 by OECD-DCD

II. SWEDEN

Sweden Unit for Planning, Monitoring and Evaluation/Swedish International Development Co-operation Agency (Sida) The Expert Group for Aid Studies (EBA) Evaluation Mandate The government bodies in charge of evaluation are Sida and the Expert Group for Aid Studies (EBA), which was established in 2013 to replace the earlier Swedish Agency for Development Evaluation (SADEV). Sida reports to the Government, Ministry of Foreign Affairs (MFA), Department for International Development Cooperation. So does EBA, albeit with a more independent mandate. In addition, the Agency for Public Management (Statskontoret) may at the request of the Swedish government carry out studies and evaluations of the management of International Development Cooperation. The National Audit Office, reporting to the Swedish Parliament audits governmental activities including development co-operation. The MFA has adopted the Guidelines, Methods and Procedures to Handle Relevant Evaluations within the Ministry (MFA 2012), which set out the roles and responsibilities of the various government actors engaged in the evaluation of Sweden’s aid. Most of these actors have evaluation policies, which draw on DAC guidance and good practices disseminated by the DAC Evaluation Network. The MFA’s guidelines will be updated in light of the closure of SADEV and the creation of the EBA. Sida’s Unit for Planning, Monitoring and Evaluation has a dual mandate of supporting the Agency’s various units regarding decentralised independent evaluations of Sida funded programmes and undertaking more independent and strategic evaluations. Sida’s Unit for Planning, Monitoring and Evaluation does not have a single evaluation policy, rather a set of methodologies and guidelines. The Unit’s Evaluation Manual (Sida 2007) highlights Sida’s approach to evaluation concepts and processes. EBA is an independent committee appointed by the Swedish Government and established to evaluate and analyse Sweden’s international development co-operation. It was established based on a decision in the Swedish Parliament that an independent organisation that evaluates development co-operation was needed. Its mandate is to commission, compile, implement and communicate evaluations, analyses and studies on development co-operation (EBA 2013). The group’s mandate is set by the founding Terms of Reference from 2013. A later internal EBA-document on working methods (EBA 2014) and an operational strategy (EBA 2015b) further detail the working procedures for conducting evaluation studies, as well as outline the long-term direction of EBA’s activities and its future work.

Organisational Structure and Reporting Lines Sida’s Unit for Monitoring and Evaluation belongs to the Department for Organisational Development (Avdelningen för verksamhetsutveckling) and reports to Sida’s DirectorGeneral. It operates based on a two-year work plan and has an independent budget. When drafting the work plan, external stakeholders such as operational units, policy makers and senior management are involved in the formulation.

200

II. SWEDEN

The EBA consists of ten members, plus an expert from the MFA (without voting rights). The secretariat of the EBA has seven employees and a budget of SEK 16.3 million. The dayto-day work of the Expert Group is undertaken by a secretariat. EBA has an independent mandate, agenda and budget and shall primarily cover activities conducted within the framework of the development assistance budget. EBA reports to the Government twice yearly on the overall direction of ongoing and planned projects. On one of those occasions EBA submits a report summarising the content and conclusions of the evaluations, analyses and studies published the year before (OECD 2013, EBA 2013). Central/main evaluation units

Reporting line

High level policy groups or ministries

Government of Sweden

Ministry of Foreign Affairs

Sida

EBA

Department for Organisational Development

Unit for Planning, Monitoring and Evaluation

Types of Evaluation Sida

EBA

• Project/activity evaluations

• Thematic evaluations

• Programme evaluations

• Organisational performance evaluations

• Thematic evaluations

• Sector-wide evaluations • Country evaluations • Policy/strategy evaluations

EBA can in practice initiate any kind of evaluation in the framework of the development assistance budget, however, it will normally not initiate project, activity or programme evaluations.

EVALUATION SYSTEMS IN DEVELOPMENT CO-OPERATION: 2016 REVIEW © OECD 2016

201

II. SWEDEN

Resources As of mid-2016, Sida will have a staff of five full-time employees and the Head of Unit working part-time on evaluations. This is an increase since 2014-15 when three full-time staff were employed at the agency. While Sida’s centralised evaluations are funded from a separate budget line (but delegated from an operational unit), all decentralised evaluations are funded from the operational budget. The budget for centralised evaluations was approximately EUR 600 000 in 2015. Sida has gradually reduced the number of centralised evaluations undertaken and from 2016 resources will be focused on increasing the quality of decentralised evaluations. Approximately EUR 4 million is spent on decentralised evaluations annually. EBA has seven employees and a budget of EUR 1.6 million. Evaluations have been carried out since the group’s establishment in 2013. Snapshot of evaluation resources Sida

Snapshot of evaluation resources EBA

Head / Director

Professional evaluation staff

0.5 Administrative / Support staff

Administrative / Support staff

EUR 600 000 0.03% of the ODA budget

EUR1 600 000 0.04% of the ODA budget

Average evaluations produced per year

Principles of Evaluation Independence Swedish aid is evaluated by several different government bodies, each with individual roles and responsibilities. In Sida, the Unit for Planning, Monitoring and Evaluation is a semi-independent evaluation function that has been integrated into line management as a unit under the Department for Organisational Development. The central evaluation plan is decided by Sida’s Director-General. All of Sida’s evaluations are done by external experts. The EBA benefits from what is labelled a “double independence”. Firstly, although it is a government committee, it works independently from the government and thus chooses independently what issues to evaluate. Secondly, once commissioned by EBA, the authors are independently responsible for their analysis, conclusions and recommendations. The Expert Group primarily assesses the quality of the analysis, and whether the conclusions and recommendations are reasonably founded.

Competence and capacity building The EBA ensures participation for its staff in conferences, seminars and meetings with other evaluation experts, as well as participation in the international discussions around evaluation, evaluation policy and methods. Sweden is one of the main funders of CLEAR

202

II. SWEDEN

(Centres for Learning on Evaluation and Results), a World Bank programme which has established regional centres to promote evaluation capacity building.

Transparency and participation Sweden documents communication with partners through its Open Aid website and make evaluation reports available on Sida’s publication database. Decentralised evaluations are all published but not actively disseminated. From 2016, the intention is to improve active dissemination of findings on conclusions from selected decentralised evaluations. All the EBA reports are published on its own webpage www.eba.se (and also on Openaid. se and biståndsdebatten.se). As a general rule, all EBA reports are publically disseminated at least once. Stakeholders from the aid community (Sida, MFA and civil society) also participate in the development of the evaluation report through the reference groups that are convened for each study. Sida has had a management response system in place since 1999. Management response is compulsory for centralised and decentralised evaluations. The complete reports, including management responses, are always made public and shared with partners (Sida 2015).

Knowledge management Swedish development co-operation is focusing on building a learning culture and is taking action towards improving the incorporation of evaluation results into decisionmaking processes. Sida has taken steps to strengthen its mechanisms for acting on the findings of evaluations. Both EBA and Sida have a database for knowledge management and information sharing. EBA also has an online web portal, where its work is systematically published, and it sends out internal synthesis reports within the organisation. EBA has a strong focus on promoting learning and uptake from studies and evaluations through seminars, media and social networks, like Twitter. The Ministry for Foreign Affairs has Guidelines for the internal handling of evaluations and studies of relevance, such as those of EBA.

Co-ordination with donors and country recipients In year 2015/16 Sida was involved in one joint central evaluation. There is no system for tracking how many of the decentralised evaluations are joint, but most of them are carried out in close co-operation with the evaluated partner.

Quality assurance Quality assurance of Sida´s decentralised evaluations is performed by the responsible Program Officer, assisted on occasion by a reference group. Quality assurance of centralised evaluations is performed by the responsible evaluation manager at Sida in corporation with a reference group. In EBA, a reference group will be set up for each study consisting of experts with the relevant profile for the study that is to be carried out. The aim of the reference group is to provide support to the authors and so increase the quality of the reports. The reference group is chaired by a member of EBA. This member is responsible for monitoring the work on the report and should give recommendations as to whether the report should be published. EBA decides on the chair of the reference group. The chair of the reference group appoints the reference group following consultation with the evaluation author.

Note to reader: The section at the beginning of Part II entitled “Introduction and key for the member profiles” provides explanatory notes on the profiles.

203