II. LUXEMBOURG
Luxembourg Monitoring, Audit and Evaluation, Quality Control Division, Ministry of Foreign and European Affairs (MFEA) Expertise and Quality Directorate, Luxembourg Development Cooperation Agency (LuxDev) Evaluation Mandate In 2015, a new overall Evaluation Policy was developed and launched by the Directorate for Development Cooperation and Humanitarian Action at the Ministry of Foreign and European Affairs (MFEA) in consultation with Luxembourg Development Cooperation Agency (LuxDev), the Ministry’s implementing agency for development co-operation. The Policy is an overarching framework for the entire evaluation system of Luxembourg’s development interventions, determining key objectives and priorities. The Policy sets out the distribution of evaluation tasks between the MFEA and LuxDev, with the MFEA handling external evaluations and LuxDev delivering internal, independent evaluations. Internal and external evaluations have different but complementary objectives which mean that there is close consultation between the two actors in the evaluation process (OECD 2012). The evaluation mandate of the MFEA is defined in the 2015 Evaluation Policy. This document is complemented by a series of additional internal documents, further detailing and guiding the MFEA’s work in relation to various types of evaluations and evaluation co-ordination and implementation processes. The revision of existing and development of additional resource documents is currently ongoing. LuxDev systematically performs internal, independent mid-term and final evaluations of all its projects and programmes. LuxDev undertakes two types of internal evaluations: self-evaluations, managed by the Operations Directorate and the implementing partners themselves (with a more operational focus), and independent evaluations, managed by the Evaluation and Knowledge Management Department (Expertise and Quality Directorate). The internal, independent evaluations are driven by the Evaluations and Knowledge Management Department, which reports to the Expertise and Quality Directorate (LuxDev 2015). Apart from the general 2015 Evaluation Policy, its work is based on internal policy from 2014, mandating it to identify good practice and lessons for the future, as well as to provide information on the effectiveness of LuxDev’s work, to meet the agency’s accountability obligations.
Responsibility and scope of activities Division of responsibility and types of evaluation performed by the MFEA and LuxDev MFEA
External evaluation (with a mandate for all interventions)
LuxDev
Independent evaluation (of - Mid-term evaluation LuxDev programmes only) - Final evaluation
- External evaluation of all development co-operation interventions financed by the MFEA - Strategic evaluation during or at the end of strategic country programmes - Mid-term reviews of the strategic country programmes - Thematic or sectoral evaluations - Ex-post impact evaluations
Source: (OECD, 2012), http://www.oecd.org/dac/peer-reviews/LUXEMBOURG%20in%20CRC%20template%20April%20 2013.pdf
EVALUATION SYSTEMS IN DEVELOPMENT CO-OPERATION: 2016 REVIEW © OECD 2016
163
II. LUXEMBOURG
Organisational Structure and Reporting Lines Both evaluation departments are institutionally separate from the operations teams. The Head of Evaluations at the Ministry reports directly to the Director of Development Cooperation. The LuxDev Evaluations and Knowledge Management Department reports to the Expertise and Quality Directorate. The Ministry and LuxDev co-ordinate their evaluation programmes in order to assure complementarity. Multi-annual evaluation plans are developed and co-ordinated by the Quality Control Division, based on an internal participatory consultation process within MFEA and subject to approval by the Minister of the MFEA. This plan covers all its actors and instruments and closely co-ordinates the evaluations covering bilateral activities with LuxDev, whose evaluation programme is imposed upon them by their general project programme (MFEA 2015). Central/main evaluation units
Programme/operational units
Other units with evaluation functions
Reporting line
Ministry of Foreign and European Affairs Directorate for Development Cooperation and Humanitarian Action
NGOs Obligation to evaluate own interventions from state co-finance
High level policy groups or ministries Lines of communication
LuxDev
Quality Control Division
Expertise and Quality Directorate
Monitoring, audit, evaluation Conducts independent evaluations of all development interventions (LuxDev, NGOs, bilateral)
Evaluations and Knowledge Management Conducts Internal evaluations of development interventions
Types of Evaluation • Thematic evaluations • Organisational performance evaluations • Sector-wide evaluations • Programme evaluations • Country evaluations • Project/activity evaluations
The work of the MFEA and LuxDev is guided by the OECD DAC criteria for evaluation, which is confirmed in all existing policy documents and guidelines. These criteria form a basis for the work from the evaluation planning phase to the publication of evaluation reports. The MFEA and LuxDev are also systematically assessing thematic and cross-cutting themes, namely good governance, gender, and the environment and climate change as part of evaluations (MFEA 2015).
164
II. LUXEMBOURG
Resources The Evaluation Unit of the MFEA has one full-time employee in charge of organising the evaluations within the Quality Control Division, which is an increase from a part-time employee. The evaluations are conducted by independent consultants, engaged through public tenders. The staff at LuxDev have also been reinforced and the piloting of evaluations has been transferred exclusively to the Expertise and Quality Directorate, away from the Operations Department. This has resulted in greater independence and closer collaboration with MFEA. Two staff (1.5 full-time equivalent) in the Directorate work exclusively on evaluation. Snapshot of evaluation resources Luxembourg LuxDev
Snapshot of evaluation resources Luxembourg MFEA
Head / Director / Assistant Director Head / Director / Assistant Director Professional evaluation staff
Approximately EUR 600 000 0.15% of the ODA budget 5-6
Administrative / Support staff Approximately EUR 600 000 (no centralised budget) 0.15% of the ODA budget
Average evaluations produced per year
18-23
Average evaluations produced per year
Evaluations are separately budgeted for in the MFEA, while LuxDev evaluations are funded from project/programme budgets. The overall budget available amounted to approx. EUR 600 000 in 2015 per agency. Roughly 25-30 evaluations are carried out per year, which is expected to remain the same level in the future.
Principles of Evaluation Independence Both evaluation departments are institutionally separate from operations management. Most of the external evaluators appointed by the MFEA and LuxDev are publicly tendered. The evaluation teams will usually consist of sector specialists and regional experts. Moreover, the Ministry and LuxDev require that local experts be associated to each mission in order to complement the external consultants’ view with their local insight.
Competence and capacity building Internal staff in the evaluation departments are provided general training on evaluations and take part in several international knowledge sharing and capacity building networks. This ensures a better understanding of the challenges faced in evaluations and in project implementation.
EVALUATION SYSTEMS IN DEVELOPMENT CO-OPERATION: 2016 REVIEW Š OECD 2016
165
II. LUXEMBOURG
Transparency and participation Information on the evaluation work of the MFEA and LuxDev is publicised on their respective websites, through executive summaries of all evaluations carried out. The full evaluation reports are distributed internally and to project stakeholders. Reporting on evaluations and results is done via regular meetings with senior management in the Ministry and LuxDev, via distribution to partner countries, and in bi-monthly meetings with NGO representatives and Ministry staff. The MFEA has established close working relationships with NGOs who are obliged to audit and evaluate their own activities from a specific amount of state co-finance for their activities. Evaluations also feed into the Annual Declaration on Development Cooperation by the Minister to the Parliament. At the MFEA, a system for management response has been formalised, ensuring that recommendations are taken into account to improve the quality of future interventions or to address any issues identified.
Knowledge management Since 2010, joint sector expertise units have been established in the MFEA and LuxDev to encourage effective knowledge management among others. LuxDev has developed a database on evaluations and has organised internal thematic or sectoral workshops to share the lessons drawn from the evaluations. The MFEA also has a knowledge management system in place, storing evaluation reports for internal use.
Co-ordination with donors and country recipients Partner country stakeholders frequently participate in designing evaluations and in reference groups for both agencies. They are also invited to define evaluation plans and work programmes. Joint evaluations are carried out to some extent – two such evaluations have been implemented in the past five years. As country programmes are signed jointly with other partner countries, the country programme reviews are per se joint exercises. The MFEA is part of the Multilateral Organisation Performance Assessment Network (MOPAN) for the evaluation of multilateral organisations, including joint assessments.
Quality assurance The Directorate of the MFEA contains an Evaluation and Quality Control Division, which manages external impact evaluations of ODA overseen by the Directorate for Development Cooperation and Humanitarian Action. The Division is responsible for ensuring the implementation of the evaluation policy and programme, monitoring the follow-up process to recommendations made in specific evaluations and functions as an administrative coordinator. LuxDev’s Quality Department has guidelines and templates in place to guide staff in the elaboration of Terms of Reference for evaluation missions, in the briefing of experts and the editing of evaluation reports. Ad-hoc evaluation committees provide additional oversight (OECD 2010, MFEA 2015).
Note to reader: The section at the beginning of Part II entitled “Introduction and key for the member profiles” provides explanatory notes on the profiles.
166