II. NEW ZEALAND
New Zealand Evaluation and Research Team/Ministry of Foreign Affairs and Trade (MFAT) Evaluation Mandate The last five years have seen a period of significant change for the organisation and management of New Zealand’s development co-operation. In 2009, the management of the New Zealand Aid Programme (the Aid Programme) was transferred from a semiautonomous unit (known as NZ Aid) and integrated into the Ministry of Foreign Affairs and Trade (MFAT). MFAT’s Evaluation Policy (updated in June 2014) sets the core requirements for conducting strategic, programme, policy and practice, and activity evaluations. The policy reflects the OECD DAC Principles of Evaluation for Development Assistance and applies the DAC Evaluation Quality Standards for Development Evaluation. It is supported by operational guidelines, knowledge notes, evaluation resources and staff training. The Evaluation Policy highlights that evaluation: (i) provides evidence to assess whether the Aid Programme is making a difference by using the most effective and efficient methods to support sustainable development; (ii) assesses the distinctive contribution of the Aid Programme to the generation of results and (iii) contributes to a body of evidence and experience to support future policy directions and development practice improvements. New Zealand utilizes evaluation findings for decision-making, learning and accountability purposes. The Evaluation and Research (ER) team operates outside the design, delivery and management of the New Zealand Aid Programme’s development interventions and its planning and reporting functions. The ER team is responsible for managing and delivering the strategic (sectoral, thematic, programme, policy and practice) evaluations. It is also responsible for the implementation of the Evaluation Policy and provides advice and support for activity evaluations. Programme evaluations are commissioned every six to ten years, in line with the programme cycle. Activity evaluations are performed at any point in time during the activity lifetime and/or after completion. These are led by an Activity Manager. Evaluations of activities can be commissioned if the evaluation is deemed to be useful for decision making, learning or accountability. However, they are mandatory where the Aid Programme’s investment exceeds NZD 10 million.
Organisational Structure and Reporting Lines In 2013, the ER team was re-established with new roles and responsibilities. The primary focus was to create a new evaluation function, system, processes and to build an evaluative culture. This included establishing a coherent multi-year evaluation work programme across three co-ordinated levels: strategic, programme and activity level. The intention is that evaluations at the different levels complement and reinforce each other. An independent Evaluation and Research Board (ERB) was also established to provide advice and support to the Aid Programme’s senior leadership team and to the ER team on evaluation and research policy, practice and function. The multi-year evaluation and research work programme is also endorsed by the ERB.
EVALUATION SYSTEMS IN DEVELOPMENT CO-OPERATION: 2016 REVIEW © OECD 2016
171
II. NEW ZEALAND
The Aid Programme’s senior leadership team is responsible for the overall governance and implementation of the evaluation and research programme. This includes consideration and response to the evaluation findings and lessons learned. Central/main evaluation units
Programme/operational units
Other units with evaluation functions
Reporting line
High level policy groups or ministries Lines of communication
Deputy Secretary for International Development
Aid Programme Divisions
Evaluation and Research Board (ERB) Ensures links with planning
Director of Development, Strategy & Effectiveness
External evaluation providers Undertakes strategic, programme & activity evaluations
Partnership, Humanitarian and Multilateral (PHM) Sustainable Economic Development (SED) Pacific Development (PACDEV) Global (GLO) Manage activity evaluations
Evaluation & Research Team (ER) Manages the delivery of the strategic evaluations & research
Types of Evaluation • Thematic evaluations • Organisational performance evaluations • Sector-wide evaluations • Programme evaluations • Country evaluations • Policy/strategy evaluations • Project/activity evaluations
During the last five years, over 55 activity evaluations have been completed. Five strategic sector-wide evaluations and four country programme evaluations have also been delivered. The evaluation programme is reviewed annually to ensure it is responsive to the strategic and operational needs of the Aid Programme. Strategic evaluations are selected to address significant issues (e.g. sectoral, thematic, programme, policy or process) across the Aid Programme. New studies are also identified and the prioritisation of evaluations is discussed based on a set of agreed criteria. These criteria ensure that the work programme is aligned to the Aid Programme’s three year strategic plan and its core priorities.
172
II. NEW ZEALAND
Resources
Snapshot of evaluation resources New Zealand
The ER team has three staff members. Two staff members are focused on evaluation and research; one staff member (0.5 full-time equivalent) is focused on disseminating results evaluation and research findings. The ER team is responsible for:
Head / Director
Professional evaluation staff EUR 727 709 (NZD 1 150 000) 0.5% of the ODA budget
•
the development of the New Zealand Aid Programme’s evaluation programme
•
commissioning research
•
providing assistance to staff commissioning activity evaluations to help ensure they are relevant, robust, timely and useful
•
ensuring that the New Zealand Aid Programme implements good development evaluation practice (through evaluation policy, resources and training)
•
disseminating evaluation results and building an evaluative culture.
strategic
evaluation
and
16
Average evaluations produced per year
The ER team has a centralised and dedicated three-year budget for the execution of its multi-year (2015-2019) strategic evaluation and research programme. This amounts to NZD 8.5 million, which is an increase from the previous three years. A three-year work programme also exists for activity evaluations. These evaluations are funded from the activity budgets.
Principles of Evaluation Independence The ER team operates outside the design, delivery and management of the Aid Programme’s development interventions and its planning and reporting functions. Evaluations are carried out by external specialists who have no vested interest in the outcomes of the evaluation and are independent of those responsible for policy making and for the delivery of development assistance. These external specialists are selected through competitive tenders with transparent criteria. The evaluation governance structures, management processes and quality assurance mechanisms also help safeguard independence, as does involving a broad range of partners and stakeholders in the evaluation process.
Competence and capacity building The Aid Programme’s evaluation learning programme places stronger emphasis on realtime support based around individuals’ needs, which complement scheduled evaluation training courses. This ensures that support is relevant and timely and more directly contributes to the usefulness of evaluations. This is reinforced by evaluation resources available on the Aid Programme’s intranet. Partners are included in evaluation governance groups, promoting capacity development and increasing ownership and usage of the evaluation findings. Partner representation in our programme evaluation working groups is actively encouraged. These groups include subject matter experts. Proposals from indigenous-led evaluation teams or from evaluation teams which include indigenous evaluators are also encouraged.
EVALUATION SYSTEMS IN DEVELOPMENT CO-OPERATION: 2016 REVIEW © OECD 2016
173
II. NEW ZEALAND
Other capacity building initiatives with partners include training workshops, secondment opportunities and the provision of ongoing advice and support to help build evaluation capacity, when and where opportunities arise.
Transparency and participation There is systematic dissemination of evaluation results and lessons. All evaluations are published on MFAT’s intranet and Internet. Evaluation results are reported to senior management and the ER Board. Findings are also shared with partners and stakeholders. Interim findings from the Samoa and Tokelau country programme evaluations were recently shared with Government officials, Apia post, and other key stakeholders. Evaluation findings are also used in a range of reports, including external reports such as the 2012/13 - 2014/15 Triennium Results Report Development that Delivers: Results Achieved (yet to be published) and MFAT’s annual report to Parliament, which includes information from evaluations (OECD 2015, 2010).
Knowledge management In 2013, New Zealand developed a knowledge and results strategy with a five-year time frame. This strategy outlines a series of initiatives to share evaluation, research and results and to build an evaluative culture. This involves a commitment to using the right evidence to inform decision making, planning, strategies, policies and practice. A dedicated knowledge intranet site also makes evaluation resources available to Aid Programme staff, including evaluation briefs and evaluation insights that highlight lessons learned. Evaluation outcomes (as well as experience and good practice on aid management challenges) are communicated through interactive sessions (including ‘brown bag lunches’ and ‘catch up cafes’), seminars and workshops. Other initiatives include regular result and evaluation stories, a synthesis of lessons learned from activity evaluations, and sector synthesis reports that present key themes and lessons learned on each of the Aid Programme’s priority sectors.
Co-ordination with donors and country recipients New Zealand has experience with both joint and partner-led evaluations and has participated in nine during the past five years. The Evaluation Policy encourages partnerled or joint evaluations (where this is considered to be appropriate). The Aid Programme also works in partnership to ensure OECD DAC quality standards apply in order to promote evaluation quality, utilisation and learning. Where partners lead the evaluation, New Zealand uses partners’ evaluation systems and accesses the evidence they generate.
Quality assurance Over the past five years a number of initiatives to improve the quality of the Aid Programme’s evaluation have been instigated. These include:
174
•
using independent evaluation advisors for strategic evaluations to provide specialised evaluation technical and subject knowledge
•
undertaking evaluative assessments as part of stage one of an evaluation - to determine the feasibility of undertaking the evaluation and reinforcing the “stop/go points” to ensure that the evaluation only proceeds when assured of delivery value
II. NEW ZEALAND
•
establishing supplier pools of evaluation providers that will deliver quality work to the standards and quality expected (includes application of the OECD DAC Evaluation Quality Standards for Development Evaluation)
•
undertaking independent quality analysis of activity evaluations against the New Zealand Aid Programme Quality Standards for activity evaluations
•
providing real-time and ongoing advice and support for activity evaluations by the ER team
•
undertaking independent peer reviews of evaluation deliverables.
Note to reader: The section at the beginning of Part II entitled “Introduction and key for the member profiles” provides explanatory notes on the profiles.
EVALUATION SYSTEMS IN DEVELOPMENT CO-OPERATION: 2016 REVIEW © OECD 2016
175