II. KOREA
Korea Evaluation Office/Korean International Cooperation Agency (KOICA) Evaluation Team/Economic Development Cooperation Fund (EDCF) Korea’s development aid is provided through the Korean International Cooperation Agency (KOICA) and the Economic Development Cooperation Fund (EDCF). KOICA is responsible for bilateral grants and the EDCF for bilateral loans. Both agencies have their own evaluation unit. Central/main evaluation units
Programme/operational units
Other units with evaluation functions
High level policy groups or ministries
Reporting line
Lines of communication
Committee for International Development Cooperation, chaired by the Prime Minister Sub-committee for Evaluation on International Development Cooperation
Ministry of Strategy and Finance via Head of EDCF Group EDCF Evaluation Committe
EDCF Evaluation Team (Operations Services & Evaluations Department
Ministry of Foreign Affairs via President of KOICA KOICA Evaluation Committe KOICA Evaluation Office
EDCF Operations Department
KOICA Project Department
Evaluation Office/Korean International Cooperation Agency (KOICA) Evaluation Mandate The Evaluation Office evaluates KOICA’s development co-operation activities. The Framework Act on International Development Cooperation entered into force in 2010 mandates KOICA to prepare an evaluation plan and evaluate achievements and outcomes of ODA projects. The Committee for International Development Cooperation (CIDC) is responsible for preparing guidelines on evaluation of Korea’s international development co-operation. KOICA’s evaluations comply with its «Development Cooperation Evaluation Guidelines» established in 2008 and updated in 2014.
Responsibility and scope of activities The Evaluation Office is responsible for:
156
•
establishing evaluation policies, strategies and plans
•
drawing up evaluation institutions, regulations, and guidelines
•
monitoring evaluation of the project implementing departments
II. KOREA
•
executing evaluations based on the annual evaluation plans
•
studying and developing evaluation methodologies by reviewing international evaluation issues and agendas
•
reinforcing organisational evaluation capabilities
• enhancing partnership with other donor organisations and partner countries. The Evaluation Office undertakes evaluations including ex-post evaluations and checks evaluation progress and consults other departments by laying out the evaluation direction and standards, while project departments administer ex-ante, interim, and end-of-project evaluations. The Evaluation Office also conducts evaluability assessments to select evaluation targets, design the evaluation implementation methods, develop the evaluation questions, and prepare for publication.
Organisational Structure and Reporting Lines The Evaluation Office reports to the President of KOICA, who in turn reports to the Ministry of Foreign Affairs. Both KOICA and EDCF adopt the same reporting structure, where the ministries of each agency report to the Sub-Committee for Evaluation on International Development Co-operation (SC). The SC consists of senior-level officials of the respective ministries, executives of KOICA and EDCF, and civilian experts chaired by the Deputy Minister for National Agenda of the Prime Minister’s Office (PMO). The SC subsequently reports to the Committee for International Development Cooperation (CIDC). The CIDC consists of 25 members chaired by the Prime Minister together with ministers of relevant government departments, and external experts. Every year, the Evaluation Office establishes an annual evaluation plan and convenes the KOICA Evaluation Committee for deliberation. After the approval of the President, the plan is submitted to the Sub-Committee for Evaluation by the CIDC for deliberation and approval. The Evaluation Office implements annual evaluations based on the approved annual plan. Evaluation results are reported and reviewed at the Sub-committee for Evaluation’s regular meeting.
Types of Evaluation • Thematic evaluations • Policy/strategy evaluations • Sector-wide evaluations • Country evaluations • Project/programme evaluations
Over the past five years, KOICA conducted approximately 400 ex-ante evaluations, 250 end-of-project evaluations, 73 ex-post evaluations, 3 impact evaluations. The Evaluation Office mainly conducts ex-post evaluations according to the annual evaluation plan. The Evaluation Office carries out evaluability assessment for every project within two or four years after the completion. Based on the guidelines and standards for selecting projects, the Evaluation Office examines relevance to policies, applicability, usefulness, and evaluation feasibility. The Evaluation Office develops guidelines and standard indicators with the aim of maximising the use of evaluation findings. All evaluations comply with DAC Quality Standards for Development Evaluation.
EVALUATION SYSTEMS IN DEVELOPMENT CO-OPERATION: 2016 REVIEW © OECD 2016
157
II. KOREA
Resources The evaluations conducted by the Evaluation Office and project departments are financed within KOICA’s annual operational budgets. Evaluations conducted by project departments, including exante evaluation and end-of-project evaluations, are not included in the central evaluation budget. The Evaluation Office is staffed by 7 full-time employees. The current human resources are insufficient considering the number of evaluations conducted per year by the Evaluation Office.
Principles of Evaluation
Snapshot of evaluation resources Korea KOICA
Head / Director / Assistant Director
Professional evaluation staff EUR 1 100 000 0.23% of the ODA budget 15-16
Average evaluations produced per year
Independence With increasing attention on development effectiveness, KOICA established an evaluation team in 1996. The evaluation team was reorganised into the Evaluation Office in 2006 and was then elevated to an independent unit placed directly under the president of KOICA in 2011 to ensure the independence of evaluations. This is reinforced through the recruitment of external consultants are hired on a competitive basis for evaluations managed by the Evaluation Office.
Competence and capacity building Evaluation staff usually receive online and on-site training opportunities in evaluation field for capacity development. The Evaluation Office takes the role of reinforcing organisational evaluation capabilities. The staff in project departments are encouraged to participate in training programmes to deepen their understanding of project evaluation. Moreover, evaluation manuals issued by the OECD DAC are constantly translated and published in Korean, in order for the staff to improve their understanding of international evaluation practices and standards.
Transparency The Evaluation Office publishes and disseminates evaluation results to the KOICA headquarters, overseas offices, other government departments and agencies in partner countries. The reports and annual evaluation plan for the following year are also publicly available on the website.
Knowledge management Since 2009, KOICA has operated a feedback system to effectively reflect evaluation outcomes to future projects in an attempt to execute programmes in a more efficient way reflecting upon evaluation results. The Evaluation Office summarises recommendations produced in evaluation reports, assesses their feasibility in consultation with project departments and determines the final response. The Evaluation Office supports the responsible department to map out a feedback plan and monitors execution of the feedback actions.
158
II. KOREA
Since 2013, the Evaluation Office adopted a Project Result Rating System in the ex-post evaluation to indicate the effectiveness of projects. Each project is assessed and rated in the evaluation report for its relevance, effectiveness, impact, efficiency and sustainability in accordance with OECD DAC evaluation criteria. The Office constantly delivers evaluation results to the public in a more objective and comprehensive way with active application of the evaluation rating system.
Co-ordination with donors and partner countries KOICA exchanges information and conducts joint evaluations with the evaluation departments of several bilateral agencies. Recently KOICA evaluated a health project in Vietnam with USAID and a public administration project in Tanzania with JICA. In addition, the Evaluation Office also carries out joint evaluations with other domestic ODA agencies to strengthen their network and share the findings within KOICA and with related institutions. The Evaluation Office requests experts in partner countries to join KOICA’s end-ofproject evaluations. The Office also holds ODA evaluation seminars on a regular basis since 2013 to strengthen the capacity of partner countries.
Quality assurance The Evaluation Office conducts quality assurance for evaluations conducted both by internal and external experts according to KOICA’s ‘Quality Assurance Criteria.’ The Criteria help ensure the quality of evaluations, which is also subject to the OECD DAC evaluation quality standards. The Office makes a concerted effort to assure the credibility and enhance the quality of evaluation reports to ensure findings from reports are valid and meaningful. Evaluation reports can be released only after the approval process of the independent evaluation committee consisting of internal and external members is completed. The Evaluation Office monitors end-of-project evaluations and provides advisory support for project departments upon request.
Evaluation Team/Economic Development Cooperation Fund (EDCF) Evaluation Mandate The EDCF supports partner countries in providing of loans for their economic development. The EDCF’s loans are administered by the Export-Import Bank of Korea under the supervision of the Ministry of Strategy and Finance. The Evaluation Team conducts evaluations for the EDCF activities. The Framework Act on International Development Cooperation is also the legal basis for the EDCF’s ODA activities and provides the EDCF with the mandate to conduct evaluation of its operations. The Evaluation Team in the EDCF Coordination Group conducts evaluations for the EDCF’s activities. It is responsible for: •
planning evaluations
•
managing evaluations
•
verifying the project completion evaluation implemented by the Operations Department
•
providing lessons learned and recommendations obtained from the evaluation reports to the Operations Department and the Planning Department.
EVALUATION SYSTEMS IN DEVELOPMENT CO-OPERATION: 2016 REVIEW © OECD 2016
159
II. KOREA
The primary purpose of project evaluation is to improve quality of the EDCF assistance by incorporating lessons learned and recommendations drawn from similar projects in the past while ensuring accountability and transparency by providing evaluation results to various stakeholders (EDCF 2015a).
Responsibility and scope of activities The Operations Department implements ex-ante evaluations, which is an assessment/ review of important indicators before project appraisal. Project completion evaluations are conducted in the simplified form by checking the consistency between the original implementation plan and the outcome of the project by the Operations Department as well within one year after the project completion. Ex-post evaluations focuses on the results and goals achieved and are performed two years after the project completion evaluation. In order to maintain objectivity, ex-post evaluations are performed by external experts contracted by the Evaluation Team.
Organisational Structure and Reporting Lines The EDCF Evaluation Team belongs to the Operations Services and Evaluations Department of the EDCF Coordination Group. The Head of the Evaluation Team reports to the Head of the Department and the Executive Director of the EDCF Coordination Group. The Executive Director subsequently reports to the Minister of Strategy and Finance. The EDCF Evaluation Committee is headed by the Executive Director of EDCF Group and is composed of heads of related Departments. The EDCF Evaluation Committee examines and deliberates the basic policies and contents of EDCF project evaluations, relevance of evaluation ratings, rigour of evaluation reports and a way to feedback evaluation results. An annual evaluation plan is drafted by the Evaluation Team in collaboration with the Operations Department. The Executive Director of the EDCF Coordination Group approves the evaluation work plan. Then the plan is subsequently submitted to the Sub-Committee for Evaluation at the CIDC for final approval.
Types of Evaluation • Thematic evaluations • Sector evaluations • Country evaluations • Project evaluations
During the past five years, the EDCF conducted 92 ex-ante, 30 ex-post, 6 impact evaluations, 5 evaluability assessment and 7 systematic reviews. The Evaluation Team foresees a significant increase of ex-post evaluations, as many projects will soon be mature enough for evaluation. EDCF evaluations are primarily guided by the OECD DAC criteria. For impact evaluations, the EDCF has however introduced “significance” instead of “impact” on a trial basis. This is designed to measure the EDCF’s contributions to concrete economic development in the partner countries. EDCF has also added new interpretations to the other four criteria. This pilot experience was shared with the OECD DAC Evaluation Network. For other evaluations, the criteria on crosscutting issues, including gender, environment, and human rights are added.
160
II. KOREA
Resources There are five full-time staff equivalents employed in the Evaluation Team. The financial resources have increased to accommodate a growing number of the evaluations to be performed.
Snapshot of evaluation resources Korea EDCF
Head / Director / Assistant Director
Professional evaluation staff
Principles of Evaluation Administrative / Support staff
Independence The Evaluation Team is independent from the Operations Group. The Team reports to the Executive Director of the EDCF Coordination Group and to the Director-General of the EDCF Operations Services and Evaluations Department.
EUR 622 000 0.1% of the ODA budget 10-11
Average evaluations produced per year
Competence and capacity building The EDCF enhances the capacity of evaluation personnel by carrying out evaluationrelated training and conducting joint evaluations with other donor agencies or partner countries.
Transparency and participation Not only evaluation reports and annual reports are publicly available on the website, but also contractual agreements, selection of experts, operational results, and funding resources. At the end of the each year, an annual wrap-up meeting is held to present the evaluation results to the Executive Director and the Director-Generals. The results confirmed by the meeting will be the basis of management responses and turned into long and short-term action plans for project improvement.
Knowledge management As is the case in KOICA, the EDCF shares the evaluation results across the organisation as well as the agencies in partner countries. Evaluation results obtained in joint evaluations are shared with the partner countries through mid-term workshops and discussed further. In 2013, the CIDC constructed an ODA monitoring system, which is a computerised web-based database. Data of all the ODA projects is entered not only by KOICA and the EDCF, but also all other Korean ODA agencies. It provides the information on where and what the each agency is doing and hence make the co-ordination between the two agencies more effective and efficient. The system is still under improvement and the information entered is so far used to build a database, not yet for informed decision making on ODA planning. The EDCF and KOICA have regular meetings to discuss the co-ordinated and harmonised ODA implementation.
Co-ordination with donors and country recipients
EVALUATION SYSTEMS IN DEVELOPMENT CO-OPERATION: 2016 REVIEW Š OECD 2016
161
II. KOREA
The EDCF hosted its first joint evaluation with recipient countries in 2011 and co-hosted the EDCF-OECD International ODA Evaluation Seminar in 2013. Since then, EDCF has been engaged in a number of joint evaluations with recipient countries as well as KOICA. The EDCF has also established regular consultation channels with other ODA agencies.
Quality assurance The Evaluation Team validates the project completion report, which is a form of selfevaluation at the end of the project. The validation conducted by the Team is to improve the quality of the project completion evaluation. The EDCF has also adopted the new quality standard for evaluation quality assurance.
Note to reader: The section at the beginning of Part II entitled “Introduction and key for the member profiles� provides explanatory notes on the profiles.
162