II. JAPAN
Japan ODA Evaluation Division/Ministry of Foreign Affairs (MOFA) Evaluation Department/Japan International Cooperation Agency (JICA) The Ministry of Foreign Affairs (MOFA) and Japan International Cooperation Agency (JICA) are responsible for development co-operation in Japan. Each organisation has a division or department that is responsible for evaluation activities although there are interlinkages between them. Central/main evaluation units
Programme/operational units
Other units with evaluation functions
Reporting line
Development Project Accountability Committee
ODA Evaluation Division Conducts ODA evaluations at the policy and programme levels
Lines of communication
Japan International Cooperation Agency (JICA) Management Board
International Cooperation Bureau (MOFA)
Minister's Secretariat (MOFA)
High level policy groups or ministries
Advisory Committee on Evaluation
Evaluation Department Conducts evaluations mainly at the project level
ODA Evaluation Division, MOFA Evaluation Mandate The ODA Evaluation Division conducts and administers evaluations for development co-operation activities provided by MOFA based on the Order for Organization of MOFA. The Development Cooperation Charter which was approved by the Cabinet in February 2015 emphasises the importance of evaluation to implement effective and efficient ODA, stating: “In the light of the importance of evaluation not only for improving effectiveness and efficiency but for accountability to the public, Japan will conduct evaluations at the policy and programme/project levels and feed the results back to the decision-making and programme/project implementation processes”.
Responsibility and scope of activities MOFA implements evaluations at the policy and programme levels. MOFA conducts 1) policy-level evaluation, e.g. country/regional assistance and priority issue evaluations, and 2) programme-level evaluation, e.g. aid modality and sector programme evaluations, by contracting external consultants. The ODA Evaluation Division supervises the consultants contracted externally and ensures the evaluation reports are prepared in a consistent manner, monitors the progress, and provides support during the desk and field study.
EVALUATION SYSTEMS IN DEVELOPMENT CO-OPERATION: 2016 REVIEW © OECD 2016
151
II. JAPAN
Organisational Structure and Reporting Lines In 2011, the ODA Evaluation Division was relocated to the Minister’s Secretariat from the International Cooperation Bureau which is in charge of ODA policies. Before the restructure, the Evaluation Division was a subsidiary of the Aid Policy and Management Division of the International Cooperation Bureau. The ODA Evaluation Division prepares an annual ODA evaluation plan together with the operational units and this plan is subsequently approved by the Deputy Vice-Minister of the Minister’s Secretariat. The plan is reported to the Development Project Accountability Committee held by the International Cooperation Bureau of MOFA. The Committee was established in 2011 and plays a central role as the feedback mechanism in Japan to discuss new ODA projects. The Committee consists of external experts from NGOs, the private sector, academia and the press. The Director of the ODA Evaluation Division reports evaluation results to the Deputy Vice-Minister of the Minister’s Secretariat, the Director-General of the International Cooperation Bureau within MOFA, as well as to the Development Project Accountability Committee, as illustrated in the chart above. The ultimate responsibility rests with the Deputy Vice-Minister of the Minister’s Secretariat. After evaluation lessons are disseminated to the ODA policy makers and relevant implementing divisions of MOFA and JICA, response measures for the evaluation recommendations are developed through consultations of the Internal Follow-up Meeting on ODA Evaluation held by the ODA Evaluation Division. MOFA prepares the Annual Report on Japan’s ODA Evaluation that contains a summary of the findings and recommendations, response measures and the implementation status of these measures.
Types of Evaluation • Sector-wide evaluations • Programme evaluations • Country evaluations • Policy/strategy evaluations
Having used the OECD DAC criteria as reference, MOFA developed the following three: relevance of policies, effectiveness of results, and appropriateness of processes. Moreover a new evaluation criterion, referred to as the diplomatic viewpoints, has been added recently, enabling MOFA to assess the extent to which ODA is furthering Japan’s diplomatic interests.
Snapshot of evaluation resources Japan MOFA
Head / Director / Assistant Director (one of them also works as a professional evalaution staff)
Administrative / Support staff
Resources There are nine full-time employees in the ODA Evaluation Division. One of the Directors doubles as a professional evaluation staff. The budget of ODA evaluation is funded within the operational budgets of MOFA.
152
EUR 942 000 0.03% of the ODA budget 6-10
Evaluation produced per year (5-9 centralised evaluations and 1 decentralised evaluation)
II. JAPAN
The budget of the ODA Evaluation Division was reduced thus the Division selects themes that are aligned with Japan’s core competencies to develop the ODA evaluation reports. The ODA Evaluation Division conducts a mix of centralised evaluations and decentralised evaluation (led by partner governments or organisations) each year. This combination helps to improve partner country evaluation capacity.
Principles of Evaluation Independence The relocation of the ODA Evaluation Division contributes to its strengthened institutional independence. The method of selecting the Director of the Division changed in 2011 and the post is filled through an open recruitment procedure. The term is two years with no possibility for him/her to take up other positions.
Competence and capacity building The skill constraints of the internal evaluation staff are dealt with in occasional study sessions. The Division appreciates involvement of certificate holders of the Japan Evaluation Society when selecting external consultants.
Transparency and participation In order to improve the “visualisation” of evaluation effectiveness, MOFA has introduced a rating system for policy and programme-level evaluations, which is assessed by external experts. The system aims to improve the quality of the evaluation and is also used in JICA. The Division also sends summarised versions of the ODA evaluation reports to the embassies of partner countries. Since 2001, MOFA has been hosting ODA Evaluation Workshops and promoting understanding of the evaluation issues and methodologies in the Asia-Pacific region to improve evaluation capacity.
Co-ordination with donors and country recipients One joint evaluation has been conducted in FY 2015 with the National Economic and Development Authority (NEDA) of the Republic of the Philippines. Joint evaluations are considered to be a constructive exercise for enhancing development of the evaluation capacity of partner countries, as well as promoting mutual accountability.
Evaluation Department, JICA Evaluation Mandate JICA implements ODA projects, of which all are subject to evaluations. The legal basis of JICA’s evaluations is the Act on the Japan International Cooperation Agency, Independent Administrative Agency, which provides the Evaluation Department with the mandate to execute ancillary tasks of the implementation of Japanese development co-operation. The Evaluation Department is mainly responsible for the planning and co-ordination of project evaluations, the development of evaluation methodologies, and management and implementation of evaluations. The evaluation scope, process, and setup are described in the JICA Guidelines for Operations Evaluation.
EVALUATION SYSTEMS IN DEVELOPMENT CO-OPERATION: 2016 REVIEW © OECD 2016
153
II. JAPAN
Responsibility and scope of activities The Evaluation Department carries out ex-post evaluations in collaboration with external evaluation consultants (centralised evaluation) for the projects over JPY 1 billion, to ensure a more objective assessment. The overseas offices carry out ex-post evaluations internally for projects from JPY 200 million to less than JPY 1 billion with the support of the Evaluation Department (decentralised evaluation). The operational departments of JICA implement ex-ante evaluations for individual projects supported with technical cooperation, ODA loans, and grant aid. This division of tasks ensures that the knowledge of project details and contexts stored in operational departments or overseas offices is incorporated in the evaluation reports. The Evaluation Department itself supports the decentralised evaluations by commenting on drafts of evaluation reports and training JICA staff.
Organisational Structure and Reporting Lines The Evaluation Department is one of the headquarter divisions in JICA. The Head of the Evaluation Department reports directly to the Management Board of JICA. JICA established the Advisory Committee on Evaluation in 2010 to enhance the quality of evaluations and strengthen feedback of evaluation results. It meets twice a year and advises on further improvement of JICA’s operations evaluations. The Chair of the Committee is an independent external expert who is supported by professionals appointed by the President of JICA. The evaluation work plan covers up to one year and is first drafted by the Evaluation Department and approved by the Management Board. The results of evaluations are reported by the Evaluation Department to the Management Board.
Types of Evaluation • Project and programme evaluations • Impact evaluations • Thematic evaluations
The evaluations are conducted in line with the OECD DAC criteria. For the past five years, 1 262 ex-ante, 795 expost, 10 impact, and 12 thematic evaluations were conducted. The number of ex-post evaluations is expected to increase. Mid-term reviews are carried out, but often considered as part of project monitoring management.
Resources JICA’s Evaluation Department is staffed by 29 full-time employees. The budget of the Evaluation Department is independent from other departments. Budget and staffing increased due to the increase in the number of evaluations that were required.
Snapshot of evaluation resources Japan JICA
Head / Director / Assistant Director
Professional evaluation staff
Administrative / Support staff EUR6 571 000 0.07% of the ODA budget 126
Evaluation produced per year (2014)
154
II. JAPAN
Principles of Evaluation Independence The Head of the Evaluation Department reports to the Management Board and safeguards an independent reporting line. The Evaluation Department itself is impartial of the operations. The use of external consultants for large-scale projects also contributes to independence.
Competence and capacity building The Evaluation Department provides training programmes on the development of M&E indicators for JICA staff. The evaluation staff also participate in seminars and lectures for basic and more advanced evaluation expertise.
Transparency and knowledge management JICA has developed an online knowledge website, which provides access to information on projects, know-how and lessons learned per sector. The Evaluation Guideline has continuously been revised for further improvement and the current version is the second edition. Guidelines are available on the JICA website, as well as the Annual Evaluation Reports and the Standard Indicator Reference, to improve JICA employees’ capability. When projects are formulated, lessons learned from past projects are taken into consideration. The Evaluation Department also holds seminars for other departments to inform them about evaluation results. The Development Project Accountability Committee holds a meeting at MOFA, which is attended by the Evaluation Department when new JICA projects are discussed. Moreover, JICA and MOFA hold a Collaborating and Learning Meeting once or twice a year in order to share information.
Co-ordination with donors and country recipients A few operations evaluations are co-implemented with both multilateral and bilateral donors and government partners. JICA exchanges information with their counterpart evaluation units, which facilitates mutual learning and improvement of evaluation skills.
Quality assurance As the internal evaluation is conducted jointly by the Evaluation Department and field offices, validation of the internal evaluations is not practiced. The rating system mentioned above has also been adopted for the evaluations conducted by external evaluators.
Note to reader: The section at the beginning of Part II entitled “Introduction and key for the member profiles” provides explanatory notes on the profiles.
EVALUATION SYSTEMS IN DEVELOPMENT CO-OPERATION: 2016 REVIEW © OECD 2016
155