United States Evaluation Profile - Evaluation Systems Review 2016

II. UNITED STATES

United States US Department of State US Agency for International Development (USAID) US Millennium Challenge Corporation (US MCC) Evaluation Mandate Three US state agencies are tasked with evaluation responsibilities within development aid assistance; the US Department of State, the US Agency for International Development (USAID), and the US Millennium Challenge Corporation (US MCC).

Under the Department of State, the Office of Foreign Assistance and Resources is tasked with providing technical assistance and overviewing the implementation of the Evaluation Policy (updated 2015) by the individual bureaus. According to the Policy, each bureau manages grant and contract funded programmes in order to develop annual evaluation plans and ensure the evaluation of important programmes. It is compulsory for each bureau to conduct at least one evaluation per year.

In USAID, the Office of Learning, Evaluation and Research is a specialised office located in the Bureau of Policy, Planning and Learning. The Office is responsible for catalysing USAID’s transformation into an effective learning organisation by providing guidance, tools and technical assistance on evaluation to the Operating Units (USAID regional offices, technical offices, oversees missions etc.). Decentralised project evaluations are planned and performed by the Operating Units in the countries, whereas selected thematic and meta evaluations are planned and carried out from the central level. The work is guided by the Evaluation Policy released in 2011, which brought about key changes, strengthening the evaluation system within USAID’s programme cycle: •

Budget requirements: Each Operating Unit should spend 3% of its programme funds on evaluation.

•

Transparency requirements: All evaluations are posted online in full, with only limited exceptions.

•

Requirements for impact evaluations: USAID must conduct impact evaluations for pilot or innovative projects that are testing a new approach that anticipates being expanded in scale or scope.

•

New requirements for project-level evaluations that will examine achievement of higher level outcomes.

The evaluation work of the MCC is guided by a 2012 Policy on Evaluation, which integrates evaluation into the entire project cycle of a programme with a focus on measuring results as well as outcome and impact. The Department of Policy and Evaluation manages MCC’s annual country eligibility process; the development and implementation of threshold me; promotes effective policy improvement and reform; performs economic analysis and monitoring, including rigorous independent evaluations; and provides institutional leadership on interagency engagement.

220

II. UNITED STATES

Organisational Structure and Reporting Lines The US Foreign Assistance has a complex structure with 25 departments and agencies under the umbrella of the State Department. The links among the evaluation units of the three agencies are rather loose. They have individual policies, their own budget and their procedures and methods are not fully harmonised. There is a pending legislation, not yet voted by Congress that aims at standardising policies and methods.

The State Department is the lead representative of the US government overseas and the Secretary of State is the President’s principal foreign policy advisor. USAID´s Administrator reports directly to the Secretary of State. Since 2010 there has been a reform aiming to strengthen USAID’s mandate and capacities. The most significant change in USAID is the re-establishment of a central evaluation office called Office of Learning, Evaluation and Research located in the Bureau for Policy Planning and Learning (PPL/LER). In addition an evaluation point of contact is established in every USAID field mission where the operational staff is responsible for the commissioning and oversight of evaluations.

In MCC the Department of Policy and Evaluation reports to the Office of the Chief Executive Officer that provides strategic direction for the agency; manages the agency’s Investment Management Committee; and oversees overall agency performance and day-today operations. MCC is overseen by a Board of Directors, chaired by the Secretary of State.

Central/main evaluation units

Programme/operational units

Other units with evaluation functions

Reporting line

High level policy groups or ministries Lines of communication

Secretary of State

Office of Foreign Assistance and Resources Administrator Deputy Administrator MCC Board of Directors Chaired by Secretary of State Bureau for Policy, Planning and Learning Office of the Chief Executive Officer

Operations Divisions USAID

Department of Policy and Evaluation MCC

EVALUATION SYSTEMS IN DEVELOPMENT CO-OPERATION: 2016 REVIEW © OECD 2016

221

II. UNITED STATES

Types of Evaluation The State Department

MCC

• Organisational performance evaluations

•

• Sector wide evaluations

•

Performance evaluations Impact evaluations

• Programme evaluations • Policy/strategy evaluations

USAID

• Project/activity evaluations

•

• Thematic evaluations

•

• Cross-cutting evaluations

Performance evaluations Impact evaluations • Cross-sector evaluations • Meta-evaluations

Resources State Department’s Office of Foreign Assistance and Resources has a total of six staff.

As all evaluations are decentralised, the annual Congressional Budget Justification provides an overall budget request for both State and USAID foreign assistance. Information on specific evaluation budgets is not available due to diverse sources of funding.

USAID has 22 employees and a budget of EUR 14 million available, which is primarily used for capacity building of Operating Units. Decentralised evaluations are funded by each Operating Unit, who should aim at allocating three percent of their total programme funds to support evaluations. All Operating Units (approximately 100) have an evaluation pointof-contact.

MCC has a total of 25 staff and an annual budget of approx. EUR 20 million for centralised evaluations. Resources for evaluations activities are included in compact budgets, primarily related to data collection during project execution. Independent evaluators are engaged with resources from the agency’s due diligence budget.

Snapshot of evaluation resources USAID

Snapshot of evaluation resources Department of State

Snapshot of evaluation resources MCC

Head / Director / Assistant Director

Head / Director / Assistant Director

Head / Director / Assistant Director

Professional evaluation staff Professional evaluation staff Administrative / Support staff

Professional evaluation staff

Administrative / Support staff Approximately EUR 14 000 000 for capacity building, crosssector and meta evaluations

Approximately EUR 20 000 000 100

200 Produced 100 evaluations per year Produced 200 decentralised evaluations per year

222

II. UNITED STATES

Principles of Evaluation Independence The Evaluation Policy of the Department of State includes ‘Independency and Integrity’ as one of its three standards for evaluation. The Policy specifies that all “…bureaus should ensure that the evaluators are free from any pressure and/or bureaucratic interference.

Independence does not, however, imply isolation from managers”. In fact, active engagement of bureau staff and managers is encouraged during much of the evaluation process to ensure that the results are implemented.

The USAID Evaluation Policy is in line with the OECD DAC evaluation standards and protects the independence of the evaluation function and prevents conflict of interest by placing a barrier between contractors and evaluators in the sense that the Team Leader must always be external to the agency as wells as the contractors. The independence and integrity of USAID and the MCC is furthermore supported by the Office of the Inspector General (OIG), which is a unit reviewing the integrity of the programmes and operations of USAID and MCC among others, including the work of the evaluation departments. The Office is semi-autonomous, with its own budget and personnel authorities.

At MCC, the evaluation function is located within the Department of Policy and Evaluation. All evaluations are performed by independent entities. The Agency’s M&E policy provides for complete independence in terms of method, content, and dissemination of evaluation findings.

Competence and capacity building In the State Department, the introduction of the evaluation policy also established a joint community of practice, where over 400 members meets monthly and resources and tools are developed to build capacity. To date, more than 150 staff have been trained via two courses.

Since the introduction of the USAID Evaluation Policy in 2011 there has been significant focus on improved evaluation competences and fostering a culture of evaluation in USAID and Operating Units has been a key priority. To date more than 1 600 staff have been trained in evaluation and online learning communities are available.

At MCC, agency resources are used for financing training of M&E staff in evaluation methods. M&E staff also conduct periodic training of operational and programme staff in the principles and practices of monitoring and evaluation.

Transparency and participation The conduct of evaluations is consistent with institutional aims of capacity building and engagement with all partners. The USAID evaluation policy emphasises that findings should be made public, enhancing transparency of the aid programme. To date reports are systematically circulated internally and sometimes also made available on the website, in internal synthesis reports and circulated to external development partners. Likewise, MCC’s principles on dissemination and transparency indicates that all evaluation products (design reports, questionnaires, baseline, interim and final reports, as well as the underlying data) are made public in evaluation catalogue at all stages including at the design stage.

State Department’s evaluations are posted publicly within 90 days of completion.

EVALUATION SYSTEMS IN DEVELOPMENT CO-OPERATION: 2016 REVIEW © OECD 2016

223

II. UNITED STATES

Knowledge management Comprehensive guidance is available for staff from the three agencies on evaluation.

At USAID, two online learning communities have been developed to facilitate knowledge sharing among staff and partners. One is ProgramNet is designed to accommodate programme planning and design work that is procurement sensitive and therefore not public.

The other, Learning Lab, is available to partners and the public to facilitate collaboration and learning among all development actors. Learning Lab includes an extensive evaluation toolkit for staff and partners. Both of these sites have sections on evaluation and areas for sharing learning around topic areas or geographic regions.

Co-ordination with donors and country recipients All three agencies focus on working collaboratively with partners to strengthen their evaluation capacity, facilitate mutual learning and reduce costs. The agencies typically involve stakeholders from country partners to participate in setting evaluation plans and work programmes, designing evaluations and in reference or steering groups. The State Department and USAID actively engage in joint evaluations with other bilateral and multilateral agencies.

Quality assurance The Evaluation Policies of the three agencies set out guidelines for quality assurance of the evaluation process. USAID has furthermore introduced an Evaluation Management Review Process. One objective is to heighten the quality of the evaluations and the evaluation process. A similar review process has been introduced by MCC.

Note to reader: The section at the beginning of Part II entitled “Introduction and key for the member profiles� provides explanatory notes on the profiles.

224