Australia Evaluation Profile - Evaluation Systems Review 2016 by OECD-DCD

II. AUSTRALIA

Australia Office of Development Effectiveness (ODE), Department of Foreign Affairs and Trade (DFAT) Evaluation Mandate In late 2013 the Australian Government decided to integrate management of the aid programme into the Department of Foreign Affairs and Trade (DFAT) in order to better align Australia’s diplomatic, trade and development efforts and provide a stronger platform for an improved aid programme. This was followed in June 2014 with the announcement of a new aid policy and performance reporting system. Policies and procedures relating to aid evaluation remained relatively consistent throughout this period, with the exception that Office of Development Effectiveness (ODE) mandate was expanded to include a role in quality assuring the assessments made in DFAT’s annual Performance of Australian Aid report. The ODE is an operationally independent unit within the Department of Foreign Affairs and Trade (DFAT). ODE builds stronger evidence for more effective aid. It assesses DFAT’s internal aid management systems, evaluates the performance of the Australian aid programme and contributes to evidence and debate about aid effectiveness. ODE’s work is subject to the external oversight of the Independent Evaluation Committee (IEC).

Responsibility and scope of activities ODE’s responsibilities span three core areas: •

It carries out performance and quality analysis to test and quality assure DFAT’s internal aid performance assessment system.

•

It conducts its own strategic evaluations with a policy, programme, sectoral or thematic focus.

•

It supports and reviews the operational evaluations commissioned by DFAT programme managers, and on occasion takes the lead in conducting these.

ODE’s strategic evaluations are often across multiple countries, regions or sectors. Operational evaluations typically cover individual activity or regional programmes. DFAT’s current interim policy covering operational evaluations encourages managers to adopt a principle-based approach and to commission an evaluation at the time when it is most likely to be of value from a management perspective. As the central point of expertise on aid evaluation ODE also: •

monitors compliance with DFAT’s evaluation policy and drafts internal guidelines on how to manage evaluations

•

provides technical support to staff managing operational evaluations

•

periodically analyses the quality of completed operational evaluations including examining the robustness of evidence, highlighting findings for wider use and dissemination, and recommending measures to improve evaluation quality and utility.

ODE is redrafting DFAT’s aid evaluation policy, with the objective of encouraging fewer, better evaluations.

EVALUATION SYSTEMS IN DEVELOPMENT CO-OPERATION: 2016 REVIEW © OECD 2016

II. AUSTRALIA

Organisational Structure and Reporting Lines ODE is separate from other divisions in DFAT, with the Head of ODE reporting directly to a Deputy Secretary. Central/main evaluation units

Programme/operational units

Other units with evaluation functions

Reporting line

High level policy groups or ministries Lines of communication

Deputy Secretary of Department of Foreign Affairs and Trade

Independent Evaluation Committee

Office of Development Effectiveness (ODE)

Programme Managers

The IEC was established in 2012 to provide an additional element of independence, quality control and credibility to ODE’s work. It is an external advisory body consisting of three independent members (including the Chair) and one Departmental representative. The IEC reviews and endorses all ODE products, providing independent advice during their production. IEC external members are appointed by the Minister of Foreign Affairs. ODE’s work plan covers two years and is formulated through a process of extensive consultation within and outside DFAT. The work plan is endorsed by the IEC, approved by the Secretary DFAT and published on ODE’s website.

Types of Evaluation ODE’s performance and quality analysis work monitors the department’s aid performance management and reporting systems, and independently quality assures the assessments they produce. Findings from this work support ODE to fulfil its role of quality assuring and verifying the assessments made in an annual Departmental report on the Performance of Australian Aid (PAA). ODE independently analyses the performance assessments made in the PAA to ensure that they are robust. In doing so, ODE helps the department and the Australian Government to assess how the Australian aid programme is delivering against its objectives.

EVALUATION SYSTEMS IN DEVELOPMENT CO-OPERATION: 2016 REVIEW © OECD 2016

II. AUSTRALIA

ODE supports and conducts operational evaluations, which are independent assessments of the performance of Australian aid investments. They provide evidence to inform programme management decisions, contribute to broader learning and help to enhance the accountability of Australian aid. • Thematic evaluations • Orgnisational performance evaluations • Sector-wide evaluations • Programme evaluations • Country evaluations • Policy/Strategy evaluations • Project/activity evaluations

ODE’s strategic evaluations are broad assessments of Australian aid that focus on key policy directions, specific development themes and sectors or large programmes, be they geographic or global. These evaluations usually examine a number of investments and often across multiple countries, regions or sectors. They may also include assessments of aid funds expended by other government departments. By virtue of their scope, these evaluations tend to be time, and resource, intensive. Evaluations would typically cover all elements of the DAC criteria, but also usually address questions that are tailored to requirements of the programme and its key stakeholders.

Resources Snapshot of evaluation resources Australia

ODE has a full time equivalent staff cohort of 14 employees. ODE has its own budget allocation although, at times, operational evaluations are funded by programme areas. Resources are considered appropriate. The budget allocated for centralised evaluation for 2015-2016 is AUD 1.7 million, which is 0.04% of the development budget.

Principles of Evaluation

Head / Director / Assistant Director

Professional evaluation staff

Administrative / Support staff EUR 1 152 000 (AUD 1 700 000) 0.04% of the ODA budget 11

Independence Evaluations produced per year (2015)

Independence is supported through the organisational setup, with ODE reporting directly to a Deputy Secretary (rather than through a Division) and the oversight of the IEC. ODE’s position within DFAT provides it with direct access to aid management systems and personnel, putting it in a good position to understand, and influence, the work of the department.

II. AUSTRALIA

Competence and capacity building ODE employees are considered to have qualified competence and their skills are maintained and constantly improved through training, mentoring and external academic study.

Transparency and participation ODE synthesises, publishes and disseminates lessons from its evaluations via various methods such as newsletters, workshops and seminars. Both operational and strategic evaluation reports, quality assessments, lessons learned, and aid evaluation policy are made available on the website. It regularly co-hosts evaluation forums with the Development Policy Centre at the Australian National University and participates in a range of professional conferences and seminars. DFAT has a formalised management response system where the responses to ODE recommendations are cleared at First Assistant Secretary level and provided within 28 days. Management responses are published as an integral part of ODE reports and ODE monitors and reports on implementation progress at regular intervals.

Knowledge management ODE ensures that programme areas are provided with evaluation tools such as guidance documents, a set of M&E standards and examples of good evaluation products. In collaboration with professional evaluators and organisations, ODE provides ad-hoc courses, seminars and workshops on evaluation for the DFAT staff. ODE is tending to hold recommendation workshops with all key stakeholders prior to finalising its reports.

Co-ordination with donors and country recipients During the past five years, ODE has conducted two evaluations that included aid funding appropriated to other government departments (Australian Centre for International Agricultural Research; Attorney General’s Department and the Australian Federal Police); each of which required close inter-departmental co-ordination and collaboration.

Quality assurance ODE conducts regular reviews of DFAT’s operational evaluations and programme reporting systems. ODE’s most recent meta-evaluation of operational evaluations (DFAT 2014), found the majority of independent evaluations to be credible, however the design of some evaluations and the capacity to manage evaluations had room for improvement. Note to reader: The section at the beginning of Part II entitled “Introduction and key for the member profiles” provides explanatory notes on the profiles.