4.3. The SAI of Canada – auditing for accountability and inclusivity

4 – SUPREME AUDIT INSTITUTIONS’ INPUT INTO POLICY EVALUATION AND OVERSIGHT – 139

Box 4.1. The SAI of Canada – assessing programme evaluation in the Federal Government (continued)

Outcomes

The six departments OAG examined followed systematic processes to plan their effectiveness evaluations, and completed most of the evaluations they had planned. However, each department’s evaluations during the audit period covered a relatively low proportion of its total programme expenses: between 5% and 13% annually across the six departments. The actual rate of coverage was even lower because many of the effectiveness evaluations OAG reviewed did not adequately assess programme effectiveness. Departments had often not gathered the performance information needed to evaluate whether programmes were effective. Of the 23 evaluation reports OAG reviewed, 17 had inadequate data, which limited the assessment of programme effectiveness.

The departments OAG examined said that it remains a challenge to find experienced evaluators and that they had made extensive use of contractors to meet requirements. Departments expressed concern about their capacity to evaluate all direct programme spending from 2013, as required by the 2009 policy on evaluation. To ensure full coverage (which includes grants and contributions), departments will have to evaluate an average of 20% of their direct programme spending each year of the five-year cycle.

The Treasury Board of Canada Secretariat has introduced initiatives to address the need for improvements in evaluation across government. However, it did not provide sustained support for effectiveness evaluation. In particular, it made little progress on developing tools to assist departments with the long-standing problem of a lack of sufficient data for evaluating programme effectiveness. With the exception one department that has processes in place to identify needed improvements, the audited departments do not regularly identify and address weaknesses in effectiveness evaluation. The audit report included recommendations related to several of the findings summarised above. The criteria for this audit were set out in the audit plan accepted by the audited departments.

Good practices used

Since the beginning of evaluation audits in the 1970s, the evaluation function has had periods of comparative strength and weakness, some of which were documented by OAG audits. Overall, the function has strengthened, partly due to a more coherent policy framework and to capacity building within departments. Audit findings have been taken into account by the responsible central agency as it reviewed and refined its policies on effectiveness evaluation.

Lessons learned

Periodic attention by the SAI to the evaluation function over a 30-year period has contributed to its strengthening.

Further reading

Office of the Auditor General of Canada: Fall 2009 Report, Chapter 1, Evaluating the Effectiveness of Programs: www.oag-bvg.gc.ca/internet/English/parl_oag_200911_01_e_33202 .html.

Office of the Auditor General of Canada: Spring 2013 Report, Status Report on Evaluating the Effectiveness of Programs: www.oag-bvg.gc.ca/internet/English/parl_oag_201304_01_e_38 186.html.

Sources: OECD Survey of Peer Supreme Audit Institutions; further reading links above.

SUPREME AUDIT INSTITUTIONS AND GOOD GOVERNANCE: OVERSIGHT, INSIGHT AND FORESIGHT © OECD 2016

140 – 4 – SUPREME AUDIT INSTITUTIONS’ INPUT INTO POLICY EVALUATION AND OVERSIGHT

Box 4.2. The SAI of Brazil – maturity index for government programme evaluation systems (iSA-Gov)

Objective

The maturity index survey aimed to evaluate the maturity of the Brazilian public administration’s monitoring and evaluation systems. In 2013, the Federal Court of Accounts Brazil (TCU) carried out an audit of 27 agencies belonging to the direct federal public administration with the purpose of characterising their government programme evaluation systems and providing a diagnosis of their capacity to monitor and evaluate programmes.

Type

Performance Audit.

Scope and methodology

The focus of the survey was to identify and measure if public direct administration evaluation systems met the needs of managers and promoted improvements in the provision and supply of public outputs and services. To help with this, the iSA-Gov index was developed. This index sought to quantify the level of institutionalisation of public administration evaluation systems that monitor the performance and results of the programmes, actions and sectorial policies. It aimed to identify the level of maturity of evaluation systems, contribute to their improvement and indicate improvement opportunities.

Multicriteria Methodology Decision Support (MCDA) was used for data analysis and allowed for the construction of the maturity index of the evaluation systems. The evaluation process was conducted with the support of the Center for Advanced Studies of Government and Public Administration at the University of Brasilia (CEAG/UnB).

The maturity level of the government programme evaluation systems, iSA-Gov, expressed public managers’ perception of the adequacy of the mechanisms and instruments employed to demand, produce and use the available evaluative knowledge. The evaluation model consists of four areas of analysis: 1) evaluative demands; 2) production of evaluative knowledge supply; 3) organisational learning capacity; and 4) use.

Twenty-seven ministries answered the survey. Analysis of the collected data showed how and for what purposes entities of the direct administration produce and use information about the performance and outcomes of programmes. For data analysis, preliminarily frequency tables and measures of position (mean, mode, median and standard deviation) were used to describe the general perception of respondents regarding the criteria investigated.

Criteria

Country laws (Decree 7.889/2012 - which established the principles and procedures for the management of the multi-year plan 2012-2015.), Standards and principles.

Resources

The team that participated in this study was composed of two auditors. The planning phase took place between 22 March 2013 and 28 June 2013; the implementation phase took place between 1 July 2013 and 23 December 2014; and the report stage took place between 20 January 2014 and 31 March 2014. Researchers at the Center for Advanced Studies of Government and Public Administration at the University of Braslia provided support to the project.

Outcomes and good practice

The iSA-Gov index observed that the public administration’s mechanisms and instruments used to evaluate systems are partly sufficient to meet the evaluative information demands of management.

SUPREME AUDIT INSTITUTIONS AND GOOD GOVERNANCE: OVERSIGHT, INSIGHT AND FORESIGHT © OECD 2016

4 – SUPREME AUDIT INSTITUTIONS’ INPUT INTO POLICY EVALUATION AND OVERSIGHT – 141

Box 4.2. The SAI of Brazil – maturity index for government programme evaluation systems (iSA-Gov) (continued)

The survey resulted in individualised reports being prepared for each responding agency and a consolidated general report. The general report identified several opportunities for improvement in evaluation and monitoring systems and provided a comparative analysis of participating entities, which allowed the visualisation of different maturity levels of monitoring and evaluation systems in public administration.

The survey also enabled each entity to carry out a self-analysis that was tailored to its own peculiarities, and consider suggestions on how it can contribute to improving the performance and outcomes of government programmes in order to ensure the efficiency, efficacy and effectiveness of public policies.

Work surrounding the maturity index provided a systemic view of the maturity of Brazilian public administration’s monitoring and evaluation systems and enabled the monitoring and improvement of evaluation instruments in public administration.

Lessons learned

This work has shown the importance of a broader and more strategic assessment of public administration. The diagnosis of faults and opportunities for improvement at the strategic level, such as the ability of agencies to carry out monitoring and evaluation of public policies, enables problems in policy implementation to be prevented.

Further reading

http://portal2.tcu.gov.br/portal/page/portal/TCU/english/publications/institucional_publications/ www.tcu.gov.br/Consultas/Juris/Docs/judoc/Acord/20140516/AC_1209_16_14_P.doc Sources: OECD Survey of Peer Supreme Audit Institutions; further reading links above.

Box 4.3. The SAI of Canada – auditing for accountability and inclusivity

There are approximately 700 000 veterans in Canada. By March 2013, Veterans Affairs Canada (VAC) had identified 16 700 veterans with a recognised psychiatric disability, including approximately 12 000 with post-traumatic stress disorder. The number of veterans with a recognised psychiatric disability being assisted by VAC increased by 63% between 2007 and 2013, and is expected to continue to increase. Parliamentary committees in both the House of Commons and the Senate have demonstrated an ongoing interest in programmes and support for veterans, and have issued a number of reports. Media interest in this issue has also been ongoing, with recurring topics including: homeless veterans, which is often linked to mental illness; lack of support for families of veterans; weaknesses of the “New Veterans Charter”; and independent groups that provide support for veterans with post-traumatic stress disorder, such as service dogs, free gym memberships, and an online support group.

Objective

The OAG undertook an audit to determine whether Veterans Affairs Canada facilitates timely access to services and benefits for veterans with mental illness. The OAG responsibility associated with this audit was to conduct an independent examination of the management of programmes and resources that aim to facilitate health care access and treatment for veterans with mental health conditions. The aim of the audit was to provide objective information, advice, and assurance to assist parliament in its scrutiny of the government’s management of programmes and resources.

SUPREME AUDIT INSTITUTIONS AND GOOD GOVERNANCE: OVERSIGHT, INSIGHT AND FORESIGHT © OECD 2016