4 – SUPREME AUDIT INSTITUTIONS’ INPUT INTO POLICY EVALUATION AND OVERSIGHT – 139
Box 4.1. The SAI of Canada – assessing programme evaluation in the Federal Government (continued) Outcomes The six departments OAG examined followed systematic processes to plan their effectiveness evaluations, and completed most of the evaluations they had planned. However, each department’s evaluations during the audit period covered a relatively low proportion of its total programme expenses: between 5% and 13% annually across the six departments. The actual rate of coverage was even lower because many of the effectiveness evaluations OAG reviewed did not adequately assess programme effectiveness. Departments had often not gathered the performance information needed to evaluate whether programmes were effective. Of the 23 evaluation reports OAG reviewed, 17 had inadequate data, which limited the assessment of programme effectiveness. The departments OAG examined said that it remains a challenge to find experienced evaluators and that they had made extensive use of contractors to meet requirements. Departments expressed concern about their capacity to evaluate all direct programme spending from 2013, as required by the 2009 policy on evaluation. To ensure full coverage (which includes grants and contributions), departments will have to evaluate an average of 20% of their direct programme spending each year of the five-year cycle. The Treasury Board of Canada Secretariat has introduced initiatives to address the need for improvements in evaluation across government. However, it did not provide sustained support for effectiveness evaluation. In particular, it made little progress on developing tools to assist departments with the long-standing problem of a lack of sufficient data for evaluating programme effectiveness. With the exception one department that has processes in place to identify needed improvements, the audited departments do not regularly identify and address weaknesses in effectiveness evaluation. The audit report included recommendations related to several of the findings summarised above. The criteria for this audit were set out in the audit plan accepted by the audited departments.
Good practices used Since the beginning of evaluation audits in the 1970s, the evaluation function has had periods of comparative strength and weakness, some of which were documented by OAG audits. Overall, the function has strengthened, partly due to a more coherent policy framework and to capacity building within departments. Audit findings have been taken into account by the responsible central agency as it reviewed and refined its policies on effectiveness evaluation.
Lessons learned Periodic attention by the SAI to the evaluation function over a 30-year period has contributed to its strengthening.
Further reading Office of the Auditor General of Canada: Fall 2009 Report, Chapter 1, Evaluating the Effectiveness of Programs: www.oag-bvg.gc.ca/internet/English/parl_oag_200911_01_e_33202 .html. Office of the Auditor General of Canada: Spring 2013 Report, Status Report on Evaluating the Effectiveness of Programs: www.oag-bvg.gc.ca/internet/English/parl_oag_201304_01_e_38 186.html. Sources: OECD Survey of Peer Supreme Audit Institutions; further reading links above.
SUPREME AUDIT INSTITUTIONS AND GOOD GOVERNANCE: OVERSIGHT, INSIGHT AND FORESIGHT © OECD 2016
