4 – SUPREME AUDIT INSTITUTIONS’ INPUT INTO POLICY EVALUATION AND OVERSIGHT – 143
Box 4.3. The SAI of Canada – auditing for accountability and inclusivity (continued) The OAG also found that VAC’s method of calculating performance was not representative of the experience of the recipient. For instance, the OAG looked at the elapsed time from a veteran's first enquiry about mental health benefits, to the time a decision was made, and found that VAC “started the clock” not at the time of first enquiry, but from the time the department had received the benefit application and determined that the application was complete.
Good practices used The OAG went beyond using the entities’ performance criteria by comparing VAC’s measurement against the perception of the benefit recipient. In doing so, the OAG challenged how VAC calculated their performance. This audit received significant media attention when it was tabled on 25 November 2014 and continues to be frequently referenced in the press as of 30 April 2015. This audit put a focus on looking at the audited programme from the recipient's perspective, asking: “What would success look like?” and “How is the entity assessing success?” These messages resonate with citizens and thereby cover governance issues that are of wide-ranging concern. The approach the OAG took was evidence-based and replicable by the audited entity. Considerations for replicability by other SAIs are:
It is noteworthy and significant if the entity is found to not be tracking the information needed to assess the effectiveness of major programmes and activities, i.e. major sums are being spent without the entity assessing whether these expenditures provide meaningful results.
It is important to undertake the work necessary to determine the integrity of the information produced by the audited entity that underlies analyses, A thorough understanding of the policies and processes underlying the data analyses is critical.
Lessons learned Examining how the entity assesses performance against measures and targets is essential in forming an appropriate conclusion on results. The OAG’s audit highlighted a lack of client focus in establishing the targets, and a lack of transparency in terms of conveying results. When assessing an organisation's performance, comparisons against the measures and targets of comparable organisations in other jurisdictions can be difficult and often impossible. The legislative basis of entitlement, and the relative significance of the activity, can vary widely depending on the jurisdiction. Therefore, the perspective of the benefit recipient is a pragmatic and beneficial approach.
Further reading OAG Report: www.oag-bvg.gc.ca/internet/English/parl_oag_201411_03_e_39961.html. The VAC Departmental website: www.veterans.gc.ca/eng/. Sources: OECD Survey of Peer Supreme Audit Institutions; further reading links above.
SUPREME AUDIT INSTITUTIONS AND GOOD GOVERNANCE: OVERSIGHT, INSIGHT AND FORESIGHT © OECD 2016
