Performance audit report process

Performance Audit Report Process1 By Muhammad Akram Khan2 JEL Code: M42

___________________________________________________________________ Abbreviations used AQMW

Audit Quality Management Wing

DG

Director General

FAO

Field Audit Office

PAC

Public Accounts Committee

SAI

Supreme Audit Institutions

ABSTRACT The performance audit reporting process passes through the same procedures as a compliance audit or internal audit would do. The auditor-in-charge of a performance audit assignment ensures orderly completion of audit, organization of work papers, discussions of audit findings and recommendations with the client and initiation of a draft audit report. The audit management ensures that the work done by the audit team meets the auditing standards and quality assurance requirements. The culmination of the process is transmittal of the final audit report to the client and the public accounts committee of the legislature.

INTRODUCTION The performance audit enters its reporting phase after audit execution. Generically, the whole process of performance auditing is like any other audit. However, there are differences in details. The execution phase involves at least the following steps: 1. (a) (b) (c) 2. (d) (e) (f) 1

Audit execution phase Developing an audit program Audit examination as per audit program Preparing audit work papers for the work done during audit examination Concluding the audit execution phase Developing draft audit observations Communicating audit findings to client as the work proceeds Drawing audit conclusions

Based on present writer’s Performance Audit Manual (2012) written for the Department of the Auditor General of Pakistan. Presented to the participants of International Intensive Training Program on Performance Audit organized

by Performance Audit Wing of the Auditor General of Pakistan on 14 march 2017 at Hotel Flatties, Lahore, Pakistan.

2 makram1000@gmail.com

(g) Formulating audit recommendations (h) Ensuring orderly completion of audit (i) Holding exit conference (j) Documenting minutes of exit conference 3. Preparing to enter reporting phase (k) Internal clearance of the audit observations (l) Revising draft audit observations (m) Drafting audit report and cross-referencing to audit work papers (n) Independent validation of facts 4. Reporting, Transmitting and following-up (o) Approval of the draft report by senior management (p) Transmitting draft report to the client (q) Revising the draft audit report considering client response (r) Drafting a final version (s) Issuing the final version to parliament and all concerned (t) Defending the audit report in the PAC

The present note does not deal with the execution phase. It has been mentioned only to put the discussion in a perspective.

CONCLUDING THE AUDIT EXECUTION PHASE Communicating with the client as the work proceeds Generally, arrangements are made with the client to ensure that scheduled, periodic meetings are held during the audit. These periodic meetings are meant to keep the client informed of audit progress, but do not preclude day-to-day, ad-hoc meetings necessary to collect information, discuss and resolve issues, and otherwise facilitate the conduct of the audit.

In addition to maintaining regular contact with the client during the audit, it is considered advisable to verify the findings, conclusions and recommendations with the client after completing work on each audit objective. As each audit objective is linked to a report heading, this also facilitates production of the draft paragraphs for the report.

The nature of the interim communication will depend on the sensitivity and significance of the issues involved, and on the status of client relations. This communication usually takes the form of a written Audit Note signed by the auditor in-charge and requires a response within 3-4 days, making it explicit that in case the response is not received, the auditors would consider that the management has accepted the audit note.

Another method of communication is through an e-mail message to the subject of the audit step, to clear facts, confirm the understanding as expressed in the conclusion and to serve as a platform to discuss the usefulness and effectiveness of any proposed recommendations.

These audit notes/e-mail messages are clearly marked ‘for discussion purposes only’. The client is briefed that these communications are for the purposes stated above, are part of audit work papers, and should not be confused with the audit report which will be issued after the completion of the audit.

Such findings that have been reported and satisfactorily resolved need not be mentioned in detail in the final report except to indicate that minor issues disclosed during the audit were discussed

with the client’s representatives and were satisfactorily resolved. Those findings that have not been satisfactorily resolved after discussion with the client should be communicated to the client in writing, and if still not resolved, should be considered for inclusion in the final report.

Ensuring orderly completion of audit An important step in execution phase is to ensure that the auditor complete the entire planned work since after the completion of the work, it is not easy and welcome to visit the client organization once again or even to ask for more information. The clients consider it over-bearing.

For this purpose, most of the SAIs prescribe an audit completion check-list. The auditor in-charge ticks off all items of the checklist and ensures that nothing is left undone. If there is something on which information was not available or for some other reasons some steps had not been completed, the auditor in-charge should document it and attach it to the audit completion checklist. The matter should also be brought to the notice of the supervisors. An example of audit checklist is at Annex-1 of this note.

Scheduling and conducting exit conference An exit conference is scheduled during the final stages of the fieldwork phase. After completing fieldwork about each objective of the Audit Plan, the auditor in-charge prepares a Summary of Reportable Issues for discussion with the client during the exit conference. A reportable issue is a significant condition that warrants reporting to someone able to act. Reportable, deficiencyoriented issues or findings generally require five basic elements: Criteria, conditions, causes, conclusions and recommendations. The summary is documented in the work paper file. The director of audit attends the exit conference, as needed, along with the auditor-in charge and team members.

Documenting the exit conference The auditor in-charge prepares a written record of the exit conference and documents it in the work papers. The record includes the time, place, names and titles of the participants, and the comments offered by the client for each of the issues or findings discussed.

The exit conference focuses on the results of the specific audit, including the findings, conclusions and recommendations. The exit conference is not to be the first time that the findings, conclusions, or recommendations are disclosed or discussed with the client. On-going clientaudit team communications are conducted throughout the course of the entire audit.

Based on the conclusions, the auditors prepare a fact-sheet and discuss it with the auditee management. The discussion takes place both at the operational level with those managers and staff who deal with those issues as well as with senior level managers (though not the chief executive at this stage.) Considering these discussions, the auditors adjust their conclusions and then proceed to clear the report first with their own management and then with the chief executive of audit entity. The discussion with the chief executive is also known as ‘exit conference’ and is an indication of the completion of the audit. The auditors discuss at this stage the possible recommendations with the auditee management and try to assess their initial reaction if these recommendations are made. The objective is to frame the recommendations in a manner that would be acceptable to management and they would be willing to implement them. The recommendations which are resisted by the auditee management have a little utility

DEVELOPING DRAFT AUDIT OBSERVATION

In developing reportable findings, it is essential that all the evidence, factual material collected in support of a probable finding, is sufficient, competent, and relevant.

Cross-referencing to audit program and work papers The audit observations are prepared in a manner that allows cross-references to: (a) audit program; (b) audit working papers.

The Auditor In-charge prescribes a numbering scheme for audit observations and all members of the team must follow it. The audit working papers and the audit program should have cross reference to the audit observation number(s).

The text of the audit observation should show clearly and in bold letters (in a distinct color) the reference to working paper pages on which the observation is based. It should be possible to trace the audit evidence from the audit observation. It should also be possible to trace the audit observation from working papers and audit program.

The auditors should use the Template for Audit Observation for documenting the audit observations prescribed by the department. (Annex-2)

Refer to Illustration [Attachment] STRUCTURE OF AUDIT OBSERVATION Support for each of the findings that are to be reported to the client must to the extent possible include all the elements of a finding: criteria, condition, cause, conclusion, client comments, and cure of the situation.

Audit Criteria The audit criteria provide the basis for audit observations. The criteria consist of expectations of the auditors from the auditee management and entity operations. While planning for the audit, the auditors would have discussed the audit criteria with the auditee management. They should now have the confidence that the auditee management would not question the audit criteria.

Conditions found While following the audit program, the auditors document conditions found on the ground. If the conditions on the ground meet the criteria, the auditors have a positive conclusion. They should mention it in the audit report by acknowledging achievements of the auditee management.

The audit team should identify any gaps between the audit criteria and the conditions on the ground. The gaps indicate qualified or negative conclusions which constitute potential material for the performance audit report. However, the audit team must validate their findings for accuracy of the facts. In reaching a decision on the observation, the auditors should also look at interrelated facts and other relevant evidence to have confidence on their conclusions.

Causes: Analysis of gaps Once the facts are confirmed to be valid, the auditors should commence with analysis of the gaps between the audit criteria and the conditions found. The analysis requires that the auditors probe into causes for the gaps. There could be reasons which are beyond control of the auditee management. In that event, the auditors must recognize this fact and mention it in their report.

Similarly, if the gaps exist because some of the internal controls are weak or missing, the auditors should discuss the matter with the auditee management to confirm their understanding. It is also possible that the gaps exist because of systemic factors such as absence of appropriate legislation, inadequate funding over long periods of time, existence of vacancies in key personnel positions,

international factors (such as policies of other countries leading to domestic difficulties in achieving the targets), etc. Such factors are often beyond control of the auditee management. In brief, the auditors should go deep into the whole question of causes of gaps between the criteria and the conditions. The analysis should lead the auditors to identify (a) controllable causes; and (b) uncontrollable causes. The former category leads the auditors to suggest actions that can help improve performance or to hold persons responsible for poor performance. The latter category will provide material for the SAI’s report to the parliament for long-term and strategic action plans.

Audit conclusions: Effect of non-performance Conclusions are logical inferences that should be specific and unequivocal and based on the evidence supporting the findings that have been developed. The strength of a conclusion depends on the persuasiveness of the evidence supporting the findings, and the logic used to formulate the conclusions. Conclusions should not merely be opinions. They should flow from the fully documented evidence collected and be free from personal biases or prejudices, and totally objective.

With the analysis of data, the auditors are ready to draw their conclusions. While finalizing their conclusions, they try to seek answers to such questions:  Is the deficiency isolated or systemic? 

What is the cause?



Can the organization fix the problem?



What are the impacts?

Answers to such questions determine the audit conclusions.

Answering the question: So what?

The auditors often come across situations where they point out gaps between criteria and conditions and show that the management could do better. However, the auditee management may disagree with the auditors. They might ask: if we have not met the audit criteria, so what? What has been the impact of that on our operations? The onus to answer this question is on the auditors and they should be well prepared for that. They must be able to show that the gap between criteria and conditions has significant impact on the performance of the organization.

The auditors should try to provide evidence from operations of the auditee and examples of cases noted during the field work and show how performance of the audited organization has been lower than expected and how there was room for improvement. While doing that, the auditors should attempt to quantify their conclusions as far as possible. In some cases, extracts from auditee files, written replies to audit queries and interview replies could be appropriate bases to show impact of not meeting the audit criteria. Examples of the impact could be higher cost in procurement, longer than expected time in completing certain tasks, poor quality of service, improper targeting of service delivery, weaker controls creating opportunities for corruption and fraud, dissatisfaction of citizens expressed through media and other means, bad governance and misuse of discretion, etc. The auditors should try to determine who is affected by the issue (for example, other units in the organization, central agencies, third parties, etc.).

The auditors should determine auditee management’s awareness of the issue. If the management is aware of the issue and the corrective action is under way, the issue may have less significance for reporting purposes. Certainly, it will change how the matter is reported.

The auditors should have confidence about their analysis based on evidence they have gathered and from discussions with the auditee staff during the field work. The Director during review of

the field work and the DG during mock-run for Quality Assurance Review should challenge the audit team about audit findings, audit analysis and impact of audit recommendations. The objective should be to get sure confidence about audit findings and recommendations.

Quantifying effect of findings The actual or potential effect of every finding should be determined and quantified, if possible.

The quantified effects should be expressed in economic terms where possible. Economic quantification permits the measurement and presentation of audit results in commonly understood terms. Savings accomplished, losses prevented, erroneous overpayments recovered, efficiencies achieved, effectiveness improved, and other such value-adding results can be calculated and compared with audit costs. Expressing such results in economic terms includes determining the financial impact of such outcomes as:  Cost savings, making scarce human and financial resources available for other operation/mission-related use; [For example, computerization should reduce the operational cost.]  Cost avoidances, reducing expenditures making funds available for other essential purposes; [For example, informing pensioners about their pensions through email rather by regular post.]  Increased income from the revenue-producing activities; [For example, cutting down on line losses of electricity distributed.]  Increased efficiency and effectiveness, maximizing the quantity and timeliness of outputs and outcomes; [ For example, increase in the income of farmers by installing tube wells to control water-logging and salinity.] and  Enhanced quality of services, optimizing the client’s contributions to the Organization’s mission. [For example, reducing response time in case of fire-fighting service.]

Fig.1 Developing Audit Observations

Conditions (What did we find?)

Criteria (What was expected?)

Gaps

Causes (Why it happened?)

Effects (So what?)

Recommendations (What should be done now?)

Cure: Formulating recommendations The ultimate objective of all audit engagements is to develop recommendations for improving efficiency and effectiveness and assure that recommendations will be implemented. The audit team needs to consider the following questions to develop good recommendations:     

What needs to be done?

Why does it need to be done?

Where does it need to be done?

Who is responsible for doing it?

What are the expected effects of the recommendations?



Could implementation of the proposed recommendation have negative effects elsewhere?

The litmus test of an appropriate recommendation is its acceptance by the auditee management.

If the auditee management finds the recommendation impracticable, cumbersome, expensive, pre-mature, unfair or unsuitable for any reason, it would never implement it. The primary objective of performance audit is to help management improve its performance for benefit of the people at large. If the auditee management finds a recommendation unsuitable or unacceptable, the FAO should review the recommendation at the highest level and if necessary at the SAI level. It is only when the SAI considers that the recommendation must be made despite

auditee resistance, that the FAO should include it in the draft performance audit report. In all other cases of auditee resistance, the FAO should either modify the recommendation or withdraw it.

An appropriate audit recommendation has following characteristics: (a) The recommendation should flow logically and directly from the audit findings. [For example: The result of a class has gone down because of non-availability of teachers.] (b) The recommendation should be practical. Preferably, it should not require additional resources. The cost of implementing the recommendation should not outweigh the benefits expected from it. [ For example: The security of the building suffered because the cameras installed were not functional.] (c) The recommendation should not deal with actions that fall outside the domain of the auditee management such as political or legislative action. However, in case the FAO considers that a change in law is also necessary, the matter must be discussed at the level of SAI before including it in the performance audit report. [For example: The income from telecom license could be enhanced by amending the applicable law.] (d) The recommendation should propose some action by the management which can be verified once completed. A good idea not supported by expected action will not make a good audit recommendation. [Increasing the number of fire-fighting trucks can reduce the response time to act against a fire.] (e) The recommendation should state what should be changed and avoid telling how this should be done. The exact method of bringing the change is domain of the auditee management and the auditors should avoid getting into that. [ The management should ensure open and competitive bidding for procurement but should not say how to go about this.] (f) The recommendation should identify the person, entity or the level which should implement it. The public-sector organizations generally have many persons and several organizational levels. Until it is clearly mentioned who should act, the recommendation remains dormant. Everyone keeps waiting for someone else to act.

When the audit report clearly mentions that such and such person or entity should act, the accountability requirements are also well-defined and it becomes possible to find out who has or has not acted on the recommendation. It further means the audit team should discuss the recommendation with the prospective auditee staff that would be required to implement it. This would inform the auditors any hidden resistance at the implementation level quite early and they can fine-tune the language and content of the recommendation appropriately. It would increase the rate of acceptability of audit recommendations. [ For example: the procurement section should take action.] (g) The recommendation should be formulated in a stand-alone style so that it is possible to understand it even in absence of detailed audit report. It means the wording of the recommendation should be such that it does not require reference to any specific part of the audit report. [ For example: The management committee of the school should

ensure that the number of students enrolled increase every year by 10 percent for next five years.] (h) The recommendation should, as far as possible, give a time-frame during which it should be implemented. Open ended recommendations (for future only) do not cut much ice. The auditee organizations find such recommendations easy to respond by saying: “Noted for compliance.” Quite often, such a response is without much substance. The auditors should discuss the question of the time-frame with the auditee management during the field work and in Exit Conference and embed it in the recommendation. It would be possible, later, to monitor implementation of the recommendation. [ The project manager should install additional equipment for water purification by 31 December …] (i) The recommendation should be persuasive and show the advantage clearly if it is implemented. [ For example: The installation of additional equipment would reduce the delays in responding to public queries by half time.]

PREPARING TO ENTER REPORTING PHASE Internal clearance of the audit observations Every member of the audit team drafts the audit observation emanating from his or her audit work. The auditor In-charge reviews the audit observations prepared by the team members and ensures that the format and content meet the audit standards.

After analyzing and synthesizing the audit evidence, the audit observations pass through the formal internal approval procedure of the SAI. Generally, the auditor In-charge will present draft of the audit observations to Director responsible for performance audit. The Director will review the audit observations and may like to discuss them in detail with the audit team to confirm his or her understanding of the audit conclusions.

The Director will review the relevant working papers based on which the audit observation has been prepared. He/she will record a certificate to the effect that the observation is supported by sufficient, relevant and valid audit evidence.

The Director General (DG) reviews each audit observation and satisfies himself/herself that:     

The audit observation is related to the audit objectives.

The observation meets the overall policy and approach of the FAO on the subject.

The auditee management agrees with the audit recommendation or the disagreement is recorded in the observation.

The recommendation has a reasonable probability of acceptance at the level of SAI and PAC. The risk exposure of the FAO is minimal by issuing the observation.

Revising draft audit observations The Auditor In-charge undertakes revision of the draft audit observations considering comments and guidance by the Director and Director General. The revised audit observations are approved by senior audit management and made basis for the draft audit report.

Preparing draft audit report The scene is now set for preparing draft audit report. At this stage the work is taken up, generally, by senior management of the audit office. The audit report organizes the audit observations per general practice of the audit office. Usually, more important observations with greater potential impact are brought in the beginning.

Independent validation of facts An important step in the finalization of the draft audit report is to make sure that all facts are correct and supported by valid evidence. For this purpose, a team of auditors not involved in the work of audit is assigned the duty of verifying the facts regarding work paper files. The team validating the facts records its certificate of having done this work. Wherever it finds that the facts are not supported by work papers, it refers the matter to the auditor in-charge. If the auditor-in-charge is unable to answer the reason for gap in evidence, the matter is reported to senior management and most likely the observation is modified or even dropped from the draft audit report.

REPORTING, TRANSMITTING AND FOLLOWING-UP Approval of the draft report by senior management After independent validation of the facts, the auditor in-charge prepares a self-contained copy of the draft report. The draft report follows the format prescribed by the SAI. Generally, it should have following sections: (a) (b) (c) (d) (e) (f)

Title page Executive summary Table of contents Text of the report containing recommendations with each observation Summary of recommendations at the end Annexes (g) Acknowledgements Along with the draft report, the auditor in-charge prepares all supporting evidence. The supporting evidence is documented in work paper files. All facts are cross-referenced to work paper files clearly and usually in red ink.

The report is critically examined by all levels of senior management and signed by them in token of their approval. In the process, they might refer various questions to auditor in-charge.

Transmitting draft report to the client After approval of the report by the senior management, the report is transmitted to the client with a time-frame (usually four weeks) to respond. On the expiry of the time-limit, if the client does not respond, they may be reminded once with some more time to respond beyond which the report will be finalized anyway.

That

Revising the draft audit report considering client response On receipt of management response, the auditor in-charge reviews the response and modifies the report where necessary. However, in some cases, the client disagrees with the audit findings, conclusions or recommendations. If the auditor in-charge, in consultation, with the senior management incorporates the client’s view and makes further comments to defend the auditors’ position. That leads the audit office to prepare the final version of the report.

Issuing the final version to parliament and all concerned The final version of the report is issued to the client and submitted to the parliament. The PAC fixes the time and date for discussion. That provides a forum to the auditors for defending the report in the presence of PAC and clients’ representatives. The auditors keep a note of the PAC directives which are also formally communicated to all concerned by the parliament secretariat.

Following-up the audit report The last leg of audit is follow-up of the audit report. The auditors like to track the implementation of the audit recommendation. Where the client reports that the recommendations have been implemented. The submits the evidence of the fact, the recommendations are closed in the audit data base. However, in some cases, the auditors might visit the client organization for verification of the client response before closing the recommendation.

Audit Completion Checklist Performance Audit Completion Checklist OFFICE OF THE DIRECTOR GENERAL OF AUDIT (Insert name) PERFORMANCE AUDIT OF [Insert title of assignment] YEAR ___________

Annex-1

Auditor In-charge ______________ Assignment No ____________ Notes: 1. The Director shall complete this checklist while submitting the Draft Performance Audit Report to DG for approval.

2. Column (4) will refer to explanatory notes which will be attached to the Checklist.

S#

Review Questions

(1)

(2)

1

Assignment audit plan approved by DG (date)

2

Audit notification letter issued (date)

3

Entry Conference held? (date)

4

Final audit program approved by DG (date)

5

Significant changes in planned objectives or scope?

6

Changes notified to auditee?

7

Fieldwork completed (date)

8

Audit observations issued to auditee?

9

Working papers completed by auditor in-charge?

10

Working papers review completed by the Director?

11

Audit work covers all planned objectives?

12

Exit Conference involved main auditees?

13 17

Draft Performance Audit Report Quality Assurance Review completed by AQMW? Major disagreement with auditee?

18

The audit report follows the prescribed template?

19

The report incorporates replies/comments of the auditee where necessary?

Overall Assessment

20

Date Yes/No/NA (3)

Note# (4)

Name and Signatures of Director Date: Name and Signatures of DG Date:

Annex-2 Template for Audit Observation OFFICE OF THE DIRECTOR GENERAL OF AUDIT (Insert name) PERFORMANCE AUDIT OF [Insert title of assignment] YEAR ___________ Auditor In-charge ______________ Assignment No ____________ Reference: Mention: [Audit Program procedure #] Subject: [Mention main heading of the Audit Observation as it would be inserted in the Draft Audit Report] ‌ Topical Sentence: [Summarize the main message of the observation as it would be inserted in the draft audit report] ‌ Criteria

Conditions [ Insert working papers reference in the text within brackets and in distinct color]

Causes

[ Insert working papers reference in the text within brackets and in distinct color]

Effect

[ Insert working papers reference in the text within brackets and in distinct color]

Recommendation(s)

Name and Signature of the Auditor charge Date

Name and Signature of the Auditor InDate

Certified that I have reviewed the audit observation and related working papers. I am satisfied that the audit evidence is sufficient, relevant and valid.

Name and Signature of the Director Date Name and Signature of DG Date

Attachment Illustration of Draft Audit Observation OFFICE OF THE DIRECTOR GENERAL OF AUDIT (Insert name) PERFORMANCE AUDIT OF [Insert title of assignment] YEAR 2015-16 Auditor In-charge: G.M. Khan Assignment: Audit of waste management in Lahore No #12 of 2015____________ Reference: Mention: [Audit Program procedure # 2.5] Subject: [Need for database on waste management and disposal Topical Sentence: The Lahore Waste Management Company (LWMC) needs to establish a database for waste collection and disposal.

Segment Description Reference to WP Criteria One of the principles of EMS is that an WP/2.5/p.1 organization should measure and monitor the achievement of its environmental objectives and targets, and the overall effectiveness of its operations. A database should be maintained to capture information pertaining to contaminated sites, the nature of contaminants, the types of waste being generated, and the methods of disposal including re-use, recovery and recycle.

Capturing reliable environmental data may require specialized skills, tools and techniques.

The database should also have information on various types of risks posed by a certain category of waste. The data must be analyzed to reduce risks to human health and the environment, and determine whether improvements and corrective actions are necessary. The database can also help in developing financial plans for environmental protection

Conditions

The LWMC only has a basic system to capture information on waste management. This

Causes

Effect

includes information on contaminated sites, liquid waste, waste oil collected and expired lithium batteries disposed of. However, this information is not comprehensive and does not capture data on all types of waste generated, collected, recycled, reused, recovered and finally disposed of During discussion with the management we learnt that LWMC was a newly established company and had not yet developed its wherewithal to develop a database which requires considerable investment in financial and human resource terms.

In the absence of a comprehensive database, the LWMC is not in position to plan for timely collection and disposal of waste from all sites.

This was obvious from many sites visited by the auditors where the waste had not been collected for several years. The auditors took some pictures of such sites which are enclosed.

Secondly, the database can help in minimizing the complaints as the information on sites requiring action would be readily available.

WP/2.5/p.2-6

WP/2.5/p.7-8

[ WP/2.5/pp.9-12 with pictures of contaminated sites

Recommendation(s) The audit office recommends that LWMC develop comprehensive database for waste collection and disposal and use this information for planning and complaint handling.

Name and Signature of the Auditor charge Date

Name and Signature of the Auditor InDate

Certified that I have reviewed the audit observation and related working papers. I am satisfied that the audit evidence is sufficient, relevant and valid.

Name and Signature of the Director Date Name and Signature of DG Date