Measuring Efficiency at Field Audit Offices

MEASURING EFFICIENCY AT FIELD AUDIT OFFICES1 By Muhammad Akram Khan Former Deputy Auditor General of Pakistan Makram1000@gmail.com 1. OBJECTIVE The objective of this paper is to develop a methodology for measuring efficiency in Field Audit Offices (FAOs) in the Department of the Auditor General of Pakistan. Of late, the Department is under considerable pressure both from within itself and also from the various clients it serves to improve its efficiency. The legislators and executive heads of the governments are asking for the audits which are professionally competent and which also add value to the society at large. The gap between what the society expects the state auditors to perform and what the auditors think about their own role is widening with the passage of time. The auditors are complaining that they do not have adequate physical and financial resources to meet the challenge, while the clients of the department and the government think that the department is wasting its resources and is doing a ‘useless’ work with very little utility. In this scenario, where the need of more resources cannot be denied, the department also needs to do some soul searching. The fact is that the department has never tried to measure its own efficiency either at the level of individuals or at the organizational level. The department is operating like any other public sector organization with all the handicaps and built-in inadequacies. The department of the Auditor General is supposed to act as a watchdog of the other departments. With the addition of performance auditing into its role, it has also started commenting on the economy and efficiency of other departments. While it is a welcome addition to the role of the traditional role of government auditors, it has also made them more vulnerable.

Those who try to evaluate the efficiency of others must not display significant inefficiencies in their own operations. That is why, the Auditor General’s department should emerge as an office of highest efficiency and excellence. For this purpose a high degree of efficiency-consciousness is necessary among the officers of the department.

The present paper tries to develop a system of efficiency measurement in the Field Audit Offices. At present no such system exists. As a result, the efficiency of the offices is not measured. No one can say with any precision about the quality and efficacy of the work done by an office or its officers. In this scenario, some people are able to score a higher mileage not because of actual output but by their acumen to influence their audience.

Also, those who are really hard working but cannot do enough of “marketing” for their work, seem to have not done enough. As a result, it is difficult to define merit and excellence. Such a situation promotes a value system that is conducive to cronyism and corruption. The present paper is an attempt to devise a mechanism that defines merit in as objective terms as possible.

2. THE CHALLENGE OF DEVELOPING A COMPOSIT INDEX OF EFFICIENCY While trying to devise a suitable mechanism for an audit office, the real challenge is to develop a compost index of efficiency. In the literature on performance measurement, there is hardly anything on the efficiency of government audit offices. The usual symbols of efficiency in an audit office in Pakistan are number of audit observations, number of audit reports, recoveries affected at the instance of audit and the number of audit paragraphs printed in the audit report. Although these variables are valuable but they do not tell the entire story. For example, they do not tell anything about the cost of audit, time taken to develop the audit observations, the ratio of resources expended on financial audit and performance audit, resources spent on developing audit infrastructure for improving the effectiveness of office in the long run, etc. Thus there is a need for developing an index of efficiency that takes care of multiple aspects of efficiency. A further challenge is to reduce all the variables to one common denominator so that they can be interpreted. Another problem is to express a judgement on the importance of different variables. This requires assigning weights to different variables. By its very nature such a task is subjective. But we have still adopted this method, since having subject and measurable weights is better than not having any system of measurement. If we apply the subject weights consistently the bias is neutralized to some extent.

3. SCOPE The present paper covers the economy and efficiency aspects and does not include effectiveness aspects. The effectiveness of a FAO has several dimensions and its measurement requires much extensive data than can be collected economically in the present scenario. It will be a subject of some future exercise.

The present paper covers the efficiency of the FAOs and not individuals working in these offices. We shall address to the question of individual efficiency in a subsequent paper.

4. PARAMETERS OF EFFICIENCY The parameters of efficiency of a FAO are as follows: Output Parameters • • •

Quantitative Output Quality of Audit Developing Management Infrastructure

Input Parameters • •

Number of person days consumed Cost of audit operations

2

As we shall illustrate later, we shall adopt person years as the main determinant of efficiency in each case. Therefore, it will not emerge as an independent index efficiency.

We shall elaborate these parameters in the following discussion.

4.1 Quantitative Output The quantitative output of a FAO consists of the following: • •

Number of audit reports printed for the PAC Work-in-progress

4.1.1 Number of Audit Reports The number of audit reports prepared by a FAO is the foremost indicator of its performance. But since audit assignments differ in size, significance, and scope of work, it is not possible to adopt the number of audit assignments as a variable for comparing the efficiency of two FAOs. Moreover, it will generate a tendency to increase the number of audit reports on flimsy grounds. Therefore, we need to adopt a more uniform variable, keeping intact the issue of audit reports printed by a FAO. We think, the Audited/ Expenditure (AE) is a more uniform measure. It is not under the absolute control of the FAO also, although the FAOs have a limited flexibility to increase or decrease the coverage of AE. Thus we can adopt the AE of the audits completed and printed as audit reports as one variable.

4.1.2 Work-in-progress At the close of a year, every FAO will have some work done in the pipeline. We should take that into account. Or this purpose, following factors may be adopted to convert the work-in-progress (WIP) into complete audit reports: • • • • • • •

Audits only planned: 10% of the AE of the planned audit Audits in progress 20 % Audits completed but AIR not issued to the auditee 35% Audits completed and AIR issued to the auditee 50% Audit Report issued to the PAO 75% Audit Report approved by the AGP 85% Audit Report sent to press 95%

Once we are able to add up the AE we can work out a ratio of AE with person days and cost of audit.

3

For performance audits we can use these percentages to add up to the number of performance audit reports instead of the AE. 4.2 Quality of Audit 4.2.1 It is very difficult to measure quality of audit directly. We have to adopt a proxy measure. One such measure could be the extent of supervision of audits. We can adopt the person-days spent on audit supervision by the Deputy Director, Directors or Directors General. Information on this will be available from their tour/ local visits program, which should be documented properly. This can be compared with the planned supervision person-days and an index can be calculated.

4.2.2 Recoveries made at the instance of audit One important indicator of quality of audit is the amount of recoveries made at the instance of audit. This can be compared with the cost of audit. We can then say for each rupee spent on audit led to so many rupees of recoveries.

4.2.2 Savings made on recommendations of audit Another indicator of quality of audit is to the acceptance of audit recommendations by the executive leading to savings. This can then be compared with the cost of audit. We can then say that for each rupee spent on audit led to so much saving in terms of government expenditure.

4.3 Management Infrastructure It consists of the FAO's efforts in three areas: • • •

Development of master files Arrangement for weeding out old records System for keeping the office manuals updated

It is difficult to measure output on this count directly. Therefore, we need to adopt a proxy measure. We think all these areas are equally important. Therefore, we can calculate an efficiency by dividing the planned person-days by actual person-days spent on these activities.

5. ANALYSIS 5.1 Number of Person-days consumed on various activities This will be the most important input measure. It will mean adding up all the working days of the staff from B-5 to B-20 in a FAO. We should classify the entire time into various activities. For example, first we should calculate the time for regularity audit. The time planned for regularity audit will be the time net of other activities, such as

4

administration, audit of Foreign Aided Projects (FAP), Third Party Validation of SAP (TPV) and Performance Auditing.

5. 2 Cost of Audit It is a simple concept. The total budget of a FAO is the cost of audit. But for applying this concept to efficiency measurement, we need to refine it. The FAOs carry out several types of audits, like regularity audit, special audit, audit of foreign aided projects, Third party Validation, audit of SAP expenditure, special assignments requiring detailed investigations, etc. Therefore, we need to find out a mechanism for allocating cost to various activities. We do not have detailed cost accounting system. Therefore, it is not possible to have a precise and refined allocation system of cost. In the present scenario, trying to develop a cost accounting system will be self-defeating and the whole exercise of efficiency measurement will suffer. Therefore we can adopt a proxy system of allocating cost. We can adopt the concept of Average Cost of a Person Day (ACPD). This can be determined by dividing the total budget of an office by the total number of person days for audit activities. It means we shall first calculate the total number of staff from B5 to B-20, multiply it with the number of working days in a year and divide the total budget of the FAO by these working days. This will give us Average Cost per Person Day (ACPD). The ACPD can then be used to allocate cost to various types of audit activities according to the number of person days required by each activity.

6. HYPOTHETICAL EXAMPLE OF PERFORMANCE MEASUREMENT OF A FIELD AUDIT OFFICE

Total Budgeted of the Auditee Formations: Total number of formations Budget /Expenditure of the Formations included in the audit plan No. of auditee formations Person days available with the FAO (B5- B20) Budget of the FAO No. of regularity audit reports Average time for one audit assignment* AE of regularity audit reports WIP at the end of the year Performance Audit Reports Person days for regualrity audit Person days for performance audit Average time for one performance audit report*

5

Planned Data

Actual Data

75000 M 836 28000 M

78000 836 32000

380 32000

360 30500

24 M 9 24 weeks 22000 M 6000 M 3 16000 4000 20 weeks

25M 6 30 weeks 18000M 4000 1 18000 2400 28 weeks

Person days for SAP TPV 3200 3600 Sites for validation 280 225 Absenteeism cases 325 280 Procurement cases 115 105 Recruitment 70 65 Person days for FAP 2800 2000 No. of projects 45 30 Person days for administration & co6000 4500 ordination, etc Person days for Supervision [included in 3200 2800 Regularity Audit Time ] No. of audits supervised 125 90 Person days for Audit Management 800 300 Infrastructure [included in Administration & Co-ordination Tim] * It means the time taken for all stages of audit, from planning through printing of the report. This will be relevant to those reports, which have been printed. Work in progress will not be considered for this purpose.

7. ANAYLYSIS 7.1 Quantitative Outputs: (a) AE Ratio: Actual AE/ Planned AE = 22000 / 28000 = 0.79

[Ratio 1]

(b) Cost of Regularity Audit: Average Cost of Person day = Planned: Rs 24000,000 /32000 = Rs 750 Actual: Rs 25000,000 /30,500 = Rs 820 Total planned cost of Regularity Audit: 16000 x 750 = 12 M Total actual cost of Regularity Audit 18000 x 820 = 14.75 M Regularity Audit Cost Ratio: 12 /14.75 = 0.81 [Ratio 2] (c) Timeliness of Regularity Audit: Average planned time for completing one audit assignment: 24 weeks Average actual time for completing one audit assignment: 30 Average delay: 30 - 24 = 6 weeks Efficiency in finalizing the audit assignment [ 1- (6 / 24 ) ] = 0.75 [Ratio 3] (d) Overall Efficiency Index in Regularity Audit: 0.81 x 0.79 x 0.75 = 0.48 [Index 1] (e) AE person day ratio:

6

AE planned per person day: 28000 / 16000 = 1.75 M per person day AE actual per person day: 22000/ 18000 = 1.22 M per person day AE Person day Efficiency Ratio: 1.22 /1.75 = 0.70

[Ratio 4]

(f) Cost of Performance Audit Total planned cost of performance audits: 4000 x 750 =Rs 3000,000 Total actual cost of performance audit: 2400 x 820 = Rs 1968,000 Achievement in terms of reports: 1/ 3 = 0.33 Average planned cost of producing one performance audit report: 3000,000 /3 = 1000,000 Average actual cost of producing one performance audit report: 19680000 Performance audit cost ratio: 1000,000/ 1968000 =0.51 [Ratio 5] (g) Timeliness of performance audit: Average planned time for finalizing a P.A. = 20 weeks Average actual time for completing a P.A = 28 weeks Average delay = 8 weeks Efficiency in finalizing a P.A. = [ 1 - (8 /20 ) ] = 0.60 [Ratio 6 ] (f) Overall Efficiency Index in Performance audit: 0.33x 0.51 x 0.60 = 0.10 [Index 2] (g) Social Action Program Third Party Validation Work Site verification ratio: 225/280 = 0.80 Absenteeism ratio: 280/325 = 0.86 Procurement cases: 105/115 = 0.91 Recruitment cases: 65/70 = 0.93 Overall achievement in SAP TPV Work: (0.81+0.86+0.91+0.93) /4 = 0.88 [Ratio 7] (h) SAP Cost ratio: (3200 x750) / (3600 x820) =0.81

[Ratio 8]

(i) Overall Efficiency Index in SAP= 0.88 x 0.81 = 0.71

[Index 3]

(j) FAP Efficiency Formations audited: 30 Formations planned for audit3=45 Achievement rate: 30/45 = 0.66

[Ratio 9]

(k) Cost Efficiency in FAP Work: (2000 x750)/(2800 x820) =0.65 [Ratio 10] (l) Overall Efficiency Index in FAP = 0.66 x0.65 = 0.43

7

[Index 4]

(m) Contribution of Audit 1. Recoveries: Cost ratio This ratio measures the relationship between recoveries made at the instance of audit and the cost of audit. It will be worked out as follows: Recovery: Cost Ratio = 1 - (Total Cost of Regularity Audit/ Total Recoveries at the Instance of Audit), where the amount of recoveries is larger than the cost of audit. Where the recoveries are less that the cost of audit, this ratio will be ignored.

2. Savings: Cost ratio This ratio measures any savings caused by accepting the advice of audit. This will usually be the case in respect of performance audit reports. Therefore, the cost figure in this ratio will be the cost of performance audit. although it does not debar us from adopting the figure of cost of regularity audit, if the recommendation was made by the regularity audit and accepted by the auditees.

Savings: Cost ratio = 1 - (Total Cost of Performance Audit/ Total Savings on the Recommendation of Audit), where the amount of savings is larger than the cost of audit.

Where the savings are less that the cost of audit, this ratio will be ignored.

7. 2 Quality of Audit (a) Supervision -Person days Ratio: 2800/3200 = 0.88

[Ratio 11]

(b) Supervision Coverage ratio: 90/125 = 0.72

[Ratio 12]

(c) Overall Supervision Efficiency Index: 0.88 x 0.72 = 0.63

[Index 5]

6.3 Audit Management Infrastructure (a) Infrastructure Achievement ratio: 300/ 800 = 0.38

[Ratio 13]

8. OVERALL ASSESSMENT We have calculated 13 ratios. By using these ratios we have worked out 5 efficiency indices. But since all activities are not of equal importance, we think, we should assign some weights to various efficiency indices. In the following table column 3 has a set of suggested weights. They are purely for illustrative purposes.

Particulars

Efficiency Index

Weight

Regularity Audit

0.48

0.40

8

Weighted Index 0.19

Performance audit SAP TPV Work FAP work Supervision Management Infrastructure Total

0.10 0.71 0.43 0.63 0.38

0.20 0.15 0.10 0.10 0.05

0.02 0.11 0.04 0.06 0.02 0.44

Overall Efficiency of the FAO = 44% ____________________________________ The conclusion is that if we have arrangements for collecting necessary data on planned and actual achievements of a FAO we are able to determine its efficiency, howsoever crude it may be. It will still be better than mere conjectural assertions.

9. APPLYING THE METHODOLOGY IN THE FIELD AUDIT OFICES The next question is: how do we apply this methodology in the FAOs. It requires collection and validation of a large amount of data. As a practical measure, the FAOs should be informed in the beginning of the year about the parameters on which they would be evaluated. They should then be asked to maintain necessary record for collecting information on various variables. At the year-end, the FAOs should summarize the information for each variable and submit it to the Auditor General’s office. The Deputy Auditor General concerned should then visit each FAO and ask it to produce evidence in support of the information submitted by the office. The task of the Deputy Auditor General is that of an auditor. He has to verify most of the information presented by the office with reference to the original documents maintained by it. This is a detailed exercise and can take more than a day. At the end of the examination, the Deputy Auditor General should tick off all valid information and engage one of his assistant to work out various ratios. For the facility of the FAOs, we have provided specimens of the forms required for submitting summary to the Auditor General’s.

Once the Deputy Auditor General completes his round of all FAOs and prepares the analysis of various ratios, it would be possible to present the results in the form of a spreadsheet and place different offices in comparison to one another. This spreadsheet can, then, also become basis for any incentives, rewards or prizes to be given by the Auditor General for better performing offices.

It is important to add here that the forms annexed to this paper are only a summary information. The information in these forms cannot be provided until every FAO has a detailed information, which is kept updated on a daily basis. For this purpose, each FAO must set aside human and physical resources. Preferably, the FAOs should create a small cell responsible for monitoring of the plan. This cell should maintain detail information, either manually or on computer, which should be summarized for the office of Auditor General on the forms annexed with paper.

9

10. CONCLUDING REMARKS The performance measurement system proposed in this paper should be taken with a grain of salt. The performance of an organization or a person is determined by a host of factors. Some of these factors , such as will to work, proper organization of assignments, discipline and sense of urgency , are in the control of a person or an organization. But some of these factors are beyond one’s control. Examples are: lack of financial and human resources, poor incentive system, and inadequate salary structure, lack of appropriate training facilities, political interference and human temperaments, etc.

Although performance measurement is a current rage in the administration theory, yet not much attention has been given to the impact of these controllable and non-controllable factors. The moral of the story is that it is very easy to draw conclusion about the efficiency of an organization but rather difficult to understand and analyze reasons for the particular level of efficiency. It is imperative, therefore, that once the above analysis is made, the Deputy Auditor General concerned should hold a detailed session with the field audit office to analyze the level of efficiency achieved by that office. The ultimate decision regarding the performance of an office should be seen in the light of the review session.

10

Evaluation-1 EVALUATION OF FIELD AUDIT OFFICES GENERAL INFORMATION NAME OF THE AUDIT OFFICE -----------------------------------------------YEAR: October 19----September 19---Particulars Total No. of Assignments of Regularity Audit3 Total No. of Formations Total Expenditure/Income audited4 Total No. of Available persondays (B-5 to B-20) Total Budget/Expenditure of the FAO No. of Regularity Audit Reports printed by the FAO Person-days Spent on Regularity Audits (All Assignments)6 Person-days spent on Performance Audit7 No. of P.A. Assignments Person-days spent on SAP (TPV)8 Person-days spent on SAP (FA)9

Planned2

Actual

*5

11

Remarks

Person-days spent on FAP Sites for validation under SAP TPV Absenteeism Cases for Validation under SAP TPV Procurement cases for validation under SAP (TPV) Recruitment cases for validation under SAP TPV No. of Projects under FAP Person-days spent on Supervision10 No. of Audits supervised 11 Person-days for Administration & Co-ordination work Person-days for master files Person days for weeding out old record Person days for updating office manuals Recoveries made at the instance of audit12 Savings affected on the advice of audit13

12

Evaluation-2 EVALUATION OF FIELD AUDIT OFFICES DATA ON ASSIGNMENTS COMPLETED14 POSITION AS ON 30.9.19---Title/No. of Assignment Persondays

Planned Expenditur e/Income Audited

Total

13

Calend ar days15

Perso ndays

Actual Expen Calend diture/I ar days ncome Audite d

14

Evaluation-3 EVALUATION OF FIELD AUDIT OFFICES DATA ON AUDIT ASSIGNMENTS IN PROGRESS AS ON 30.9.19---Title of Assignment

Total Audited

Expenditure Stage Work16

Stages of Work; 1.

2.

3.

4.

5.

6.

7.

Audits only planned Field work in progress Audit completed but Audit Inspection Report not yet issued Audit completed and Audit Inspection Report issued to the auditee Audit Report issued to the principal accounting Officer Audit Report approved by the Auditor General Audit Report sent to press for printing

15

of

1*

original.file It means the data provided in the original approved plan. Please ignore all subsequent revisions for the purpose of evaluation of your office.

3 Also include any special assignments relating to regularity audit 4 It means the total of expenditure or income audited in case of government organizations. It does not mean the total of the sample. In case of organization which keep accounts on double entry basis and prepare a balance sheet and an income statement, it means total of all assets plus total of all expenses. Include any special assignments relating to regularity audit.

5 Please account for any transfers without replacements, leave by the staff and officers as actually availed or posting in of any personnel.

6 Fill in Form Evaluation-2. The time spent on Regularity Audit will include all time spent on various stages of Regularity Audit from planning to printing of the reports. It will also include time in the processing of the reports, DACs, Pre-PACs, PACs, and inter-departmental meetings, etc.

2

7

It will include time spent on all stages of P.A. Include any special assignments pertaining to performance audit.

It will include time spent on all stages of TPV. 9 It will include time spent on all stages of SAP (FA).

10 It refers to the supervision in the field by officers like Deputy Directors/ Directors/ Directors General. This is already included in various categories of person-days enumerated above. The person-days under regularity audit stated above will include the time for supervision. At this stage, we mention it as a part of the total mentioned above.

11 It refers to the number of formations where the supervisory officer went.

12 Recoveries made during the period under report, even though were pointed out in earlier audit reports. Please give only actual figures. This cannot be planned.

13 Please give actual figures only. There cannot be any planned figure. Also support it with evidence.

14 Assignment includes all types of assignments such as regularity audit, performance audit, SAP TPV, special assignment.

Please mention them in blocks of information under proper headings. Classify any special assignment into regularity audit, performance audit and others as the case may be.

15 It means the time from planning of audit assignment to printing of the audit report.

16 Please indicate the stage no. from the list given below.

8