Data Quality Assessment of ATASP-1 KPIs by Akinwale P. AKINTUNDE

AGRICULTURAL TRANSFORMATION AGENDA SUPPORT PROGRAM PHASE ONE

2021

DATA QUALITY ASSESSMENT: PERCEPTION ANALYSIS OF VALIDITY, RELIABILITY, TIMELINESS, PRECISION AND INTEGRITY OF THE DATA ON ATASP-1 KEY PERFORMANCE INDICATORS.

December, 2021

Citation: Arabi, I.M., Egba, R.S., Manta I.H., Auwalu A.S., Abubakar A., Adewumi A., Ojumoola A.O., Akintunde A.P., Ejiogu L.C., Falmata Z.G., Musa M.D., Onyekineso Jp.C., Mallam M., Bashir J.Y., Sani S.G. ISBN: 978-978-970-203-9

ACKNOWLEDGMENTS We express our profound gratitude to the amiable, hardworking and supportive National Program Coordinator of the Agricultural Transformation Agenda Support Program Phase-1 (NPC-ATASP-1), Dr. Muhammad Ibrahim Arabi who gave us the golden opportunity to carry out this important assignment. We equally acknowledge and extend our gratitude to the management staff of ATASP-1 at the headquarters for their tremendous cooperation and the unflinching support received. Our sincere appreciation also goes to all the Zonal Program Coordinators and their teams (especially the M&E and other Data Officers) in Adani-Omor, Bida-Badeggi, Kano-Jigawa and Kebbi-Sokoto Zones respectively for working assiduously with us and making great contributions throughout the period of the assignment to ensure its success. Thank you for providing all the needed support towards ensuring successful collection of accurate, valid and reliable data on the key performance indicators of the Program. We cannot forget the pleasant hospitality we enjoyed during the period of the field work in the various Zones. God bless you all. Finally, we appreciate everyone who made tremendous and numerous contributions towards the successful completion of this assignment. The Almighty God will reward you all abundantly. Many thanks to all those who have contributed in one way or the other in putting this report together. We accept responsibility for all views and errors in this report. Thank you. Dr. Adeoluwa Adewumi, Team Leader.

TABLE OF CONTENTS LIST OF FIGURES:................................................................................................................................................iv LIST OF TABLES:.....................................................................................................................................v LIST OF ABBREVIATION AND ACRONYMS: .....................................................................................vi GLOSSARY OF TERMS:....................................................................................................................vii EXECUTIVE SUMMARY: ................................................................................................................viii 1.0

INTRODUCTION TO DATA QUALITY ASSESSMENT:............................................................1

1.1

Objective:....................................................................................................................................1

1.2

Data Quality: ..............................................................................................................................2

2.0

DIMENSIONS OF DATA QUALITY ASSESSMENT (DQA): ...........................................3

2.1

Data Quality Dimensions:.........................................................................................................4

2.1.1

Accuracy of Data: ..................................................................................................................5

2.1.2

Reliability of Data:..................................................................................................................7

2.1.3

Data Consistency: ..................................................................................................................8

2.1.4

Data Completeness: ..............................................................................................................9

2.1.5

Data Relevance: ...................................................................................................................10

2.1.6

Accessibility of Data:............................................................................................................11

2.1.7

Timeliness of Data: ..............................................................................................................12

3.0

KEY COMPONENTS OF DQA:................................................................................................13

3.1

Data Management and Reporting System Assessment ..............................................13

3.2

Compliance to Data Encoding and Submission Standards:.......................................14

3.3

Data Verification: .................................................................................................................15

4.0

RESULTS ON PERCEPTION ANALYSIS OF INDICATORS:.........................................16

4.1

Validity of the Key Performance Indicators:................................................................17

4.2

Reliability of the Key Performance Indicators: ...........................................................19

4.3

Timeliness of the Key Performance Indicators:...........................................................21

4.4

Precision of the Key Performance Indicators: .............................................................23

4.5

Integrity of the Key Performance Indicators ...............................................................25

5.0

CONCLUSIONS AND RECOMMENDATIONS: .....................................................................28

5.1

Conclusions:..............................................................................................................................28

5.2

Recommendation: ...................................................................................................................28

REFERENCES:.......................................................................................................................................29

iii

LIST OF FIGURES: Figure I: Dimension of DQA...........................................................................3 Figure II: Perception Scores for Dimension of DQA........................................16 Figure III: Global Perception Average Scores for Dimension of DQA..............17 Figure IV: Perception on Validity of the Key Performance Indicators...............19 Figure V: Perception on Reliability of the Key Performance Indicators.............21 Figure VI: Perception on Timeliness of the Key Performance Indicators...........23 Figure VII: Perception on Precision of the Key Performance Indicators............25 Figure VIII: Perception on Integrity of the Key Performance Indicators...........27

LIST OF TABLES:

Table 1: Validity of the key Performance Indicators.........................................18 Table 2: Reliability of the Key Performance Indicators....................................20 Table 3: Timeliness of the Key Performance Indicators...................................22 Table 4: Precision of the Key Performance Indicators.....................................24 Table 5: Integrity of the Key Performance Indicators......................................26

LIST OF ABBREVIATION AND ACRONYMS:

AIDS

Acquired Immunodeficiency Syndrome

ATA

Agricultural Transformation Agenda

ATASP-1

Agricultural Transformation Agenda Support Programme Phase 1

Data Quality

DQA

Data Quality Assessment

HIV

Human Immunodeficiency Virus

Information Management

Innovation Platform

Information Technology

KPI

Key Performance Indicators

M&E

Monitoring & Evaluation

Microsoft

PIRS

Performance Indicator Reference Sheet

SCPZ

Staple Crop Processing Zones

GLOSSARY OF TERMS:

Data Quality Assurance - A process for defining the appropriate dimensions and criteria of data quality, and procedures to ensure that data quality criteria are met over time. It involves a process of data profiling to unearth inconsistencies, outliers, missing data interpolation and other anomalies in the data.

Data Quality Assessment – A review of program or project M&E/IM systems to ensure that quality of data captured by the M&E/IM system is acceptable.

Data Quality - refers to the condition of a set of values of qualitative or quantitative variables. It is a perception or an assessment of data's fitness to serve its purpose in a given context, be it in operations, decision making or planning.

vii

EXECUTIVE SUMMARY:

he Agricultural Transformation Agenda (ATA) Program was launched by the Nigerian government in 2012 with the goal of reducing food imports by increasing production of five major crops: rice, cassava, sorghum, cocoa, and cotton. The Program aims to restore agriculture to its former position at the heart of Nigeria's economy, thereby addressing rural poverty, youth unemployment, and overreliance on imported foods. It's also the vehicle via which Nigeria may replicate the agricultural-driven economic success stories of countries like Brazil, Thailand, China, Malaysia, Indonesia, Kenya, and Malawi. The Agricultural Transformation Agenda Support Program Phase-1 (ATASP-1) was formed in 2015 as a lynchpin Program for ATA and is now being executed in four staple crop processing zones (SCPZs) with funding from the African Development Bank. The Program's specific development objectives include improving food and nutrition security, creating jobs, and increasing the program beneficiaries' incomes and shared wealth on a long-term basis through rice, sorghum, and cassava value chains. Anambra and Enugu (Adani-Omor Zone), Niger (Bida-Badegi Zone), Kano and Jigawa (Kano-Jigawa Zone), and Kebbi and Sokoto (Kebbi-Sokoto Zone) are the states covered. The program is currently offering interventions in over 200 rural areas across 33 LGAs in the seven states that are participating. Data Quality Assessment (DQA) was conducted on the key performance indicators of the Program across the four staple crop processing zones. The objective of the DQA initiative is to provide a common approach for assessing and improving overall data quality. The tool helps to ensure that standards are harmonized and allows for joint implementation between partners and with ATASP-1 supported program. Specifically, the exercise was to determine validity, reliability, timeliness, precision and integrity of the data collected and analyzed for the key performance indicators during the project across the zones. The Program's key performance indicators include; Number of Direct Beneficiaries, Number of Youth Trained, Number of New Jobs Created, Food Production, Amount of Income Generated, Number of Producers Trained in Technical Skills, Number of Producers Trained in Business Skills, Number of producers and entrepreneurs obtaining loans (of which women) for economic purposes, Amount of loan obtained by producers and entrepreneurs (of which women) for economic purposes, Number of people sensitized on HIV/AIDS and other prevalent diseases, Number of campaigns on HIV/AIDS and other prevalent diseases, Area under Cultivation, Number of Farmers networked, Commodity Yield, Number of IPs formed/assisted/capacitated by crop value chain, Number of youths trained in technical skills (of which women) for economic purposes, Number of youths trained in business skills (of which women) for economic purposes.

viii

The results obtained from perception analysis of the data on ATASP-1 key performance indicators had high level of validity, reliability, timeliness, precision and integrity with percentage scores of 95.29%, 98.90%, 87.74%, 90.20% and 77.94% respectively. More so, the Global perception average score was 90.01% which implies that the data and results on the key performance indicators have an excellent level of accuracy and adequacy to be used by management for decision making in the program. Conclusively, it can be inferred that the DQA carried out on the key performance indicators of ATASP-1 was successful. The data and results on the indicators were valid, reliable, timely, precise and of integrity. It is recommended among other things that to further improve and sustain accurate and adequate data collection from the field on the key performance indicators of the Program in the future, the management should continually organize capacity building for the M&E officers and other staff involved in the program's data collection and management. Accordingly, the Program should carry out DQA bi-annually (every 6 months) to ensure the validity and integrity of the data collected and analyzed by Program Officers and Consultants. This will ensure that the management is not misled during key decision-making processes that affects the Program when designing future interventions.

1.0

INTRODUCTION TO DATA QUALITY ASSESSMENT: A Data Quality Assessment (DQA) is a periodic review that helps donors and the implementing partner determine and document “how good the data is” and also provides an opportunity for capacity-building of implementing partners. This is a strategy that is used by organizations to assess the strengths and weaknesses of the data in relation to data quality dimensions (e.g., accuracy, reliability, consistency, relevance, accessibility and timeliness). A DQA is usually performed to fix subjective issues related to professional processes, such as the generation of accurate reports, and to ensure that data-driven and data-dependent processes are working as expected. It is important for an organization to conduct DQA on a regular basis at all stages of project cycle. DQAs can be used for purposes of; 1) verifying the quality of reported data for key indicators at selected sites, 2) the ability of data management systems to collect, manage and report quality data. 3) putting up corrective measures with action plans for strengthening the data management and reporting system and improving data quality. This gives an organization opportunity to make necessary adjustments on how they are implementing the project. 4) capacity improvements and performance of the data management and reporting system to produce quality data. DQAs expose technical flaws in data and allow the organization to properly plan for data cleansing and enrichment strategies. This is done to maintain the integrity of systems, quality assurance standards and compliance concerns. Generally, technical quality issues such as inconsistent structure and standard issues, missing data or missing default data, and errors in the data fields are easy to spot and correct, but more complex issues should be approached with more defined processes. The objective of this DQA initiative is to provide a common approach for assessing and improving overall data quality. The tool helps to ensure that standards are harmonized and allows for joint implementation between partners and with ATASP-1 supported program. The tool allows for programs and projects auditing to assess the quality of data and strengthen data management and reporting systems. 1.1 Objective: To determine validity, reliability, timeliness, precision and integrity of the data collected and analyzed for the key performance indicators during the project.

1.2 Data Quality: Data quality assurance is the process of data profiling to discover inconsistencies and other anomalies in the data and it also assist in performing data cleansing activities (e.g., removing outliers, missing data interpolation) to improve the data quality. The data quality assurance suite of tools and methods include both data quality auditing tools designed for use by external audit teams and routine data quality assessment tool designed for capacity building and self-assessment. This allows implementers of programs to make necessary adjustments to the design of the program and how implementers can best execute the whole exercise. A good data quality assurance plan outlines strategies in the routine monitoring system to reduce:∗ * Estimation error and bias * Measurement error and bias * Transcription errors * Data processing error A data quality assurance plan also describes how and when internal data quality assessments will be implemented. On the other hand, a typical Data Quality Assessment approach might be identifying which data items need to be assessed for data quality and typically this will be data items deemed as critical to program operations and associated management reporting. DQA also assess which data quality dimensions to use and their associated weighting. In our proposed approach, each data quality dimension has its values or ranges representing good and bad quality data defined. It is important to note that a data set may support multiple requirements, therefore, a number of different data quality assessments may need to be performed. At the same time, some assessment criteria should be applied to the data items while reviewing the results and determining if data quality is acceptable or not. Where appropriate corrective actions should be taken like cleaning the data and improve data handling processes to prevent future recurrences. It is important to repeat these processes on a periodic basis to monitor trends in data quality. The outputs of different data quality checks may be required in order to determine how well the data supports a particular program need. Data quality checks will not provide an effective assessment of fitness for purpose if a particular program need is not adequately reflected in data quality rules. Similarly, when undertaking repeat data quality assessments, organizations should check to determine whether business data requirements have changed since the last assessment (DAMA-UK, October 2013).

2.0 DIMENSIONS OF DATA QUALITY ASSESSMENT (DQA): A Data Quality (DQ) Dimension is a recognized term used by data management professionals to describe a feature of data that can be measured or assessed against defined standards in order to determine the quality of data. (DAMA-UK, October 2013).

DIMENSIONS OF DQA

Figure 1: Dimensions of DQA

It is important to note that these dimensions are not always 100% met, for example, data can be accurate but incomplete, or it can meet all criteria except for timeliness. As managers have to make decisions based on data, it is very important to perform a short audit of data before compiling key performance indicator (KPI) results in a performance report, based on the quality dimensions presented above. Therefore, if data is not complete or there is a uniqueness issue, data users must be informed in order to keep this in mind when deciding.

2.1 Data Quality Dimensions:

Data Quality S/N Dimension Definition Main dimensions of data quality 1.

Accuracy/ Validity

A measure of the correctness of data. Accurate data should represent what was intended or defined by the original source of the data. Data measure what they are supposed to measure.

-Also known as validity. Accurate data are considered correct: the data measure what they are intended to measure. Accurate data minimize error (e.g., recording or interviewer bias, transcription error, sampling error) to a point of being negligible. 2 Reliability The d ata generated by a program’s information system are based on protocols and procedures that do not change according to who is using them and when or how often they are used. The data are reliable because they are defined, measured and collected in the same. Sub dimensions of data quality 3.

Consistency

Data are consistent when the value of any given data element is the same across applications and systems.

Completeness

The extent to which the expected attributes of data are provided; all required data elements are captured in the database system. Completeness may mean that an information system from which the results are derived is appropriately inclusive: it represents the complete list of eligible units and not just a fraction of the list.

Relevance

Accessibility

Timeliness

The extent to which data are applicable and useful for the task at hand. Accessibility is the extent to which data are available or easily retrievable. The degree to which data are current and available for use as specified and in the time frame in which they are expected. Data are timely when they are up -to-date (current), and when the information is available on time. Timeliness is affected by: (1) the rate at which the program’s information system is updated; (2) the rate of change of actual program activities; and (3) when the information is actually used or required.

Other dimensions of data quality 8

Integrity

Data has integrity when the system used to generate them are protected from deliberate bias or manipulation for political or personal reasons. Confidentiality Confidentiality means that clients are assured that their data will be maintained

Uniqueness

according to national and/or international standards for data. This means that personal data are not disclosed inappropriately, and that data in hard copy and electronic form are treated with appropriate levels of security (e.g., kept in locked cabinets and in password protected files. points out that there should be no data duplicates reported. Each data record should be unique, otherwise the risk of accessing outdated information increases. Looks at issues to do with a single view of the data sets and not having cases that are duplicates

2.1.1 Accuracy of Data: Title Accuracy /Validity Definition

Reference

Measure

The degree to which data correctly describes the "real world" object or event being described. The degree to which the data item correctly describes the object in context of appropriate real-world context and attributes. The real-world context may be identified as a single version of established truth and used as a reference to identify the deviation of data items from this reference. Specifications of the real-world references may be based on business requirements and all data items that accurately reflect the characteristics of real -world objects within allowed specifications may be Ideally the "real world" truth is established through primary research. However, as this is often not practical, it is common to use 3rd party reference data from sources which are deemed trustworthy and of the same chronology. The degree to which the data mirrors the characteristics of the real-world object or objects it represents.

Scope

Any "real world" object or objects that may be characterized or described by data, held as data item, record, data set or database. Unit of Measure The percentage of data entries that pass the data accuracy rules.

Type of Measure:

Assessment, e.g., primary research or reference against trusted data.

Continuous Measurement, e.g., age of students derived from • Assessment the relationship between the students’ dates of birth and the current date. Discrete Measurement, e.g., date of birth recorded. • Continuous • Discrete Related Dimension Optionality

Validity is a related dimension because, in order to be accurate, values must be valid, the right value and in the correct representation. Mandatory because - when inaccurate - data may not be fit for use.

Applicability Example(s)

A European school is receiving applications for its annual September intake and requires students to be aged 5 before the 31st August of the intake year. In this scenario, the parent, a US Citizen, applying to aEuropean school completes the Date of Birth (D.O.B) on the application form in the US date format, MM/DD/YYYY rather than the European DD/MM/YYYY format, causing the representation of days and months to be reversed. As a result, 09/08/YYYY really meant 08/09/YYYY causing the student to be accepted as the age of 5 on the 31st August inYYYY. The representation of the student’s D.O.B. –whilst valid in its US context–means that in Europe the age was not derived correctly and the value recorded was consequently not accurate.

Pseudo code

((Count of accurate objects)/ (Count of accurate objects + Counts of inaccurate objects)) x 100

2.1.2 Reliability of Data: Title Reliability Definition

Reference Measure Scope

The degree to which the result of a measurement, calculation, or specification can be stable consistent or repeatable over time or by different people. Established through primary research in relation to data quality. It is a measure of the stability or consistency of test scores. Characterized or described by data, held as data item, record, data set or database.

Unit of Measure Typ e of Measure

Overall consistency of a measure

Related Dimension

Consistency and validity are related dimension because, in order for data to be reliable it has to be consistent and accurate. Mandatory because - when data is not reliable - data may not be fit for use. Frontline workers in health use MUAC tapes to measure malnutrition risks in under-five children and pregnant women.

Optionality Example(s)

This means that different frontline workers are expected to come up with same results after measuring same child. Pseudo code

N/A

2.1.3 Data Consistency: Title Consistency Definition

The absence of difference, when comparing two or more representations of a thing against a definition. This measure represents the absence of differences between the data items representing the same objects based on specific information requirements

Reference

Data item measured against itself or its counterpart in another data set or database.

Measure

Analysis of pattern and/or value frequency.

Scope

Unit of Measure

Assessment of things across multiple data sets and/or assessment of values or formats across data items, records, data sets and databases. Processes including: people based, automated, electronic or paper. Percentage.

Type of Measure:

Assessment and Discrete.

• Assessment • Continuous • Discrete Related Dimension(s) Optionality

Validity, Accuracy and Uniqueness

Example(s)

School admin: a student’s date of birth has the same value and format in the school register as that stored within the student database. Select count distinct on ‘Date of Birth’

Pseudo code

It is possible to have consistency without validity or accuracy.

2.1.4 Data Completeness: Title Completeness Definition Reference Measure

The proportion of stored data against the potential of "100% complete" Business rules which define what "100% complete" represents. A measure of the absence of blank (null or empty string) values or the presence of non-blank values.

Scope

0-100% of critical data to be measured in any data item, record, data set or database

Unit of Measure

Percentage

Type of Measure:

Assessment only

•

Assessment

•

Continuous

•

Discrete

Related dimension

Validity and Accuracy

Optionality

If a data item is mandatory, 100% completeness will be achieved, however validity and accuracy checks would need to be performed to determine if the data item has been completed correctly

2.1.5 Data Relevance: Title Relevance Definition Reference Measure

Scope

Unit of Measure

Type of Measure:

An assessment of data's fitness to serve its purpose in a given context. Established through primary research in relation to data and data quality the degree to which something is related or useful to what is happening or being talked about it is the closeness between data consumer need and data provider output. An aspect of (-->) data quality: a level of consistency between the (-->) data content and the area of interest of the user It is measured as percentage of all data required divided by all data provided. [100% is best] Assessment, e.g., primary research or reference related to what is happening.

• Assessment Related Dimension Optionality Applicability Example(s)

The ability to use data with maximum efficiency. Not having to sort through information you don’t need. Having beneficiary’s database covering more communities and using queries one can be able to retrieve the exact data they are looking for

Pseudo code

2.1.6 Accessibility of Data: Title Accessibility Definition

It is a generic term referring to a process which has both an ITspecific meaning and other connotations involving access rights in a broader legal and/or political sense. In the former it typically refers to software and activities related to storing, retrieving, or acting on data housed in a database or other repository.

Reference Measure Scope

The ease at which data stored can be easily retrieved or manipulated. Extent to which a consumer or user can obtain a good or service at the time it is needed.

Unit of Measure

Yes – Completely Accessible, No, Partially Accessible

Type of Meas ure:

Nominal

Related Dimension

Transparency

Optionality Applicability Example(s)

Users who have data access can store, retrieve, move or manipulate stored data, which can be stored on a wide range of hard drives and external devices. accessibility At community level volunteers submit reports to frontline workers who later report to district officers. In this scenario if anyone in this chain holds data will affect the whole reporting process

Pseudo code

2.1.7 Timeliness of Data: Title Timeliness 1 Definition

The degree to which data represent reality from the required point in time.

Reference

The time the real-world event being recorded occurred.

Measure

Time difference

Scope

Any data item, record, data set or database.

Unit of Measure

Time

Type o f Measure:

Assessment and Continuous

•

Assessment

•

Continuous

• Discrete Related dimension

Accuracy because it inevitably decays with time.

Optionality

Optional dependent upon the needs of the business.

Example(s)

Tina Jones provides details of an updated e mergency contact number on 1st June 2013 which is then entered into the student database by the admin team on 4th June 2013. This indicates a delay of 3 days. This delay breaches the timeliness constraint as the service level agreement for changes is 2 days. Date emergency contact number entered in the student database (4th June 2013) minus the date provided (1 st June 2013) = a 3 Day delay.

Pseudo code

Each data set will have a different proportion of volatile and non-volatile data as time acts differently on static and dynamic records

3.0 KEY COMPONENTS OF DQA: The DQA tool is composed of three components: (i) systems assessments (ii) compliance to data encoding and submission standards; (iii) data verification; 3.1

Data Management and Reporting System Assessment This enables qualitative assessment of the relative strengths and weaknesses of functional areas of a data management and reporting system at all levels. The purpose of assessing the data management and reporting system is to identify potential threats to data quality posed by the design and implementation of data management and reporting systems. The systems assessment questions are asked to the persons responsible for managing data and preparing reports at the different levels.

The systems assessment section of the DQA tool includes the following five functional areas: 1. M&E Structure, Functions and Capabilities: Availability of M&E organizational structure, training plan, and trained data management staff. 2. Indicator Definitions and Reporting Guidelines: Availability of indicator definitions and guidelines on reporting i.e., when, where and to whom reports should be sent. 3. Data Collection and Reporting Forms and Tools: Availability, appropriateness and utilization of standard data collection and reporting tools. 4. Data Management Processes: Availability of data quality controls, data back-up procedures, confidentiality of personal data, and feedback on quality of reported data. 5. Links with National Reporting System: Use of / adherence to national reporting system i.e., data tools, reporting channel, reporting deadlines, and sites identification. Using the Excel DQA tool, scores are generated for each functional area. The scores are an average for all responses to the qualitative questions in each functional area, with each question coded 3 for “yes, completely,” 2 for “partly,” and 1 for “no, not at all.” The scores are intended to be compared across functional areas to guide program implementers on which systems strengthening activities to prioritize. It would be reasonable to consider investing more resources in an area whose score is low compared to that whose score is relatively high. In order to complete both the Data Verification and Perception Assessment parts of the DQA tool, the assessment team will have to make some observations, do a

recount, and ask questions to the appropriate respondents. The emphasis of DQA is to verify the quality of reported data and identify potential challenges to data quality created by the data management and reporting system. It is intended to improve the quality of reported data and systems but not to change already reported data (NTD-Guidelines, December 2013). The Program Information Management System of a program/project provides the overall procedures in collecting, processing, and managing data. As mentioned earlier, quality data is highly dependent on the systems in place. Strong system should produce better quality of data. The systems assessment will look into the functionality and effectiveness of the following: v Competencies of M&E officers and other staff involved in data collection and management v Capacity building and technical assistance v Data collection, processing and management v Use of paper-based forms and computer-based templates v Encoding and submission v Internal quality control v Data utilization and reporting v Storage and retrieval Moreover, the objective of this assessment is to help the management understand the underlying limitations and problems encountered during data collection, processing and management, determine possible area or source of data errors, identify measures to improve the capabilities of staff involved in the process and strengthen data management at all levels. 3.2 Compliance to Data Encoding and Submission Standards: Compliance programs are seen as an effective mechanism to assure compliance with regulations and minimize risk of fraud. A coding compliance program should be a key component of any program. That is by complementing, not conflicting with, the compliance program. This provides timely and complete data to management, partners and other stakeholders thereby facilitating better and informed decision-making. This component deals with the completeness and timeliness of submission of data from the partner/district up to the national level based on the reporting requirements and standards set and provided to all levels. The partner, district, regional and national teams will be evaluated on the level of their compliance to these standards.

3.3 Data Verification: Data Verification is a process in which different types of data are checked for accuracy and inconsistencies after data migration is done. It involves recalculating results from source documents and uses a simple calculation to determine the accuracy of reported data. The level of accuracy is measured by percentage variance, which is defined as the variance between the recalculated value compared to what was reported. It helps to determine whether data was accurately translated when data is transferred from one source to another, is complete, and supports processes in the new system. During verification, there may be a need for a parallel run of both systems to identify areas of disparity and forestall erroneous data loss. An example of Data Verification is double entry and proofreading data. Proofreading data involves someone checking the data entered against the original document. The formula used to calculate variance is as follows:

% Variance

Reported value – Verified value = Verified value

x 100

Where a positive variance: (+) reflects over-reporting and negative (-) variance reflects under-reporting. A threshold of +/- 5% variance has been suggested for high quality data (fhi360dvt-oct2013) During project implementation different forms, templates and documents are used, completed and collected by stakeholders and other program staff to capture and document project activities implemented in various communities. Given the volume of data and information being collected, the management still gives high regard and importance to quality and providing accurate and consistent data to its stakeholders. Data verification will look into the accuracy and consistency of data from the source document, crosschecking the reported information with the paper-based forms, templates and other post documentations. It will identify, track and resolve inconsistencies and errors in the database. This exercise can be time consuming and costly.

4.0 RESULTS ON PERCEPTION ANALYSIS OF INDICATORS: To determine validity, reliability, timeliness, precision and integrity of the data collected and analyzed for the key performance indicators during the project, the perceptions of selected respondents were examined across the zones, that is, Adani-Omor, BidaBadeggi, Kano-Jigawa and Kebbi-Sokoto. The results obtained from the key performance indicators during the project were valid, reliable, timely, precise and of integrity with a percentage score of 95.29%, 98.90%, 87.74%, 90.20% and 77.94% respectively as presented in Figure II. More so, the Global perception average score is 90.01% as presented in Figure III. The results obtained for the five (5) Dimension of Data Quality Assessment are presented in the subsections 4.1 to 4.5. 100.00

95.29

98.90 87.74

90.00

90.20 77.94

80.00 70.00 60.00 50.00 40.00 30.00 20.00 10.00 VALIDITY

RELIABILITY

TIMELINESS

PRECISION

Source: computed from field survey data, 2021 Figure II: Perception Scores for Dimension of DQA

INTEGRITY

100.00

Average Score, 90.01

90.00 80.00 70.00 60.00 50.00 40.00 30.00 20.00 10.00 -

Source: computed from field survey data, 2021 Figure III: Global Perception Scores for Dimension of DQA

4.1 Validity of the Key Performance Indicators: The result obtained from the respondents on their perception about the validity of key performance indicators and how they were captured is presented in Table 1. It showed that the respondents have a very high perception on all the indicators that they clearly and adequately represent the intended result as none of the indicators scored below 80% and the average validity of the indicators is 95.29%. This suggests that all the instruments used, procedures adopted and people engaged all positively contributed to the validity of the whole exercise throughout the project. Specifically, the result indicated that outsiders or experts in the field were in agreement that the indicator is a valid and logical measure for the stated result and that the results reflect an accurate measure of the contribution of the project to the target farmers. Moreso, that respondents affirmed that there is reasonable assurance that the data collection methods being used do not produce systematically biased data. Likewise, the data collectors/enumerators were qualified and properly supervised and that known data collection problems were appropriately addressed.

Table 1:

Validity of the Key Performance Indicators Sum of Positive

Indicators

Responses

Percentage

Number of Direct Beneficiaries

Number of Youth Trained

Number of New Jobs Created

Food Production

Amount of Income Generated

Number of Producers Trained in Technical Skills

100

Number of Producers Trained in Business Skills

100

Area under Cultivation

Number of Farmers networked

100

Commodity Yield

100

Number of producers and entrepreneurs obtaining loans (of which women) for economic purposes Amount of loan obtained by producers and entrepreneurs (of which women) for economic purposes Number of people sensitized on HIV/AIDS and other prevalent diseases Number of campaigns on HIV/AIDS and other prevalent diseases

Number of IPs formed/assisted/capacitated by crop value chain Number of youths trained in technical skills (of which women) for economic purposes Number of youths trained in business skills (of which women) for economic purposes

Source: computed from field survey data, 2021

Number of youths trained in business skills (of which… Number of youths trained in technical skills (of which… Number of IPs formed/assisted/capacitated by crop… Commodity Yield Number of Farmers networked Area under Cultivation

Indicator

Number of campaigns on HIV/!IDS and other… Number of people sensitized on HIV/!IDS and other… !mount of loan obtained by producers and… Number of producers and entrepreneurs obtaining… Number of Producers Trained in Business Skills Number of Producers Trained in Technical Skills Amount of Income Generated Food Production Number of New Jobs Created Number of Youth Trained Number of Direct Beneficiaries 0 Percentage

100

120

Sum of Reponses

Source: computed from field survey data, 2021 Figure IV: Perception on Validity of the Key Performance Indicators 4.2 Reliability of the Key Performance Indicators: For the reliability of the key performance indicators, the result obtained from the perception of the respondents is presented in Table 2. It showed that all the indicators were rated above 90% in terms of reliability with an average of 98.90% for the seventeen indicators. This implies that the respondents judged that the data reflected stable and consistent data collection processes and analysis methods throughout the project. Summarily, the results suggested that consistent data collection process was used across time, locations, and data source. There were appropriate written procedures in place for periodic review of data collection, maintenance, and processing which were consistently practiced. Similarly, the results also suggested that data collection and analysis methods adopted ensured consistency of procedures throughout the project.

Table 2:

Reliability of the Key Performance Indicators Sum of Positive

Indicators

Responses

Percentage

Number of Direct Beneficiaries

100

Number of Youth Trained

100

Number of New Jobs Created

100

Food Production

93.75

Amount of Income Generated

100

Number of Producers Trained in Technical Skills

100

Number of Producers Trained in Business Skills

100

Area under Cultivation

93.75

Number of Farmers networked

100

Commodity Yield

93.75

100

Source: computed from field survey data, 2021

Indicator

Number of campaigns on HIV/!IDS and other prevalent… Number of people sensitized on HIV/!IDS and other… !mount of loan obtained by producers and… Number of producers and entrepreneurs obtaining loans… Number of Producers Trained in Business Skills Number of Producers Trained in Technical Skills Amount of Income Generated Food Production Number of New Jobs Created Number of Youth Trained Number of Direct Beneficiaries 0 Percentage

100

120

Sum of Reponses

Source: computed from field survey data, 2021 Figure V: Perception on Reliability of the Key Performance Indicators

4.3 Timeliness of the Key Performance Indicators: Timeliness of information in terms of availability and accessibility by management for decision making process is germane to successful project implementation and achievement of set goals. Results on timeliness of data availability on key performance indicators to influence management decision making process is presented in Table 3. The results revealed that the respondents have a very high positive perception on timeliness of data on all the indicators with the minimum score being 83.33% with an average timeliness of 87.74%. This implies that data were available frequently enough to inform program management decisions. The responses also affirmed that data were properly stored and readily available and accessible when needed to be used to make management decisions.

Table 3:

Timeliness of the Key Performance Indicators Sum of Positive

Indicators

Responses

Percentage

Number of Direct Beneficiaries

91.67

Number of Youth Trained

91.67

Number of New Jobs Created

91.67

Food Production

83.33

Amount of Income Generated

83.33

Number of Producers Trained in Technical Skills

100

Number of Producers Trained in Business Skills

91.67

83.33

91.67

Area under Cultivation

83.33

Number of Farmers networked

91.67

Commodity Yield

83.33

91.67

83.33

Source: computed from field survey data, 2021

Indicator

100

120

Sum of Reponses

Source: computed from field survey data, 2021 Figure VI: Perception on Timeliness of the Key Performance Indicators 4.4 Precision of the Key Performance Indicators: The results presented in Table 4 shows the perception of the respondents on the precision of data on key performance indicators. It revealed that the respondents alluded that the data collected and analyzed during the project were precise enough to inform management decision making process. All the indicators were scored above 80% with an average score of 90.20%. This implies that the data have a sufficient level of detail to permit management decision making throughout the project. Specifically, the respondents opined that the margin of error were less than the expected change being measured, and where results were obtained through statistical samples, the margins of error were reported along with the data. Moreso, the data collection method/tool being used to collect the data were fine-tuned or exact enough to register the expected change.

Table 4:

Precision of the Key Performance Indicators Sum of Positive

Indicators

Responses

Percentage

Number of Direct Beneficiaries

83.33

Number of Youth Trained

100

Number of New Jobs Created

83.33

Food Production

83.33

Amount of Income Generated

91.67

Number of Producers Trained in Technical Skills

91.67

Number of Producers Trained in Business Skills

91.67

Area under Cultivation

91.67

Number of Farmers networked

91.67

Commodity Yield

91.67

83.33

Source: computed from field survey data, 2021

Indicator

100

120

Sum of Reponses

Source: computed from field survey data, 2021 Figure VII: Perception on Precision of the Key Performance Indicators

4.5 Integrity of the Key Performance Indicators Results on the integrity of the data collected and analyzed on key performance is presented in Table 5. It revealed based on the perception of the respondents that relatively, data on the key performance indicators have very good integrity with the minimum score being 75% and 77.94% on average. This implies that the data collected have safeguards to minimize the risk of transcription error or data manipulation. It suggested that the procedures or safeguards put in place to minimize data transcription errors were adequate as data were subjected to cleaning and validation. It also implies that there was independence in key data collection, management and assessment procedures as data quality assessments were conducted. The respondents' perception also suggests that there were adequate mechanisms put in place to prevent unauthorized changes to the data as data were protected from unauthorized users.

Table 5: Indicators

Respondents’ Integrity of the Key Performance Indicators Sum of Positive Percentag Responses

Number of Direct Beneficiaries

Number of Youth Trained

91.67

Number of New Jobs Created

Food Production

Amount of Income Generated

Number of Producers Trained in Technical Skills

83.33

Number of Producers Trained in Business Skills

83.33

Area under Cultivation

Number of Farmers networked

Commodity Yield

83.33

Indicator

Sum of Reponses

Source: computed from field survey data, 2021 Figure VIII: Perception on integrity of the Key Performance Indicators

90 100

5.0

CONCLUSIONS AND RECOMMENDATIONS:

5.1 Conclusions: Judging from the results obtained, it can be concluded that the data collected and analyzed for the key performance indicators during the project were valid, reliable, timely, precise and of integrity with a percentage score of 95.29%, 98.90%, 87.74%, 90.20% and 77.94% respectively. More so, the Global perception average score is 90.01%. This is to say that the data and results on the key performance indicators were accurate and adequate to be used by management for decision making 5.2 Recommendation: Based on the findings of this exercise, the following recommendations were made. 1. Data Quality Assessment should be carried out bi-annually (every 6 months) to ensure the validity and integrity of the data collected and analyzed by program officers and consultants. This will ensure that the management is not misled during key decision-making processes that affects the program. 2. The program's database should be updated regularly as soon as new data are available and collected. Regular updating of the database will enhance the timeliness of data on key performance indicators. 3. All available data and documents on the program's key performance indicators should be reviewed when carrying our subsequent Data Quality Assessment. 4. To further improve and sustain accurate and adequate data collection from the field on the key performance indicators of the program, the management should continually organize capacity building for the M&E officers and other staff involved in the program's data collection and management. 5. Reliable and protected storage facilities such as the cloud storage should be provided for the data officers to prevent loss and unauthorized access to the program's data. 6. Only qualified and reliable Data Quality Assessors should be engaged to carry out data quality assessment on the key performance indicators of the program. 7. There should be a simple unified electronic template which should be created before start-up of any data collection for a better data management, 8. The Program should have web-base system (M&E/IMS) to further improve the quality of data generated and better data management.

REFERENCES:

BHA DQA Webinar handout, USAID/FANTA/FHI360 (2016). Retrieved from https://www.usaid.gov/sites/default/files/documents/USAIDBHA_DRAFT_Emergency_ME_Guidance_April_2021.pdf Data Quality Assessment Checklist and Recommended Procedures (November 23, 2018). Retrieved from https://www.usaid.gov/sites/default/files/documents/1865/_508_Data_Quality_ 20Assessment_Checklist.pdf Data Quality Assessment for Neglected Tropical Diseases: Guidelines for Implementation (NTD-Guidelines, December 2013). Retrieved from https://pdf.usaid.gov/pdf_docs/PA00JZS3.pdf Data Quality Dimensions adapted from (DAMA-UK, October 2013). Retrieved from https://www.researchgate.net/publication/341650593_Big_Data_Quality_Dimensions _A_Systematic_Literature_Review Food for Peace Data Quality Assessment Webinar Handouts. Retrieved from https://www.fantaproject.org/sites/default/files/resources/Handouts_DQA-webinarMar2016.pdf Participatory Data Verification & Improvement Tool (fhi360-dvt-oct2013), Retrieved from https://www.fhi360.org/sites/default/files/media/documents/fhi360-dvtoct2013.pdf

ASSESSMENT TEAM:

IBRAHIM M. ARABI National Program Co-ordinator

Engr. Ibrahim Manta

Alh. Auwalu Ado Shehu

Zonal Program Co-ordinator Bida-Badeggi SCPZ

Zonal Program Co-ordinator Kano-Jigawa SCPZ

Dr. Abubakar Aliyu Zonal Program Co-ordinator Kebbi-Sokoto SCPZ

CONSULTANT

Mr. Romanus Egba Zonal Program Co-ordinator Adani-Omor SCPZ

TECHNICAL TEAM ATASP-1

Dr. Adewumi Adeoluwa

Mr. Ojumoola Aliyu O.

Mr. Akintunde Akinwale P.

Mr. Chukwuma Ejiogu

Co-ordinator

Member

Dr. Onyekinesof JohnPaul C. Member

Mrs. Falmata Zanna G

Mr. Musa Danlami M.

Mr Mamun Mallam

Engr. Ja'afar Yusuf Bashir

Mr. Saidu Sani Gulma

Member

ASSESSMENT CHECKLIST: INDICATOR: YES NO COMMENTS (1.) VALIDITY – Data should clearly and adequately represent the intended result. 1 Would an outsider or an expert in the field agree that the indicator is a valid and logical measure for the stated result (e.g., a valid measure of overall nutrition is healthy variation in diet; age is not a valid measure of overall health)? 2 Does the indicator measure the contribution of the project? 3 Is there reasonable assurance that the data collection me thods being used do not produce systematically biased data (e.g., consistently overor under-counting)? 4 Are the people collecting data qualified and properly supervised? 5 Were known data collection problems appropriately addressed? (2.) RELIABILITY – Data should reflect stable and consistent data collection processes and analysis methods over time. 1 Is a consistent data collection process used across time, locations, and data sources? 2 Are there appropriate written procedures in place for periodic review of data collection, maintenance, and processing? 3 Are the written procedures for periodic review of data collection, maintenance, and processing consistently practiced? 4 Are data collection and analysis metho ds documented in writing and being used to ensure the same procedures are followed each time? (3.) TIMELINESS – Data should be available at a useful frequency, be current, and timely enough to influence management decision making. 1 Are data availa ble frequently enough to inform program management decisions? 2 Is data properly stored and readily available? 3 How has the information been used to make management decisions?

(4.) PRECISION – Data has a sufficient level of detail to permit management decision making; e.g., the margin of error is less than the anticipated change. 1 Is the margin of error less than the expected change being measured (e.g., If a change of only 2% is expected and the margin of error in a survey used to collect the data is +/- 5%, then the tool is not precise enough to detect the change)? 2 Has the margin of error been reported along with the data (only applicable to results obtained through statistical samples)? 3 Is the data collection meth od/tool being used to collect the data fine -tuned or exact enough to register the expected change (e.g., a yardstick may not be a precise enough tool to measure a change of a few millimeters)? (5.) INTEGRITY – Data collected should have safeguards to minimize the risk of transcription error or data manipulation. 1 Are procedures or safeguards in place to minimize data transcription errors? 2 Is there independence in key data collection, management, and assessment procedures? 3 Are mechanisms in place to prevent unauthorized changes to the data? SUMMARY Based on the assessment relative to the five standards, what is the overall conclusion regarding the quality of the data?

Significance of limitations (if any):

Actions needed to address limitations:

IF NO DATA ARE AVAILABLE FOR THE INDICATOR If no recent relevant data are available for this indicator, why not? What concrete actions are now being taken to collect and report these data as soon as possible? When will data be reported?

COMMENTS

NOTE

NATIONAL OFFICE

No. 15, Lord Luggard Street, Asokoro, Abuja FCT, Nigeria info@atasp1.gov.ng, atasp1_hq@atasp1.gov.ng 08137208947, 08036551491 www.atasp1.gov.ng Facebook/ATASPNigeria Twitter @ataspnigeria

ZONAL OFFICES ADANI-OMOR SCPZ

BIDA-BADEGGI SCPZ

ADP Complex, KM 41, Enugu-Onitsha Express way, Kwata Junction, Awka, Anambra State. a.omor_scpz@atasp1.gov.ng 07081037456

Farm Institute, Ministry of Agriculture and Rural Development, KM 12, Bida-Lemu Express way, Bida, Niger State. b.badeggi_scpz@atasp1.gov.ng, ataspbidazone@yahoo.com 08132756066

KANO-JIGAWA SCPZ

KEBBI-SOKOTO SCPZ

No. 9, Ahmadu Bello Way, Servicom Center, Kano, Kano State. k.jigawa_scpz@atasp1.gov.ng 08036923665, 08052683453

KM 11, Kalgo Junction, Bernin Kebbi-Jega Road, Bernin Kebbi, Kebbi State. k.sokoto_scpz@atasp1.gov.ng, kbsoatasp1@gmail.com 07037777213

Data Quality Assessment of ATASP-1 KPIs

Articles inside

2.1.1 Accuracy of Data

3.2 Compliance to Data Encoding and Submission Standards

2.1.2 Reliability of Data

3.3 Data Verification

1.2 Data Quality