TRAINING KIT MODULE 8 RESULT-BASED MANAGEMENT: DATA, QUALITY, COLLECTION AND ANALYSIS
Summary ● Usefulness of statistics in the RBM context ● Monitoring and Evaluation and the use of reliable data ● Principles linked to statistical data ● Concepts and methods of data collection
● Mechanisms of data storage and dissemination
M&E System: a question of data‌ A M&E system = the combination of planning, collection, analysis, data use, and the dissemination of M&E results. It takes advantage of sufficient resources and required expertise to implement M&E with the aim of improving decision-making for management and programme implementation, performance improvement, and experience building.
M&E information system: need for data Once all the monitoring and evaluation instruments of a programme have been prepared (logical framework, M&E plan, information system, etc.), it is time to suply the system with quality data ‌. Supplying the system is not limited to data collection; there are many stages beyond that: (1) data input and formatting; (2) storage; (3) processing and analysis; (4) dissemination of M&E information; and (5) use of M&E information. Data quality is an integral part of managing M&E data, and thus the manager of a development intervention must constantly pay attention to it throughout the process of producing M&E information.
Why is data quality important? Quality is defined as the totality of characteristics and properties of a product or service which confers upon it the ability to satisfy expressed and implicit needs. Source: Norme ISO 8402 (1986)
For a company or development agency, there is data with an activity’s success and the achievement of programme objectives. Their quality represents a critical issue for three stages of its life cycle. After input; During transformation and aggregation; and, During data analysis and the presentation of results.
Data Quality: Required Criteria (1/2) Quality Criteria
Characteristic
Example of Indicators
Opportunity
Does the age of the data • Date of data collection meet needs? • Date of last processing • Version control
Entirety (or Completeness)
Is all the necessary data • Entirety of optional values available? • Number of non-assigned values • Average number of default values
Coherence
What is the contradictory data and information source?
• Test of plausibility • Standard deviation value
Source: Informatica (2008
Data Quality: Required Criteria (2/2) Quality Criteria
Characteristic
Example of Indicators
Precision
Do the values match reality?
• Frequency of change in values • Reaction (feedback) of clients
Interpretation
Can the data be understood by users?
• Data validation • Sector breach
Standardisation (or Conformity)
What is the standard format for inputted, stored or publicized data?
• Certificate of conformity
Duplication
What is the repeated data?
• Number of duplicated records
Source: Informatica (2008
Data Qualtiy: Some Verifications Bad data quality is due primarily to data record errors at the moment of data collection or data input errors when beind added to a database. Orthographic mistakes, erroneous codes, incorrect abbreviations, inputting in a wrong column are also instances and sources of reducing data quality which can have harmful consequences during later stages of information processing, analysis and diffusion. Some measurement tests for data quality (statistical rates)can be used to check the quality of data collected and can prove necessary for creating a culture of quality. A very popular practice in the data management sector is the implementation of a quality control survey. This mechanism allows for random verification of the data collection and input processus and to detect potential errors.
Before Collection: Reflections‌ 1. 2. 3. 4.
5. 6. 7.
7 basic questions to ask Why collect M&E data? Which M&E data must be collected? How frequently must M&E data be collected? How should the M&E data be collected (methods and tools)? Who will be in charge of collecting M&E data? (think about necessary resources!!!) How will M&E data be stored, analysed and disseminated? What difficulties could stall the collection of M&E data? How can they be overcome?
Data Collection: Think about aggregation… Only a data analysis strategy helps determine the format in which data will be collected, processed, organised, classified, analysed, aggregated and presented.
1
Director wants to learn about the progress in the training program. It sends an email to the regional offices that says "Please inform DG on the program training before Monday"
2 The regional office 1 says: "We trained 357 people”
The regional office 1 replied: “we organized 45 training sessions"
3 Director could not aggregate the data and get a consolidated picture of progress Source: Görgens & Kusek (2009).
Collecting M&E data: what to collect? More for monitoring than for evaluation, the quality of information produced depends on the availability and collection of reliable data ( of good quality). For monitoring, data collection must be regular and concerns all achievements. For evaluation, it usually starts with sampling. A good collection of data requires (1) choosing an appropriate collection method et, (2) preparation of compehensive collection tools and controling value judgment to a large degree.
M&E data collection: which method? The choice of collection method depends on the objective of the investigation, the type of variable for measuring and the available resources/capacities.. Quantitative methods, structured or standardised, for collecting and analysing numerical data (physiological measures; structred observation measures; investigation/survey; scales). Qualitative methods, semi-structured or open, for producing in-depth and descriptive information (collective or individual interviews; measured through non-structured observation; investigation/surveys; scales). The two types can be used in complement of each other (mixed method) for gaining both numeric and descriptive data. Each method requires the development of an appropriate collection tool.
Assessment of a data collection method 1. What method will be used for measuring a parameter or a variable? Is it appropriate for this parameter or that variable? 2. Is the method clearly described? Does it guarantee the quality, accuracy and validity of the data? 3. When using the observation method, are the object of observation and the units of analysis clearly defined? Is the data collection process, by observation, clearly described? 4. In the case of an interview/survey, does the guide or questionnaire sufficiently describe the subject being considered? Does it account for anonymity and confidentiality? 5. Where and in what circumstances will the data be gathered?
Collection Tool Problems (1/2) 1. Lack of Clarity: a major source of errors in questionnaires, very often worsened by the diversity of interviewers. Ask clear, concise, simple, unequivocal questions which have the same significance for everyone. 2. Use of jargon: Technical terms and bureaucratic jargon are not always used and understood by all. Limit their use, if possible. 3. Suggestive questions: can lead to a specific response (manipulation of the study). Always avoid these kind of questions as they generate uncertain data. 4. Words or phrases with a negative or positive connotation: Two quasi-smiliar words can have different connotations. Always study the meaning of words to be used.
Developing Collection Tools (2/2) 5. Embarrassing questions: can put people being question in an uncomfortable situation. Avoid this as it can close access to a potentially useful information source. 6. Hypothetical questions: based on conjecture (for example, "if you were chief of police, what would you do to control crime?"). To be avoided because it does not permit the collection of reliable data that is representative of real opinions. 7. Preference for prestige: some informants might have the tendency of responding in a staged or misleading manner. In this case, triangulation might be necessary.
Data Input: some problems…
Data Errors and Quality Control Transposition Errors (39 inputted instead of 93), often resulting in other mistakes. Input errors(1 is inputed as 7 or 0 saisi comme O or I inputed as 1). Coding errors: entering an incoorect code (an interviewer circles 1=Oui, and the input agent records 2=Non). Sorting errors, occurs when a person fills a questionnaire and puts numbers in an incorrect place or order. Consistency errors, when there are contradictory responses on the same questionnaire (birth date and age). Range errors, when a response falls outside the range of possible or probable values.
Data Input: necessary standardisation Computerisation, storage and transfer of data The risk of error multiplies with many aggregation levels in the M&E system and/or based on the manual transcription or discontinued computerisation (multiple manual input). It is recommended to support a M&E system with an information system which reduces the number of inputs and protects data integrity. In such a system, information system access options are adapted to the roles and responsibilities of the diferent stakeholders. Data input is undertaken once and according to the appropriate data entry form which guides the data entry agent. In the case of a web platform, data aggregation according to defined levels is automated and the transmission of raw and aggregated data is instant.
Analysis and Interpretation of M&E data Analysis and Interpretation of Data Data Analysis = verifies the achievement of programme objectives and summarises data. Does not necessarily signify the use of a sophisticated information programme… Instead, it is the assessment of collected data with respect to questions asked (to know if the programme is operating as expected; to compare objectives and its actual performance). Interpretation = to discover causes for its performance resulting from M&E evidence. The analysis and interpretation of M&E data helps generate information which could help in decision-making on a programme.
M&E Data Analysis Methods (1/3) Quantitative Data Analysis Analysis of quantitative rests mainly on statistical methods that all M&E specialists must master. There are descriptive methods and deductive methods. Descriptive methods: give insight into the distribution of values of principle variables under analysis. Central Tendency measures: mode, median, average Dispersion measures: spread, variance, standard deviation, variation coefficient, etc. Association measures: Contingence table, correlation coefficient These measures are always illustrated with frequency histogrammes, sectoral diagrams(pie charts), etc.
M&E Data Analysis Methods (2/3) Quantitative Data Analysis (Continued) Deductive Methods: predicting the behaviour or characteristics of an entire population by deducing from a sample through deduction. Parametric and non-parametric tests: difference of average test (t of Student), variance analysis(ANOVA), comparison of proporations test (khi 2), correlation test(r de Pearson)… Multifaceted analysis: Covariance analysis(ANCOVA), Analysis of Multiple Variance(MANOVA), Multiple Correlation, Regressions... Without deductive methods, rely on hypothesis for testing, formulated on a theoretical framework which specifies the relationships between the variables to be analysed.
M&E Data Analysis Methods (3/3) Qualitative Data Analysis Method applied to analysing non-numeric data collected in the M&E framework. Non-structured observations, open interviews, documentaries and transcripts from work sessions all constitute qualitative analysis methods. Analysis of themes and trends: method for summarising ce what one has seen or what one has heard in the form of words, sentences, themes or common trends. Specify the frequency of specific themes in order to demonstrate the prevalence of an expressed point of view. Content analysis: systematic approach which entails the identification and smmary of the contents of a message. It is de easier to remember a quote than a document.
Thank you for your attention.
AfCoP Web Site: http://copmfdrafrica.ning.com