Tips on M&E indicators

INDICATORS: DEFINITIONS AND DISTINCTIONS An indicator is a measure that is used to demonstrate change in a situation, or the progress in, or results of, an activity, project, or programme.

TYPES OF INDICATORS Indicators are essential instruments for monitoring and evaluation, and exist in many different forms.

•

Direct

Indirect/proxy •

• Qualitative

•

Quantitative

Direct indicators correspond precisely to results at any performance level. For example, counts of children vaccinated are direct measures of output; instances of change in hand-washing behaviour are a direct measure of project/programme outcome; and a decrease in infant mortality is a direct measure of project/programme impact.

Indirect or "proxy" indicators are used to demonstrate the change or results where direct measures are not feasible. Some objectives, particularly impact objectives, are difficult to monitor. It is often necessary to select indirect or proxy indicators that may be easier for evaluators to measure. For example, the effectiveness of a child health programme may best be measured by mortality rates. These rates are difficult to determine over short periods of time. For this reason, a proxy indicator, such as the percentage of births that are attended by trained health personnel, and the availability and frequency of use of health facilities may be used.

Indicators are usually quantitative measures, i.e. a percentage or share, rate (e.g. birth rate), or ratio (e.g. inhabitants/doctors).

Qualitative indicators can be defined as people's judgements and perceptions about a subject.1 For example: the number of people owning sewing machines in a village is a quantitative indicator, the confidence those people have in sewing machines as instruments of financial independence is a qualitative indicator. Note that "quality of life" indicators, such as those that measure changes in a population's health, education or employment, are often confused with qualitative indicators, as they both refer to "quality". In fact, either qualitative or quantitative indicators can measure health, education or any other subject.

.

Global standardised Locally developed

Impact

•

•

•

Performance indicators "show results relative to what was planned" (OECD-DAC) at each level of the "results chain" (see next page) — inputs, processes, outputs, outcomes, impacts.

•

They are tools for performance-based decisions about programme strategies and activities. They can also be used later to evaluate project/programme success.

Outcome Output Input

In UNICEF’s work to date, health and child survival programmes can most often be monitored using standardised global indicators (comparable in all settings). For example, the procedures for measuring nutritional status (e.g., weight for age) and immunization coverage have been standardised by WHO. The indicators for measuring change in areas such as child protection have not been standardised as yet. Indicators for many such areas tend to be context-specific and must be developed locally.

1 CIDA, 1997, “Guide to Gender Sensitive Indicators”.

UNICEF M&E Training Resource

Indicators 1/3

PERFORMANCE INDICATORS Definitions

Example of hierarchy of indicators for a health education campaign

Impact indicators: Long-term results Measure the quality and quantity of long-term results generated by programme outputs (e.g. measurable change in quality of life, reduced incidence of diseases, increased income for women, reduced mortality).

Reduction in morbidity and mortality

*In the absence of more definite impact indicators, early pointers of impact may be used during project implementation to indicate progress toward achieving project objectives. For example, in many cases, output indicators and indicators of risk factors can serve as suitable intermediate or leading indicators of impact.

Outcome indicators: Medium-term results Measure the intermediate results generated by programme outputs. They often correspond to any change in people’s behaviour as a result of programme, e.g. the use of oral rehydration therapy.

Proportion of target population that meet prescribed standards of behaviour Number whose behaviour change

Output indicators: Short-term results

Number whose opinions change

Measure the quantity, quality, and timeliness of the products — goods or services — that are the result of an activity/ project/programme.

Number who learn the facts

Process indicators: Measure the progress of activities in a programme/project and the way these are carried out (for example, referring to the degree of participation).

Number who read the literature Number of people who receive the literature

Input indicators: Measure the quantity, quality, and timeliness of resources — human, financial and material, technological and information — provided for an activity/project/programme.

Amount of literature distributed Number of pieces of literature available for distribution

Source: adapted from Patton 1997: 220.

Limitations of the results chain The "results chain" is the expected sequence of results to achieve the desired programme objectives, beginning with inputs, moving through activities and outputs, and culminating in outcomes, impacts and follow-up (OECD-DAC, 2001). It is generally just a reference for developing performance indicators, as it obviously a simplification of reality. In practice, it is sometimes difficult to distinguish between a programme’s outputs and outcomes or between outcomes and impacts. An outcome indicator for a small project may represent an output for a larger programme. In addition, there are often multiple intermediate results, both for outputs and outcomes.

When defining indicators, it is important to think of the types of indicators on a continuum in logical relation to the project's/programme’s strategic objective. Where there are multiple intermediate results, you may end up with two sequential so-called "output indicators" or "outcome indicators". In other words, it doesn't matter so much whether you label something an output or an outcome indicator as long as it fits with the logical sequence of results of the programme in question and fits the characteristics of a good indicator. (See content sheet "Selecting indicators — Criteria".)

UNICEF M&E Training Resource

Indicators 2/3

Challenges posed by impact indicators Usually it is easier to define and measure simple output indicators, as impact indicators can be complex, costly, and difficult to measure. The time lag between programme implementation and impact is significant.

As we move toward impact, we see that numerous contextual factors can influence results. Before-and-after household and community surveys are excellent tools for collecting comprehensive impact data for ex-post evaluation, but the time and cost they require make them impractical for regular programme monitoring or evaluation during programme implementation.

We, therefore, often seek proxy measures that can inform us early on about a programme’s likely impact.

Monitoring performance indicators during implementation can provide this information. Leading process and intermediate indicators can serve as proxies for impact indicators. Beneficiary assessments, rapid rural appraisals, and focus group interviews are useful ways of collecting qualitative impact data.

Impact and outcome indicators must be defined and interpreted with care. Demonstrating that a project caused a particular impact is usually difficult since many factors outside the project influence the results (DAC). See Core Content Sheet "Evaluating impact" (Module 4).

INDICATORS, TARGETS, BENCHMARKS An indicator is the measure; for example, the proportion of under-fives who fall below minus 3 standard deviations from median weight-for-height of NCHS/WHO reference population.

A target is the explicit statement of desired results for a specific indicator over a specified period of time; the planned performance standard against which actual performance may be subsequently compared and measured. It is usually specified in terms of quantity or quality, target area or target group (intended primary stakeholders), and time. For example: Reduction of severe and moderate malnutrition by 20 percent by 2002.

A benchmark is also a reference point or standard against which performance or achievements can be assessed. The distinction is that a benchmark refers to the performance that has been achieved in the recent past by other comparable organizations, or what can reasonably be inferred to have been achieved in the circumstances (OECD-DAC, 2001). In the context of situation monitoring, a benchmark represents a reference to a standard of what is achievable. For example, the median weight-for-height of NCHS/WHO reference population is a benchmark in nutritional status.

Note that some organisations or sources use these terms interchangeably, which can cause confusion.

UNICEF M&E Training Resource

Indicators 3/3