The Power of Evaluation

9

PRACTICE BRIEF

Seed Aid for Seed Security ADVICE FOR PRACTITIONERS

The Power of Evaluation

T

Practitioners should embrace evaluation as an exceptional learning opportunity.

Donors need to support practitioners in contributing to the body of knowledge, rather than merely holding them accountable for mistakes made.

he current state of evaluation in seed aid is dismal indeed. In principle at least, practitioners understand and embrace the importance of evaluation in learning from experience and improving performance.

Unfortunately, however, seed interventions are often seen as straightforward, one-off, and output focused: react quickly, distribute seed and close out. Evaluations are not relevant.

This attitude has resulted in a remarkable stagnation in how seed security is understood and assessed and how interventions are planned and implemented. Fortunately, ideas are changing. With a growing realization that seed systems are complex and resilient and that local institutions – especially markets – can and should play a central role in recovery, initial seed aid responses are now seen as the first important step in an ongoing process that may last many years. Evaluation has become essential to ensure that experience leads to learning and that learning informs the next step in the process. This will lead to better projects, which in turn will result in stronger and more resilient seed systems that underpin sustainable seed security.

Evaluations should help to correct common and immediate problems such as poor targeting, unsuitable crops or varieties on offer and dependency creation.

Rather than turning their attention to evaluation at the end of implementation, practitioners should reflect on the evaluation when designing the intervention.

What should be the outcomes of the intervention and for whom? For how long, and at what intervals, will the agency need to monitor the range of effects of its assistance? Time and budget commitments should be made accordingly.

Types of evaluation There are several different types of evaluations.

Real time evaluations One can use interviews, for example just after seed distribution or as people leave seed fairs, to obtain feedback from beneficiaries. This feedback is then used immediately to inform the next planned event. Real time evaluations monitor information to ensure that the process is on track and that problems are identified and corrected as quickly as possible.

Output evaluations Interviews are conducted right after the intervention (within one month) to provide feedback from practitioners, partners and beneficiaries on the logistics of the intervention (its timing, targeting, distribution mode, etc.). This is the classic type of post-mortem evaluation that satisfies donor requirements and closes a project.

Outcome evaluations At the end of the cropping season interviews evaluate the effectiveness or outcome of the intervention in terms of impact on crop production and next

9

PRACTICE BRIEF

season’s seed security. An outcome evaluation shifts the focus from what was done (outputs) to what might be done next to support continuing recovery.

Impact evaluations Longer term follow up, conducted after three to five seasons, aims to evaluate the broader impact of the interventions on seed system resilience and food security. This type of evaluation seeks to capture and share learning and best practices for the wider practitioner community.

Meta-analyses and evaluations This type of evaluation compares several interventions at once. The interventions may be of the same type (for instance, a range of direct seed distributions) or they may represent different approaches, such as direct seed distribution and seed vouchers and fairs. Metaanalyses may even assess the totality Evaluations must of seed system address concerns of interventions in a basic intervention given geographical area. Such effectiveness. Have evaluations can activities made a also be used difference to farmers, to compare performance farming systems and across countries, the local economy? with different seed systems, experiencing different disasters and different levels of seed insecurity.

Meta-analyses generally focus on the effectiveness of the approach itself. They are of special interest to practitioners committed to learning how to improve seed-aid planning and implementation by deepening their understanding of seed systems and the strengths and weaknesses of different kinds of response.

Evaluation as the cornerstone of learning Evaluations present a wonderful opportunity for learning – first and foremost for the implementers but also for the wider practitioner and donor communities. The challenge for practitioners is to stop treating evaluations as an onerous requirement and to recognize them as the exceptional learning opportunities that they are. The discussion of whether evaluations should be internal or external misses the point – which is that they should focus on learning. That requires the practitioners to be actively involved in the evaluation with the intention of using the results to improve practices. Therefore,

2

The Power of Evaluation

perhaps one of the better models involves an externally-facilitated evaluation.

Although donors accept output-focused post mortems, they also support rigorous outcome evaluations. The challenge for seed aid donors is to become more proactive in supporting evaluation. This will require donors to embrace learning and sharing as the principle objectives of evaluation, rather than regarding evaluation as simply reporting and closing. Donors need to communicate their support for learning-focused evaluations. They should shift their focus from holding practitioners accountable for mistakes made to supporting practitioners in contributing to the body of knowledge on seed assistance. It is also vital that donors give attention to how those implementing – and everyone else involved in seed aid – subsequently apply the lessons learned during the evaluation.

Guide Questions for Different Types of Evaluation In the table opposite we suggest some of the varied issues that might be embraced by the different types of evaluations introduced in the previous section.

The list is suggestive, to give examples of key issues at different levels of evaluation, and is a long way from being exhaustive. What is important is to emphasize that: ■ The key evaluation issues change through time.

■ All four types of evaluations are important and are not interchangeable.

■ The recipients’ views and actual effects on the ground have to figure among the essential elements.

Note that current evaluation and monitoring, if done at all within seed aid projects, is generally limited to the inputs distributed and the efficiencies of the operation (its timeliness and numbers of beneficiaries reached). Evaluations have to address concerns of basic intervention effectiveness, such as whether the precise activities made a difference to the farmers in the farming system and more broadly to the local economy. While the insights of implementers are important for improving practice, the recipients’ point of view should be given equal weight; to do so requires considerable field time to be allocated for evaluation.

To reiterate, evaluations at all levels present important opportunities for learning and thus to improve practice. However, such evaluations require real reflection and commitment as well as time, energy and financial resources. In completing the cycle, practitioners have to be prepared to use the results for specific projects and to incorporate their wider lessons into future program design.

PRACTICE BRIEF

9

TABLE 1 Themes to address in evaluation Seed System Relief and Evaluation Overview: Select themes to be addressed Type of Evaluation

Agency’s assessments of

Recipients’ assessments of

Real-time (during intervention)

Insights (from diverse perspectives) on: ■ Products on offer (crop and variety choice, seed quality, seed amounts) ■ The immediate intervention process, whatever recipients signal as important, e.g.

• Length of intervention, including waiting time • Number and order of farmers served • Adequacy of support personnel

Output (after about one-month)

Insights (from diverse perspectives) on the efficiency, organization and logistics of intervention.

■ Timing (especially in relation to subsequent planting) ■ Targeting (process and perceived ‘fairness’) ■ Choice of locales ■ Choice of crops and varieties ■ Adequacy of seed quality on offer (and validity of process guiding quality verification) ■ Adequacy of preparatory information or sessions ■ Scale (numbers served, overall amounts of seed or products delivered or made accessible) What worked? What was missing? What modifications should be made in future?

Outcome (after first season)

Insights on first effects of intervention.

Recipient Focus: ■ Yield performance and farmer satisfaction with crops and varieties obtained as aid (qualitative and quantitative variety attributes) ■ Importance of seed aid in relation to farmers’ other seed sources • What proportion of the aid given was sown and why?

• What proportion of the total seed sown came from aid (versus home-saved seed, local markets, exchange) and why?

Farming System and Implementer Focus: ■ Was the impact of the disaster on farming systems sufficiently understood to guide planning (looking with hindsight)?

■ Was the general choice of intervention valid (and linked to a specific seed security need?)

■ Was the intervention actually needed? Evidence?

■ Did the intervention strengthen or protect seed security? Evidence?

■ Which broad groups were reached by the intervention and which not?

■ Were there any unanticipated positive effects?

■ Were there any unanticipated negative effects?

What worked? What was missing? What modifications should be made in future?

Impact evaluations (after several seasons)

Impact – positive and negative – of intervention on: ■ Stability of production and food security ■ Biodiversity of crops and varieties ■ Household income and local economy ■ Seed channel functioning, including local seed/grain markets and development of commercial enterprises ■ System resilience to possible next set of shocks CONTINUED ON PAGE 4

SEED AID FOR SEED SECURITY ADVICE FOR PRACTITIONERS

3

9

PRACTICE BRIEF

Seed System Relief and Evaluation Overview: Select themes to be addressed

4

Type of Evaluation

Agency’s assessments of

Meta-analyses (after cluster of interventions completed)

Content here would vary according to what is being compared. Some general guide themes.

Strengths and weakness of a specific type of intervention (e.g. Direct Seed Distribution) for specific contexts (e.g. civil strife, flood, drought).

■ For whom?

• farmers – male – female • children • traders • commercial companies ■ immediate effects • range of benefits and costs – agronomic – environmental – economic – social ■ longer-term effects • range of benefits and costs – agronomic – environmental – economic – social Comparative advantages among different interventions (e.g. cash and voucher delivery, direct seed distribution, seed vouchers and fairs, seed protection rations) ■ Which contexts?

■ For whom?

■ Immediate effects?

■ Longer-term effects?

The Power of Evaluation

Recipients’ assessments of