IPDE T
Module 4:
Understanding the Evaluation Context and Program Theory of Change
Introduction • Front-End Analysis – Identifying the Main Client and Key Stakeholders – Understanding the Context – Investigating Existing Knowledge – Constructing, Using, and Assessing a Theory of Change
IPDET © 2009
22
Front-End Analysis • • • • • • • •
Client and stakeholders Timing Time management Nature of resources Relevance of social science theory Findings of similar evaluations Role of theory of change Existing data
IPDET © 2009
33
Balancing Cost and Benefits:
Benefits • Evidence-based decision making that leads to sound decisions about: – – – –
what to scale up or replicate What to continue what to improve what to decrease or terminate
• Contributions to streams of knowledge • Building of local capacity IPDET © 2009
55
Stakeholders • People or organizations other than the client with strong interests in the intervention and its evaluation – May stand to gain or lose from the outcomes – May represent the interests of such groups
• Important to consider those who would typically not be asked to participate IPDET © 2009
88
Checklist of Stakeholder Roles Individuals, groups, or agencies
To make policy
To make To provide operational input to decisions evaluation
To react
For interest only
Developers of program Providers of other resources Heads of implementing agencies etc. IPDET © 2009
99
Theory of Change • A knowledge-based diagram of how an intervention intends to achieve results • T of C should: – depict a sequence linking inputs to activities, activities to outputs, and the results--outcomes and impacts– expected from them – identify critical assumptions underlying the intervention – Indicate other factors in the context of the intervention that might influence its success IPDET © 2009
15 15
Potential Influences on Program Results Political Environment (Governance, etc.)
Policy Context Macroeconomic Picture
Inputs Activities Outputs
Public Attitudes
Black box
Aid Players
IPDET © 2009
Outcomes Impacts
Environment
16 16
Why Use a Theory of Change • Visually shows how the intervention is supposed to work and flaws in logic • Assumptions and potential influences identified may indicate risks to the achievement of intended results • Provides questions for an evaluation • Helps build a common understanding of the intervention and expectations among stakeholders IPDET © 2009
17 17
How to Determine the Logic of the Intervention • Chain of “if-then” statements – If X happens, then Y should follow
• Ask, is this plausible? Are the activities and outputs likely to lead to the intended results? • Ask, is this feasible? Are the inputs (capabilities and resources) sufficient for the activities?
IPDET © 2009
20 20
Mapping or Drawing the Logic • Can and are done in many different ways • Usually use forms of flow-charting • Boxes with arrows show links and relationships
IPDET © 2009
21 21
Results Chain Area of Control Internal to the Organization
Inputs Inputs
(Resources (Resources ))
Activitie Activitie ss
Outputs Reach Direct Beneficiaries
Outputs Outputs
Outcomes Outcomes (Intervening) (Intervening)
Area of Influence External to the Organization
Outcomes Outcomes
Impacts Impacts (Longer (Longer Term Term Goal) Goal)
External Factors
IPDET © 2009
22 22
Results Chain for Training Program Inputs Resources ▪ Money ▪ Staff ▪ Volunteers ▪ Supplies • Eligible participants
Activities
Outputs
Services
Products
▪ Vocational skills training
▪ Avg. hours of skills training per participant
▪ Literacy education ▪ Counseling
IPDET © 2009
▪ Avg. hours of literacy ed. ▪ % of participants completing program
Outcomes Benefits % of participants ▪ increased literacy ▪ gained new vocational skills ▪ improved work attitudes
▪ % of participants placed in jobs paying at least $5 per hour
Impacts Changes ▪ % of trainees earning more over three years than those not receiving training ▪ % of trainees with higher standard of living than the control group
23 23
Simple Theory of Change Model Training Training
By training
IPDET © 2009
High High Quality Quality Evaluations Evaluations
Useful Useful Information Information
Better Better Decisions Decisions
We should get better decisions
24 24
Identifying Key Assumptions Make explicit the implicit assertions about how the intervention will work in its context to produce the results
IPDET © 2009
25 25
Theory of Change with Assumptions Report writing skills adequate to communicate with government
Needs of trainees met
Trainin Trainin gg
Enough time to learn
IPDET © 2009
High High Quality Quality Evaluations Evaluations
Useful Useful Informatio Informatio nn
Better Better Decisions Decisions
Resources given to follow what they learned
26 26
Theory of Change with Assumptions and Context National National Evaluation Evaluation Society Society
Needs of trainees met
Trainin Trainin gg
Macro Macro economic economic environmen environmen tt Report writing skills adequate to communicate with government
High High Quality Quality Evaluations Evaluations
Enough time to learn Competency Competency Based Based Movements Movements
IPDET © 2009
Useful Useful Informatio Informatio nn
Resources given to follow what they learned
Special Special Interest Interest Groups Groups Progra m Visibilit y
Better Better Decisions Decisions
Political Political Support Support
27 27
Flow Chart of Classic Logic Model Inputs
Activitie s
Outputs
Outcome s
IPDET © 2009 Impacts
28 28
Begin with the end goal?
IPDET © 2009
29 29
Intermediate Outcomes Inputs
Student s
Instruct or
Activities
Classroo m
Trainin g # of Trained Students
Outputs
Intermedia te Outcomes
Material s
% of participants indicating ability to design an evaluation has improved
Outcomes
IPDET © 2009 Impacts
% that produce higher quality evaluatio n designs
% of designs approved and implemente d
% of those communicati ng findings clearly and concisely
Useful informati on provided
Decision makers consider findings
Better decisions made Stronger development programs
30 30
Exercise: Unlabeled Theory of Change Achi evement in readi ng Improvement of condition (health, emotional)
Student morale
Teaching in terms comfortable and understandable to students
Conscientiousness of work by students
Teachers’ sympathy with children and their view of the world
Parental support and encouragement with child’s homework and school assignments
Teachers’ understanding of the home culture
Student attendance
Student’s receipt of special help
Parental support for better attendance at school
Parents’ knowledge of school’s expectations for students
Parents’ knowledge of school’s expectations for students
Identification of special problems that retard student’s achievement (health, emotional, etc.)
Sharing of views by parent and teacher
IPDET © 2009
Visits by teachers to students’ homes
31 31
Theories of Change should Address these Questions: • • • •
Is the intervention built on a knowledge base? Are there gaps in the logic chain? Are interrelationships clear and plausible? Is it realistic to assume that the intervention will result in the attainment of stated goals? • On what critical assumptions is the intervention based? • What else is operating in the context of the intervention that might affect results? IPDET © 2009
34 34
A Final Note…. “One does not discover new lands without consenting to lose site of the shore for a long time.” -- Unknown
Questions ? IPDET © 2009