Outcome Mapping: Measuring Progress in Partnerships and Networks Heidi Schaeffer, Knowledge Management and Learning Lead Association of Ontario Health Centres heidi@aohc.org outcomemapping.ca
Objectives
• Gain an understanding of Outcome Mapping, it’s key concepts and methodology. • Dabble in applying the concepts of Outcome Mapping to a Health Link network.
The fish soup story: Inputs, Outputs, Outcomes and Impacts
Inspired by Monika Jetzin, Global Water Partnership, Hungary
Inputs or resources Parents get together fish, fresh vegetables, water, barley, spices, pot, source of heat Activities Mother or father carefully prepare and cook all the ingredients
Parents control
Output Children taste the most nourishing fish soup in the world
Outcome Children consider the soup delicious and eat fish soup once a week for the rest of their lives Impact Children are healthy adults
Parents influence
Parents contribute a bit
The difference between simple and complex fish soups
? ď ś
Example adapted from Getting to Maybe, How the World Is Changed?, Frances Westley, Brenda Zimmerman, Michael Q. Patton, Random House Canada, 2006.
A simple situation The relations of cause and effect are known. The parents follow the great grandmother’s recipe for fish soup. The quantity and nature of the ingredients are spelled out, as well as the order in how they should be combined. The parents do not need expertise although of course experience in cooking helps. If they follow the recipe they will produce basically the same soup week after week.
If only life were so simple! In the real world, the results can be much less certain. The great grandmother’s recipe is lost. Her recommended fish is not available in the market every week of the year. The family’s buying power varies from year to year. Children are different and change as they grow: One becomes a vegetarian. Another goes on a diet. A third is simply rebellious. Outside factors and actors influence the children – school, TV, friends and so forth
In such a complex situation The relationships of cause and effect are unknown until the outcomes emerges. To produce a nutritious soup that their children will eat once a week for the rest of their lives, the recipe is less important than the parents’ relationships with each son and daughter, and theirs with their social environment. More than cooking experience, parent’s must rely on their sensitivity to their children’s needs and their own creativity in the kitchen. And they must accept uncertainty about the results.
When the organisation and its reality are substantially complex , such as with a network of independent organizations‌
Conventional thinking‌
Time
‌ clashes with relationships of cause and effect that are unknown
Time Ricardo.Wilson-Grau@inter.nl.net
Social change can be… • Complex:
• Unstable:
involve a confluence of actors and factors
independent of project duration
• Non-linear:
• Two-way:
unexpected, emergent, discontinuous
intervention may change
• Beyond control:
but subject to influence
• Incremental, cumulative:
watersheds & tipping points
“Being attentive along the journey is as important as, and critical to, ariving at the destination � Michael Quinn Patton
Brief definition of OM • A participatory method for planning, monitoring and evaluation • Focused on changes in behaviour of those with whom the project or program works • Oriented towards social & organizational learning
Source: A guide for project M&E: IFAD
OM’s answer
Start from observable behaviour change
Embrace different perspectives
Recognise that all interventions have limited influence
Support people to build their own well-being
Enable interventions to adapt as they engage
Apply a systems understanding
Source: Terry Smutylo / OM Lab 2012
Key concepts in Outcome Mapping
Three key concepts in OM: 1. Sphere of influence 2. Boundary Partners 3. Outcomes understood as changes in behaviour
There is a limit to our influence
Project
Sphere of control
People the project works with/through
Sphere of influence
People who benefit from the project
Sphere of concern
There is a limit to our influence
Inputs, activities, outputs
Sphere of control
Outcomes: Changes in behavior
Sphere of influence
Impact: Changes in state
Sphere of concern
People and families know where to go to have their needs met and are happy with their care
Providers learn About each other And begin to identify complex patients
Complex clients who are heavy users of the health system need coordinated care and special services to be kept well and Out of hospitals
HL partners build vision with clear roles and responsibilities are negotiated
Data sharing systems developed
Coordinated Care plans are designed
People are healthier
Complex clients are utilzing support services and programs
Complex clients and their families are engaged in care planning
Cost of delivering health care is lower without reducing the quality
Source: Terry Smutylo
Focus of Outcome Mapping
Inputs
Activities
Outputs
Outcomes
Outcome Mapping
Impacts
Where is the map? ✓OM is a guide to the journey we take with our partners. We co-create the map.
✓It focuses on the intention and what happens along the way ✓The map is not the territory, it shows the route taken ✓“The only real voyage of discovery exists, not in seeing new landscapes, but in having new eyes” (Marcel Proust)
improved human, social, & environmental wellbeing
I have a dream!
Martin Luther King, Jr. August 28, 1963
A vision statement.. •
Guides • Motivates and inspires • Is an ‘accountability-free zone’
Vision facilitation question Imagine that, 5-10 years from now, the program has been extremely successful. Things have improved beyond your most ambitious dreams.
• What changes have occurred? • What (& how) are your intended beneficiaries doing? • What are your partners doing? • Describe the better world you are seeking.
Step 2: Mission
The mission is that “bite� of the vision statement on which the program is going to focus.
Summary
Vision
Mission
✓ About the future
✓ Feasible
✓ Observable ✓ Idealistic
✓ Identifies activities and relationships
✓ Not about the program
✓ About the program
Step 3: Boundary Partners
Boundary Partners are... Those individuals, groups, & organizations with whom a program: • interacts directly to effect change • can anticipate some opportunities for influence • engages in mutual learning
Who are your boundary partners?
Programme
Beneficiaries
Stakeholders
Boundary Partners
Families
Specialists
Other service providers
Primary Care HL Leads
Community Service Partners
PCP
PCP
PCP
LHINS MOHLTC
example‌
Strategic partners • Selected for their potential to contribute to the mission • A person or group with whom the program works directly to achieve the mission, without necessarily wanting to change the partner’s behaviour as part of the mission • E.g. Funding agency, contracted service providers, Other Health Links, media agencies
Step 5: Progress Markers
Progress Markers Love to see (Deep transformation)
Like to see (Active engagement)
Expect to see (Early positive responses)
Love to see Like to see Like to see
Like to see
Like to see
Like to see
Expect to see
Expect to see
Like to see
Expect to see
Why Graduated Progress Markers?
• Articulate the complexity of the change process
• Allow negotiation of expectations between the program and its partners • Permit early assessment of progress
• Encourage the program to seek the most profound transformation possible • Help identify mid-course improvements
How can we measure: Greater awareness…
Empowered women…
Community ownership… Reduced conflict… Increased collaboration… Governmental commitment… Gender sensitivity… Equal access… Budgetary transparency… Active participation… Poverty alleviation… Strengthened capacity…
?
Facilitation questions • How can the programme know the boundary partner is moving toward the outcome? • What would they be doing?
• What milestones would be reached as the boundary partner moves towards their intended role in contributing to the vision?
Progress Marker Checklist Each Progress Marker: Describes a changed behaviour by the boundary partner Can be monitored & observed As a set, Progress Markers: Are graduated from preliminary to more profound changes in behaviour Describe the change process of a single boundary partner
Outcome Engineering • • • • •
Level 1: Knowing that there is a journey to take (P1) Level 2: Taking the first Steps (P1 level) Level 3: Investing your own resources (all) Level 4: Overcoming resistance to the change (all) Level 5: Identifying with the journey by joining with others with a similar approach (P2 level) • Level 6: Leaving a legacy (now an expert for others) (P3 level).
A PM Framework • P1 Preparation for the Journey: building Knowledge and Capacity • P2 The owned journey begins: building support, collaboration and networks • P3 The owned journey proceeds: sustained continuous actions. Institutionalization, Policies and/or Culture Change
32 sets of progress markers
P1: Knowledge acquisition processes and practices • ...attending forums where (the project) elaborates about the technology • ...raising questions and issues that (the Project) will address to encourage (the BP’s) uptake of the technology • ...seek out additional information on water and watershed issues from external sources • ...requesting position papers from the relevant departments to solicit input into decisions
P2: Getting involved, build support & enroll others • ...brokering or developing partnerships with other agencies to take local action • ...establishing mechanisms to share and review work programmes across departments, especially on research projects • ...establishing and expanding the membership base of the national organization in Indonesia • ...organize ‘popular education’ to increase critical thinking of their members
P3: Owned journey continuous and sustained
• ...contribute to the improvement of the methodology internationally to continually make it even more effective • ...generate their own funds and re-invest in (related) community projects • ...developing and putting in place a communication policy guiding how information is shared within the organization
Change is continuous
Outcome Challenge • Describes behaviour of a single boundary partner
• Sets out the ideal actions, relationships activities
• Describes the boundary partner’s contribution to the vision
Facilitation questions • Ideally, in order to contribute to the vision, what would the boundary partner be doing? • With whom would they be interacting?
• How could this boundary partner contribute maximally to the vision?
Step 6: Strategy Maps
Strategy Maps For each Boundary partner: • What will the Project do to SUPPORT desired changes in the Partner towards the Outcome Challenge? • Diverse range (an array) of ACTIVITIES:
- Aimed at Partner - Aimed at Partner’s environment
- Cause, Persuade, Support
Strategy Map causal
Partner
Environment
persuasive supportive
Facilitation Questions causal
Partner
what will be done to produce immediate outputs?
Environment
what will be done to alter the physical or policy environment?
persuasive supportive what will be done to build capacity?
how will sustained support, guidance or mentoring be provided?
how will the what networks or media or relationships will publications be be established or used? utilized?
Step 7: Organisational Practices
Organizational Practices
How does your team or organization stay relevant, viable and effective?
8 practices 1. Prospecting for new ideas, opportunities, and resources 2. Seeking feedback from key informants
3. Obtaining the support of your next highest power 4. Assessing and (re)designing products, services, systems, and procedures
8 practices 5. Checking up on those already served to add value 6. Sharing your best wisdom with the world
7. Experimenting to remain innovative 8. Engaging in organizational reflection
High expectations of planning monitoring and evaluation Address a growing call to show results
Satisfying multiple accountability needs
To show results that are difficult to measure
Stimulating internal learning processes
Learning about unexpected results 67
Extra challenging when dealing with processes of complex change? 1) Multiple actors, different perspectives and different forms of interaction 2) Multiple accountability needs 3) Unpredictable and non-linear link between cause and effect, therefore important to learn quickly what works and what doesn’t. 4) When dealing with complex change you need to be able to change as well.
Relationships
Learning
Dealing with Complex Change
Adaptive Capacity
Accountability 69
69
69
Recent Research on OM 1. The OM approach helps to clarify relationships, roles and expectations
2. The OM approach helps to learn about the progress towards the development objectives
“How can OM
contribute to dealing with complex processes of social change?�
4. The OM approach helps to strengthen upward, downward and horizontal accountability needs
20-6-2013 70
3. The OM approach helps to strengthen the internal adaptive capacity
Using OM has helped to strenghten dialogue among programme stakeholders (N=43) Strongly agree/agree
36 5 neither agree nor disagree
0 2 disagree/ strongly disagree
Nr. respondents
Not applicable / no answer
Adaptive capacity How has OM helped to strenghten elements of adaptive capacity (N=43) More time set aside for reflection about the monitoring data learning about the external context
OM contributing to changes in the internal practices of the programme increased understanding aobut how programme contributes to its effects
34
31
3 42
6 24
Strongly agree/disagree neither agree nor diagree disagree/strongly disagree
29
9 05 Not applicable / no answer
32
6 14
Nr. of respondents
20-6-2013 72
Challenges: • Time & resources • Stimulating informal learning spaces
• Capacity to facilitate reflection
OM and multiple actors and relationships (N=43)
OM has helped stakeholders to clarify their expectations
0 1
39
3
strongly agree / agree
OM has helped programme stakeholders to clarify their responsibilities
35
6
1 1
neither agree nor disagree
disagree / strongly disagree
OM has helped to strengthen trustful relationships
0
11
28
4 Nr. of respondents
73
Not applicable / No answer
Monitoring unexpected effects Has OM helped stakeholders to learn about unexpected programme effects? (N=43) At output level
At outcome level
At impact level
23
Strongly agree / agree
11 1 8 29 9
Neither agree nor disagree Disagree / strongly disagree
1 4 14 14
Not applicable / no answer
1 14 Nr. of respondents
74
E.g.: Sustainable Agriculture Programme Pacific Islands and Territories One of the love to see progress markers for the targeted villages read as follows: “sharing lessons and experiences with the other villages�. It was an unexpected surprise for the programme team to learn during the monitoring process that it was not the targeted village communities that took the initiative to share experiences but instead the surrounding villages invited them to come and share new technologies for improving their agriculture production. This happened without any support from the programme.
Case Example: McConnell Foundation Food Security Project