Developmental Evaluation:
Applying Complexity Concepts to Enhance Innovation and Use Michael Quinn Patton
Michael Quinn Patton SIG October 4,2010
1
In the beginning‌
Michael Quinn Patton SIG October 4,2010
2
Innovation and Evaluation Systems thinking & complexity science as frameworks for conceptualizing interventions: Developmental Evaluation Michael Quinn Patton SIG October 4,2010
3
5 types of DE 1. Ongoing Development Development vs. Improvement
Michael Quinn Patton SIG October 4,2010
4
Blandin Community Leadership Program
Michael Quinn Patton SIG October 4,2010
5
Beyond Formative and Summative to
Developmental Evaluation as an option in the repertoire of approaches
Michael Quinn Patton SIG October 4,2010
6
Mrs. McCave and her 23 Daves
Michael Quinn Patton SIG October 4,2010
7
Challenge: Matching the evaluation process and design to the nature of the situation:
Contingency-based Evaluation Michael Quinn Patton SIG October 4,2010
8
Conditions that challenge traditional model-testing evaluation
• • • • • •
High innovation Development High uncertainty Dynamic Emergent Systems Change Michael Quinn Patton SIG October 4,2010
Adaptive Management and Developmental Evaluation 9
First DE Type Ongoing development in adapting a project, program, strategy, policy, or other innovative initiative to new conditions in complex dynamic systems.
Michael Quinn Patton SIG October 4,2010
10
Mintzberg on Strategy Unrealized Strategy Intended Strategy Deliberate Strategy Realized Strategy Emergent Strategy Michael Quinn Patton SIG October 4,2010
11
Uncertainty and Emergence “No battle plan ever survives contact with the enemy.” Field Marshall Helmuth Carl Bernard von Moltke
“Everyone has a plan…until he gets hit.” Former World Heavyweight boxing champion, Mike Tyson
Michael Quinn Patton SIG October 4,2010
12
5 Types of DE 2. Pre-formative development of a potentially scalable innovation: • Developing an innovation to the point where
it is ready for traditional formative and summative evaluation • Pre-formative developmental evaluation works with emerging ideas and visionary hopes in a period of exploration to shape them into a potential model that is a more fully conceptualized, potentially scalable intervention. Michael Quinn Patton SIG October 4,2010
13
PANARCHY MODEL
Michael Quinn Patton SIG October 4,2010
14
DEVEOPMENTAL EVALUATION
SUMMATIVE
Stored
FORMATIVE Phases of Technological & Social Innovation
Michael Quinn Patton SIG October 4,2010
HARVESTING LESSONS 15
Polio Example Bill & Melinda Gates Foundation: $1 billion on polio eradication: Vertical, focused strategy "There's no way to sugarcoat the last 12 months," Bruce Aylward, a WHO official, told Mr. Gates in June -- the virus was rippling through countries believed to have stopped the disease. Mr. Gates asked: "So, what do we do next?“ New strategy: Disease-specific wars can succeed only if they also strengthen the overall health system in poor countries.
http://online.wsj.com/article/SB10001424052702303348504575184093239615022.html?mod=WSJ_hps
Michael Quinn Patton SIG October 4,2010
16
5 Types of DE 3. Adapting effective general principles to a new context as ideas and innovations are taken from elsewhere and developed within a new setting, the work of developmental evaluation in the dynamic middle between top-down and bottom-up forces of change.
Michael Quinn Patton SIG October 4,2010
17
Fundamental Issue: How the World Is Changed Top-down dissemination of “proven models� versus Bottoms-up adaptive management Michael Quinn Patton SIG October 4,2010
18
Prescriptive Models vs. Adaptive Principles Identifying effective principles for adaptive management (bottoms-up approach) versus Identifying and disseminating “proven� models (top down approach) Michael Quinn Patton SIG October 4,2010
19
Top-down dissemination of “proven models�
Developmental Evaluation: Navigating the murky, complex, dynamic MIDDLE
Bottom-up adaptive management Michael Quinn Patton SIG October 4,2010
20
Personal Factor ď śDamiano example, Jean Gornick
Michael Quinn Patton SIG October 4,2010
21
5 Types of DE 4. Major systems change and cross-scale developmental evaluation • Providing feedback about how major systems change is unfolding • Evidence of emergent tipping points • System inter-relationships as the “unit of analysis” • Evaluating strategy Michael Quinn Patton SIG October 4,2010
22
Systems • Parts are interdependent such a change in one part changes all parts • The whole is greater than the sum of the parts • Focus on interconnected relationships • Systems are made up of sub-systems and function within larger systems Michael Quinn Patton SIG October 4,2010
23
Michael Quinn Patton SIG October 4,2010
24
Understanding the Elephant from a Systems Perspective
Michael Quinn Patton SIG October 4,2010
25
Michael Quinn Patton SIG October 4,2010
26
Michael Quinn Patton SIG October 4,2010
27
Michael Quinn Patton SIG October 4,2010
28
Michael Quinn Patton SIG October 4,2010
29
Michael Quinn Patton SIG October 4,2010
30
The relationship between what goes in and what comes out What conceptual framework informs evaluation? Michael Quinn Patton SIG October 4,2010
31
Teen Pregnancy Program Example
Michael Quinn Patton SIG October 4,2010
32
Logic Model for Pregnant Teens Program 1. Program reaches out to pregnant teens
2. Pregnant teens enter and attend the program (participation) 3. Teens learn prenatal nutrition and self-care (increased knowledge) 4. Teens develop commitment to take care of themselves and their babies (attitude change) 5. Teens adopt healthy behaviors: no smoking, no drinking, attend prenatal clinic, eat properly (behavior change) 6. Teens have healthy babies (desired outcome)
Michael Quinn Patton SIG October 4,2010
33
Systems web showing possible influence linkages to a pregnant teenager Prenatal program staff Teachers/ other adults
Child's father & peers
Her parents & other family members
Young pregnant woman's attitudes & behaviors
Her peer group
Michael Quinn Patton SIG October 4,2010
34
Program systems web showing possible institutional influences affecting pregnant teenagers: Other Systems -- welfare -- legal -- nutrition programs -- transportation -- child protection -- media messages Context factors -- politics -- economic incentives -- social norms -- culture -- music
Prenatal program
SCHOOL SYSTEM
Prenatal Clinic and Hospital Outreach Young pregnant women's attitudes & behaviors
Church
Michael Quinn Patton SIG October 4,2010
Youth Culture
Other community-based youth programs
35
5 Types of DE 5. Developing a rapid response in the face of a sudden major change or a crisis, like a natural disaster or financial melt-down: • Exploring real time solutions and generating innovative and helpful interventions for those in need. • Dealing with high uncertainty, turbulence, turmoil, high stakes, and often conflict. Michael Quinn Patton SIG October 4,2010
36
Refugee Camps
Michael Quinn Patton SIG October 4,2010
37
Michael Quinn Patton SIG October 4,2010
38
General oe9S6Sgf result ACfU3U1E PP1 Search BooksR
Loading...
S h a k e H a n d s w i P t r A h e b v o t i u h e t e w t h D t i e h s v i b i s o l b o o k B o y k R o m ĂŠ o
This is a preview. The total pages displayed will be limited. L
D a ll a i r e
Michael Quinn Patton SIG October 4,2010
39
Michael Quinn Patton SIG October 4,2010
Photo by Lynsey Gornick
40
Seeing Through A Complexity Lens
“You don't see something until you have the right metaphor Michael to let you perceive it”. Thomas Kuhn Quinn Patton SIG October 4,2010
41
Getting to Maybe: How the World Is Changed? Frances Westley, Brenda Zimmerman, Michael Q. Patton Random House Canada,2006 Michael Quinn Patton SIG October 4,2010
42
Complex Nonlinear Dynamics • Nonlinear: Small actions can have large reactions. “The Butterfly Wings Metaphor” • Emergent: Self-organizing, Attractors • Dynamical: Interactions within, between, and among subsystems and parts within systems can volatile, changing • Getting to Maybe: Uncertain, unpredictable, uncontrollable • Co-evolution: Integrating innovation and evaluation Michael Quinn Patton SIG October 4,2010
43
Three Cups of Tea Baltistani proverb:
First cup you share, you are a stranger. Second cup, you are an honored guest. Third cup, you are in relationship. Michael Quinn Patton SIG October 4,2010
44
“A Leader's Framework for Decision Making� by David J. Snowden and Mary E. Boone, Harvard Business Review, November, 2007: Wise executives tailor their approach to fit the complexity of the circumstances they face.
Michael Quinn Patton SIG October 4,2010
45
Wise evaluators tailor their approach to fit the complexity of the circumstances they face.
Michael Quinn Patton SIG October 4,2010
46
5 Types of DE 1. Ongoing development and adaptation 2. Preformative evaluation to support exploration and innovation 3. Supporting local adaptation of general principles to navigate top-down and bottomup forces for change 4. Evaluating major systems change 5. Evaluating in turbulent, disaster situations Michael Quinn Patton SIG October 4,2010
47
Challenge: Matching the evaluation process and design to the nature of the situation:
Contingency-based Evaluation Michael Quinn Patton SIG October 4,2010
48
Michael Quinn Patton SIG October 4,2010
49
References Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use. Guilford Press, June 2010. Getting to Maybe: How the World Is Changed? Frances Westley, Brenda Zimmerman, Michael Q. Patton, Random House Canada, 2006 Utilization-Focused Evaluation, 4th ed., Michael Quinn Patton, Sage, 2008. Michael Quinn Patton SIG October 4,2010
50