Advancing Evaluation Practices in the Field of Early Childhood Development
1 3 6 9 16 December 2015 Evaluation Capacity Network University of Alberta
Introduction:
SETTING THE STAGE Evaluation Capacity Building:
WHY NOW?
Evaluation Capacity Building:
WHERE ARE WE CURRENTLY? Evaluation Capacity Building:
WHAT DO WE KNOW? Looking Forward:
WHAT ARE THE POSSIBILITIES?
Introduction: SETTING THE STAGE As you read, we invite you to draw on your own experiences with evaluation.
Advancing Evaluation Practices in the Field of Early Childhood Development is intended to generate ideas and discussion about the role of evaluation in the field of early childhood development (ECD). We invite you to engage with the content and the questions we have posed in a way that is personal to your own experiences with evaluation. This stimulus paper provides a foundation for celebrating previous evaluative efforts, challenging current practices, and forging new pathways for evaluation in ECD.
An overview of this stimulus paper.
We begin the paper by outlining the need for, and potential of, evaluation in ECD. Next, we present the need to advance current evaluation practices by building capacity to engage in evaluation. We provide an overview of evaluation strategies in ECD in Alberta and beyond, and turn to the academic literature for a conceptual understanding about evaluation capacity building more broadly. Finally, we conclude with a list of possibilities for evaluation capacity building to encourage discussion and the flow of creative ideas as we come together to collectively advance our evaluation practices.
Advancing Evaluation Practices
1
Evaluation in the field of ECD offers many opportunities and challenges.
Early childhood is a time of rapid development, learning, and opportunity for children, as well as for their parents, administrators, front-line staff, policymakers, and funders. We all want to be confident that the time and energy we devote to raising young children is enhancing their development. We want to know whether programs are making a meaningful difference in their lives. Using evaluation, we can begin to respond to these pressing questions. However, evaluation comes with many opportunities and challenges that are experienced by organizations in different ways.
The Evaluation Capacity Network aims to build evaluation capacity & promote learning in the ECD field.
The Evaluation Capacity Network (the Network) formed out of an awareness of these opportunities and challenges, with the goal of advancing evaluation practices in the ECD field. Supporting children in the early years requires expertise and input across multiple sectors and disciplines, many of which have differing perspectives on ECD and evaluation. The Network connects this diverse community to enhance evaluation capacity and to promote learning through the shared practices, perspectives, and approaches of diverse ECD stakeholders. We need to engage in open dialogue to align resources, come to a common understanding of the impacts we are having on ECD, and collectively reflect on how to make informed decisions to best support young children. As the first step, this paper provides an information base to begin this dialogue.
2
Evaluation Capacity Building: WHY NOW? Healthy experiences in the early years create a foundation for a lifetime of positive outcomes.
It is an opportune time for enhancing our evaluation capacity in ECD. Decades of research show that the early years are a time of both promise and risk, setting the foundation for healthy functioning throughout the lifespan.[1] The environments that children encounter from the prenatal period onward have a cumulative effect on all aspects of their development.[2] In order to learn effectively and grow into successful adults, children need supportive and responsive nurturing in stimulating environments. Negative experiences during early childhood can put children at risk for many challenges, which in turn have significant social and economic costs.[2]
Strong, well–rounded ECD programming supports long–term healthy outcomes, and can reduce expensive interventions in later years.
Early interventions can help lessen the impacts of negative experiences. Providing children with early supports prevents negative outcomes and promotes long–term positive development in health and cognitive, social, behavioural, and academic domains.[3, 4] Organizations and funders have recognized the cost–effectiveness of earlier interventions over those implemented at later ages[5], and are increasingly directing their efforts toward ECD.
Advancing Evaluation Practices
3
There are multiple reasons for strengthening evaluation: strategic learning, accountability to funders, and building the field of ECD.
As investment in the field of ECD grows, so too does the need for organizations to be able to participate in multiple types of evaluation. Evaluation can be used for: • Strategic learning, to inform practices and enhance programming at the organizational level to better meet the needs of children and families. • Accountability to funders, to show how resources invested in ECD programming are used, demonstrate quality of service, and track the satisfaction and outcomes for children and families. • Field–building, to provide an evidence base for the field of early childhood development, and contribute to our broader knowledge of needs, appropriate programs, and services. It is critical that ECD organizations are able to participate effectively in each of these forms of evaluation.
There is a need for evaluation alignment and capacity building at the individual, organizational, and system levels.
4
A review of the current state of program evaluation practice in Canada reported that there are not enough evaluators and evaluation resources to meet the increasing demand for program evaluations.[6] Organizations often lack the capacity, knowledge, and skills to collect the evidence they need to demonstrate their impacts and justify continued support.[7] In addition, different language and methods are used to describe and measure child outcomes across organizations, sectors, disciplines, and systems, highlighting the need for evaluation alignment and capacity building at the individual, organizational, and system levels.
At the beginning of this section, we posed the question, Why Now? The current focus on evaluation capacity building is important because we now know that the early years are critical in setting the stage for long–term health and development, and we know that ECD programs are increasingly required to provide evidence that their programs are working. New and innovative evaluation approaches are therefore necessary to capitalize and build on the existing capacity in the ECD field, and to fully support the needs of children and families.
Discussion Questions: What is your experience with evaluation? How is evaluation generally viewed in your organization?
Advancing Evaluation Practices
5
Evaluation Capacity Building: WHERE ARE WE CURRENTLY? Consistent ways of measuring, reporting, and comparing the effectiveness of ECD programs are essential.
In 2003, a report on early childhood education and care in Canada showed that ECD programs collect considerable amounts of data.[8] However, different provinces and territories often collect different information at different time intervals, using different methods. It is not often possible to compare data nationally, provincially, or even regionally. We therefore lack a solid understanding of the need for early childhood programs, the effectiveness of these programs, and the children and families they serve. Consistent ways of measuring, reporting, and comparing the effectiveness of ECD programs are essential.[9]
CMEC recently developed a common vision for early learning in the Early Learning and Development Framework.
At a national level, the Council of Ministers of Education, Canada (CMEC) has developed an Early Learning and Development Framework. This national framework presents a vision for early learning that could be applied Canada–wide to foster continuity across jurisdictions and ECD settings. The framework offers six principles to promote a common understanding of a continuum of learning and development, as well as shared values regarding what is most important in the early years.[10] (www.cmec.ca)
6
Provincial frameworks can support measurement of common outcomes at regional, provincial, and national levels.
There are also a number of frameworks in use at the provincial level. For example, British Columbia’s Early Childhood Development Evaluation Project brought together community and government stakeholders to address the lack of coordinated ECD evaluation. The project identified four long–term outcomes, as well as shared measurement tools and technology, which have been adopted across the province.[11] (www.successby6bc.ca) In 2013, the Government of the Northwest Territories departments of Health and Social Services (HSS) and Education, Culture and Employment (ECE) released Right from the Start, an updated framework for early childhood development informed by an intensive public engagement process. The framework includes seven commitments, each supported by specific areas for action, deliverables, and target objectives.[12] (www.rightfromthestart.ca) Healthy Child Manitoba has also released an ECD framework that includes guiding principles, ECD building blocks, and recommendations for measuring success and reporting on progress.[13) (www.gov.mb.ca) Common across each of these provincial frameworks is a commitment to empower families, strengthen communities, improve access to programs and services, and foster safe, secure, and supportive environments during the early years. These frameworks can support the measurement of common outcomes at regional, provincial, and national levels to inform decisions around policy, funding, and programming.
The Play, Participation, and Possibilities framework for early learning draws on stakeholder consultations to guide decision–making and align policy in the province, alongside the Alberta Social Policy Framework and Together We Raise Tomorrow initiative.
In Alberta, there have been several ongoing efforts to guide decision-making and align policies and practices. For example, Play, Participation, and Possibilities is a provincial early learning and child care curriculum framework that was produced through a partnership between the Government of Alberta, faculty at MacEwan Early Learning and Child Care, and Mount Royal Department of Child and Youth Studies program, via consultations with stakeholders across Alberta.[14] (www.childcareframework.com) The Government of Alberta also engaged in consultations to create a vision for social policy in the province. The Social Policy Framework outlines eight priority initiatives, one of which is early childhood development.[15] These conversations also led to the
Advancing Evaluation Practices
7
Together We Raise Tomorrow initaitive to support the well–being, safety, security, education, and health of children in Alberta, and included a preliminary outcomes measurement framework.[16) (www.humanservices.alberta.ca/department/previousconsultations.html)
“Are we making a difference with our ECD programs?” “Yes, I think we are.” “But how do we know that we are making a difference?” Overall, everyone working in different sectors across the ECD field aims to make a difference. The question remains: how do we know we are doing the best we can for children and families in our province? By setting out shared language and common outcomes, these frameworks are a useful starting point. However, organizations still need the capacity to conduct quality evaluation to contribute to these frameworks. Community agencies, funders, and policymakers need to identify collectively the capacity that they each require to contribute to healthy ECD. This is where evaluation capacity building becomes important. In response to the question, Where Are We Currently?, there are frameworks available to help us understand how we are making a difference for ECD on national, provincial, and territorial levels. These frameworks identify outcomes that will be essential to consider when examining how we can build evaluation capacity in ECD.
Discussion Questions: What ECD evaluation framework—if any— have you found most useful and why? How can these frameworks inform evaluation in the field of ECD?
8
Evaluation Capacity Building: WHAT DO WE KNOW? There are multiple definitions of evaluation capacity building. For the purpose of this discussion, we have chosen to use Labin’s[17), which describes evaluation capacity building as:
“an intentional process to increase individual motivation, knowledge, and skills, and to enhance a group or organization’s ability to conduct or use evaluation.” In this section, we present some research findings related to evaluation capacity building, and what they might mean for an organization. However, academic literature does not offer a complete picture. We also need to know more from you as an individual or organization working directly in the ECD field.
Advancing Evaluation Practices
9
“I got an email from our funder about annual reporting.” “What do they want us to report on this year?” “Seems they have a new framework for us to use.” “What kind of information are they looking for?” “I’ll have to take a closer look. We may need to collect some new information.” “Who’s going to take that on? My plate is already full.” “We may need to bring someone in to help us, depending on how complex it is.” “Well, once it’s done, we can get back to business.”
Mutually–Beneficial Evaluation Research tells us: Using hands–on strategies promotes the positive benefits of evaluation and improves buy–in.
The above conversation demonstrates how demands for evaluation can, at times, lack relevance in our work, and place additional stress on staff. Research shows that organizations need to directly experience the benefits of evaluation before staff will fully buy into evaluation as a tool for program improvement.[18) This suggests that efforts to build evaluation capacity should be highly participatory, with opportunities for hands–on, practical learning, and reflection throughout.
Discussion Questions: Can you think of a positive experience your organization has had with evaluation? What made it positive?
10
A Strong Evaluation Learning Culture Research tells us: Fostering a strong evaluative learning culture leads to effective evaluation capacity building.
Having an evaluative learning culture is a fundamental component of effective evaluation capacity building in an organization.[18) In organizations with a strong learning culture, staff freely share information, trust one another and feel comfortable asking questions and taking risks. Leaders encourage collaboration, seek information from others when making decisions, are open to feedback, and reward employees for learning, taking part in professional development, and engaging in evaluation.[19) In this environment, evaluation can be a system for organizational learning[18) that staff use to make informed decisions that can enhance programming.
Discussion Questions: What characteristics does your organization have that support an evaluation learning culture?
Advancing Evaluation Practices
11
Motivations, Assumptions, and Expectations Research tells us: Participants’ motivations, assumptions, and expectations should inform evaluation capacity building activities.
In line with a hands–on participatory approach, participants’ motivations, assumptions, and expectations should inform the design of evaluation capacity building activities.[19) Understanding motivation helps determine if timing is appropriate, and provides insight into which staff members should participate and which teaching and learning strategies are likely to succeed. It is also important to discuss participants’ assumptions and expectations for evaluation in their organization. For example, some staff members may not believe that evaluation leads to effective decision–making. Sharing these assumptions ensures that all participants are starting evaluation capacity building activities on the same page.
Discussion Question: What might be the motivation for you or your organization to engage in evaluation?
12
Current Capacity, Future Goals, and Specific Needs Research tells us: Understanding current capacity and future goals for your organization shapes an effective approach.
Assessing an organization’s existing capacity levels before implementing capacity building initiatives will help shape an effective approach. Goals for capacity building should also be identified at the beginning of initiatives. Some organizations may start with the objective to address staff beliefs about evaluation, or to foster a stronger organizational learning culture. Other organizations may prioritize building evaluation skills and knowledge. Regardless of the specific objectives, evaluation capacity building initiatives are more likely to succeed if these objectives are clearly defined and understood.
Discussion Question: What might be some objectives for building evaluation capacity in your organization?
Research tells us: Understanding the specific need(s) for evaluation in your organization will streamline evaluation efforts.
When implementing evaluation capacity building initiatives, it is important to clarify between the need for building capacity to do evaluation, the need for building capacity to use evaluation results, or the need for both.[18) Capacity to do evaluation is built primarily from formal and informal training to build knowledge, skills, and abilities and apply them to the workplace setting. This type of capacity building should happen on both an individual and an organizational level. On an individual level, this includes developing the abilities to plan evaluations, develop and use evaluation instruments, collect data, and analyze and interpret data, as well as a number of “soft skills� that are developed through practical experience, such as conflict resolution, collaboration, and cooperative communication. At an organizational level, evaluation capacity can be measured by the frequency and quality of evaluation efforts.
Advancing Evaluation Practices
13
We can also consider individual and organizational capacity to use evaluation results in planned and intentional ways. For example, organizations can use evaluation results to make decisions on program continuation, revision, and improvement; to reaffirm program impacts with staff and stakeholders; and to more closely align with an organizational or funders’ mandate. Factors that determine whether evaluation results are used can include the extent to which evaluation results are timely, constructive, relevant, credible, and understandable to primary evaluation users. Building capacity for evaluation use involves developing an awareness of these conditions and how they shape how evaluation results are used.[18)
Discussion Questions: How is the information gained through evaluation used by your organization?
14
Sustaining Your Efforts Research tells us: Dedicating time and effort after a learning opportunity supports sustainable evaluation capacity building.
Evaluation capacity needs to be sustained in order to make a lasting difference.[19) Capacity building participants need support to transfer and use their learning once they have completed training.[17) Organizations can also develop evaluation– specific policies and procedures, such as including evaluation responsibilities in job descriptions and resource allocation documents. Other strategies to support long–term sustainability of evaluation capacity building efforts can include making an explicit commitment to integrate evaluation into decision– making, goal–setting, and resource allocation, as well as communicating and celebrating how evaluation findings are used in reports and communications.
Discussion Questions: How realistic are these suggested practices for building evaluation capacity in your organization?
In response to the question, What Do We Know?, we understand that there are a number of elements important to effective evaluation capacity building. According to research, we need to • • • •
employ hands–on strategies, foster a strong learning culture, consider motivations, assumptions, and expectations, understand current capacity, future goals, and specific needs, and • invest time and effort post–learning. It will be useful to consider these elements when we begin to explore the possibilities for building evaluation capacity in ECD.
Advancing Evaluation Practices
15
Looking Forward: WHAT ARE THE POSSIBILITIES? We need to explore ways to advance how we collect and examine information.
If we work together, we can generate solutions to the evaluation challenges we currently face in the ECD field. There are many possibilities for evaluation capacity building, and we want to know what would help you in your work. There are already evaluation capacity building efforts in place across Canada. For example, the University of Manitoba offers a Summer Institute in Program Evaluation (www. thesummerinstitute.ca). The Institute brings together practitioners and university students to exchange information and improve evaluation skills using lectures, case studies, and group work.[20) Another example is the First Steps First: A Community– Based Workbook for Evaluating Substance Abuse and Mental Health Programs in Saskatchewan, created by the Saskatchewan Team for Research and Evaluation of Addictions Treatment and Mental Health Services (www.addictionresearchchair.ca/ creating-knowledge/provincial/s-t-r-e-a-m/first-steps-first). This workbook provides information to program staff on the steps needed to prepare for an effective evaluation.[21) The Canadian Evaluation Society (CES) also offers a number of regional and national professional development tools. The CES Essential Skills Series provides an introduction to program evaluation skills and
16
is offered on demand. CES webinars cover areas of interest within each region, and the CES Annual Conference provides a forum to discuss major theoretical, philosophical, and practical evaluation issues.[22) (www.evaluationcanada.ca/professional-learning) There are many ways we can build evaluation capacity together.
When we consider the question, What Are the Possibilities?, it is evident that there are many potential options for evaluation capacity building strategies. Examples from across Canada can stimulate ideas for implementing strategies through the Evaluation Capacity Network. Ultimately, there is a need to generate solutions that will advance how we collect and examine information, to ensure that children in our communities are thriving. It is our hope that this stimulus paper provides a starting point to open dialogue and stimulate creative ideas among ECD stakeholders to better meet the evaluation capacity building needs across sectors and disciplines.
Discussion Question: What would help you to build evaluation capacity in your organization?
The time is right to come together and work toward the advancement of evaluation practices in the ECD field. Meaningful evaluation can enhance our collective ability to engage in reflective practices that best support the well–being of young children. Advancing Evaluation Practices provides an overview of the need for building evaluation capacity, and offers some examples for doing so. However, this paper is only a starting point. For the Evaluation Capacity Network to advance evaluation practices in ECD, we need to engage in collective discussion across the province to ensure steps taken are relevant to a range of stakeholders working across varied contexts.
Advancing Evaluation Practices
17
Through our collective ideas, we can forge new pathways for advancing evaluation practices in the field of ECD. In collaboration, we can ensure that the evaluation we engage in is meaningful to our work and helps us to know that we are truly making a difference. Consider the possibilities...
The examples below are some ideas for how evaluation capacity could be enhanced in the field of ECD. Consider these as a starting point as you ask yourself, What Are the Possibilities? Explore what would be useful for you and your organization to advance evaluation practices and better understand how you are making a difference for children and families.
events
courses tools dialogue
interacting together learning from experts mentoring
18
REFERENCES 1.
Shonkoff, J. P., & Levitt, P. (2010). Neuroscience and the future of early childhood policy: Moving from why to what and how. Neuron, 67(5), 689-691.
2.
Shonkoff, J. P., & Phillips, D. A. (2000). From neurons to neighbourhoods: The science of Early Childhood Development. Washington, DC: National Academy Press.
3.
Burger, K. (2010). How does early childhood care and education affect cognitive development? An international review of the effects of early interventions for children from different social backgrounds. Early Childhood Research Quarterly, 25, 140-165.
4.
Sylva, K., Melhuish, E. C., Sammons,P., Siraj-Blatchford, I., & Taggart, B. (2004). The effective provision of pre-school education (EPPE) project: Final report. London: DfES Institute of Education.
5.
Heckman, J. J. (2006). Skill formation and the economics of investing in disadvantaged children. Science, 312(5782), 1900-1902.
6.
Gauthier, B., Barrington, G. V., Bozzo, S. L., Chaytor, K., Dignard, A., Lahey, R., Malatest, R., McDavid, J. C., Mason, G., Mayne, J., Porteous, N. L., & Roy, S. (2010). The lay of the land: Evaluation practice in Canada in 2009. The Canadian Journal of Program Evaluation, 24(1), 1-49.
7.
Bakken, L. L., Nunez, J., & Couture, C. (2014). A course model for building evaluation capacity through a university-community partnership. American Journal of Evaluation, 35(4), 579-593.
8.
Doherty, G., Friendly, M., & Beach, J. (2003). OECD thematic review of early childhood education and care: Canadian background report. Ottawa: Government of Canada.
9.
Mustard, J. F., & Young, M. E. (2007). Measuring child development to leverage ECD policy and investment. In M. E. Young and L. M. Richardson (Eds.), Early child development from measurement to action: A priority for growth and equity, (253-291). Washington: The World Bank.
10. Council of Ministers of Education, Canada (CMEC). (2014). CMEC early learning and development framework. Canada: CMEC Early Childhood Learning and Development Working Group. Retrieved from www.cmec.ca 11. Munro, C. (2006). BC evaluation project report: Can we agree on common child outcomes for BC? Retrieved from www.successby6bc.ca 12. Government of the Northwest Territories. (2013). Right from the start. Retrieved from www.rightfromthestart.ca 13. Government of Manitoba. (2013). Starting early, starting strong: Manitoba’s early childhood development framework. Retrieved from www.gov.mb.ca
Advancing Evaluation Practices
19
14. Makovichuk, L., Hewes, J., Lirette, P., & Thomas, N. (2014). Play, participation, and possibilities: An early learning and child care curriculum framework for Alberta. Retrieved from www.childcareframework.com 15. Government of Alberta Human Services. (2013). Alberta’s Social Policy Framework. Referenced in www.humanservices.alberta.ca/department/previous-consultations.html 16. Alberta Government. (2013). Together we raise tomorrow: An Alberta approach to early childhood development. Referenced in www.humanservices.alberta.ca/department/ previous-consultations.html 17. Labin, S. N., Duffy, J. L., Meyers, D. C., Wandersman, A., & Lesesne, C. A. (2012). A research synthesis of the evaluation capacity building literature. American Journal of Evaluation, 33(3), 307-338. 18. Cousins, J. B., Goh, S. C., Elliott, C. J., & Bourgeois, I. (2014). Framing the capacity to do and use evaluation. In J. B. Cousins and I. Bourgeois (Eds.), Organizational capacity to do and use evaluation. New Directions for Evaluation, 141, 7-23. 19. Preskill, H., & Boyle,S. (2008). A multidisciplinary model of evaluation capacity building. American Journal of Evaluation, 29(4) 443-459. 20. University of Manitoba (2014). Summer institute in program evaluation. Retrieved from www.thesummerinstitute.ca 21. Addiction Research Chair (2012). First Steps First: A community-based workbook for evaluating substance abuse and mental health programs in Saskatchewan. Retrieved from www.addictionresearchchair.ca/creating-knowledge/provincial/s-t-r-e-a-m/firststeps-first 22. Canadian Evaluation Society (2015). Professional learning. Retrieved from www. evaluationcanada.ca/professional-learning
20
Advancing Evaluation Practices
21
December 2015 Evaluation Capacity Network University of Alberta
This research was supported by the Social Sciences and Humanities Research Council of Canada.