Program Evaluation Report by Stringbean Design

Canadian platform to increase usage of real-world evidence

REAL-WORLD PROGRESS, REAL-WORLD LEARNING

JANUARY 2009 â&#x20AC;&#x201C;JANUARY 2011 REPORT 1 2 3 4 5

About the CAPTURE Project

Letter from the Executive Director Business objectives

capacity building connections

Project milestones Capture progress and learnings: creating a high performing, Engaging builders, users, supporters, and clients

Platform development

Seeking feedback for continuous improvement

14 15

learning organization

Provide training, support and

Acknowlegments Appendix A: Consultation and communications

Appendix B: CAPTURE reports

VISION We have effective primary prevention of chronic disease in Canada because we learn from what we do.

MISSION The CAPTURE Project will collect, share and support the use of “real-world” evidence on what works, for whom and in what context, to improve primary prevention of chronic disease.

VALUES User driven Continuous learning Servant leadership Inter- dependence Sharing resources

The CAPTURE Project

Funded by

ABOUT THE CAPTURE PROJECT In communities across Canada, there are many programs aimed at helping people reduce their risk of developing chronic diseases such as cancer, heart disease and diabetes. These primary prevention programs focus on modifying the risk factors that lead to chronic disease, such as obesity, smoking, inactivity and unhealthy eating. Practitioners and program managers want to offer the most effective primary prevention programs for their community. They need access to a range of realworld evidence – experiences, approaches and insights shared by others doing the same type of work – to help them plan, implement, evaluate and refine their own programs. They also seek a forum for disseminating their own experiences and learnings with others in different communities. This “real-world” evidence is as important as academic research for helping communities plan and manage effective primary prevention programs. However, the supports to help make this happen are currently lacking. The Canadian Platform to Increase Usage of Real-World Evidence – or the CAPTURE Project – will help chronic disease prevention practitioners and program managers learn from what they do, and share that knowledge. Funded by the Canadian Partnership Against Cancer, CAPTURE is building a web-based learning and shared measurement platform to support the increased use of evidence within programs and across the health promotion community. CAPTURE will offer tools and resources for planning, managing and evaluating interventions. It will serve as a centralized hub for information, education and networking. The platform will also support the development of communities of practice so practitioners can more easily learn from each other.

LETTER FROM THE EXECUTIVE DIRECTOR During the time I was Scientific Director of the Institute of Nutrition, Metabolism and Diabetes for the Canadian Institutes of Health Research, I heard a number of stories that stuck in my head about the mismatch between what researchers are doing and what people in the real-world of chronic disease prevention programs need. One story relates to the effort to synthesize the available evidence on school-based nutrition programs. The project team found a number of helpful studies, but they couldn’t combine and synthesize them in order to make recommendations for future action. No one was using the same framework or measurement tools, so there was no way to combine or share the data. I also have a vivid memory of the many times I found myself asking, “What is the difference between research and evaluation?” I got countless different answers over the years, though I don’t know that any of them was adequate. In the end, I came to understand that if we want more useful and available “practicebased evidence,” the paradigms of research and evaluation would need to come closer together. Changing paradigms requires a big vision, a willingness to take risks, money and a bit of time! We had the big vision and when The CAPTURE Project was started 2 years ago, the Canadian Partnership Against Cancer was looking for big change. They needed to shift their investments from the many small projects they had inherited to include some larger projects with the potential for significant impact. We knew from the outset that it was not nearly enough just to develop an IT solution and throw it out there. In the real world, it’s simply wrongheaded to believe, “If we build it, they will come.” Instead, we have sought to engage our many stakeholders to guide us and work with us on building the tools and support that will help them learn from and improve their practices – and to share that knowledge so others can also learn and improve. While absolutely essential to the usefulness and sustainability of the platform, this real-world approach takes considerable resources and time. After two years of foundational work, I think we can be proud of what we have accomplished so far. In keeping with CAPTURE’s philosophy that learning in the context of the real world is the key to improvement, this report speaks not only to our activities over the past two years, but equally to our insights and learnings. We are grateful to CPAC for providing us with the resources and time required to plan and implement this type of high risk, high reward project. It’s a unique opportunity to build a collaborative venture that promises to result in real system change for public health.

Diane Finegood 2

BUSINESS OBJECTIVES The CAPTURE Project seeks to improve

Objective #1 | Creating a high-performing,

the primary prevention landscape in Canada

learning organization

by making it easier to collect and share a

Having the right people in place to lead

range of practice-based evidence. Achieving

the Project and serve our stakeholders,

this vision requires more than just building

and modeling an organization that improves

a web-based repository for information.

through shared learning;

It also requires work at the system level to change the way practice-based evidence is

Objective #2 | Engaging builders, users,

viewed and used.

supporters, and clients Consulting, building relationships and forming

The CAPTURE Project is focused on

partnerships to help us plan and build the

achieving five key business objectives.

platform, find champions for our work, and

Each is considered fundamental to the

establish a base of users and clients;

development of a useful, sustainable platform that facilitates the use of real-

Objective #3 | Developing the platform

world evidence in public health practice.

Developing the platform â&#x20AC;&#x201C; Developing high-tech features, such as web-based tools for evaluation planning, data collection and analysis, and generating evaluation reports;

Objective #4 | Providing training, support and capacity building connections Developing high-touch supports (e.g. networking, mentorship and training) that help people gain the skills and capacity to use evidence-based decision making in their own programs, enhance knowledge exchange, and facilitate the formation of communities of practice;

Objective #5 | Seeking feedback in order to continuously improve Looking to our stakeholders to help us test, refine and evolve our offerings on an ongoing basis.

SPRING 2010

First staff hired

In-house Platform development team triples

WINTER 2011

The CAPTURE Project launches WINTER 2009

FALL 2008

PROJECT MILESTONES

11 staff 25 contractors 4 co-op students

Consultations and needs assessments

FALL 2009

SPRING 2009

1 volunteer

CAPTURE Project website launches First development partners engaged Platform development begins Work on privacy, security and data sharing begins Work on quality assurance protocols begins Gathering of user preferences begins

National needs assessment undertaken CAPTURE begins presenting at public health and evaluation conferences

First International Advisory Board (IAB) meeting

Additional funding from Public Health Agency of Canada to assess evaluation needs for communities working with Northern, Remote, and Aboriginal populations, and provide additional evaluation support

Second I AB meeting

First platform features unveiled to IAB

2008

2009

2010

WINTER 2011

FALL 2010

Co-sponsored an Evaluation for Learning forum

SPRING 2010

WINTER 2010

Work begins with CLASP, our first client

User testing begins for first set of platform features

2011

CAPTURE PROGRESS AND LEARNINGS: CREATING A HIGH PERFORMING, LEARNING ORGANIZATION Building the CAPTURE team The CAPTURE Project was launched in January 2009 with three people on board: Dr. Diane Finegood as Executive Director, David Crouch as Director of Innovation, and Executive Assistant Doreen Chiang. As the Project has ramped up, several other members have joined the team. • Dr. Marla Steinberg, Director of Evaluation • Jasmine Sharma, Manager, Public Affairs • Erinc Mullaoglu, Lead Developer • Andi Cuddington, Project Assistant • Dayna Albert, Manager, Practitioner Engagement • Lance Jian, Web Developer • John Chang, Web Developer • Dr. Michal Fedeles, Senior Project Manager • Dr. Kim van der Woerd, Senior Project Manager, Northern Engagement Project (contractor) The project has also benefited from: Simon Fraser University Co-op Students Vincent Lo, Web Developer, and Eddie Wong, Web Developer; Shaylyn Streatch, Practicum Student University of Toronto School of Health Promotion; and Ryan Lledo Communications Co-op Student, University of British Columbia. Real-World Learning: We’ve learned how important it is to use the right people within the right environment. Our initial work in platform development used an external IT development firm. Bringing staff in-house to develop the platform has ensured a better understanding of the initiative by our developers, and better support for our iterative way of working.

Learning as an organization The CAPTURE team is committed to becoming a “learning organization” within a social innovation context. This means we need to create an internal environment and infrastructure that actively seeks out and shares its learning. Within our team, we acknowledge the value of trust and risktaking, asking questions and reflecting, and working within a non-competitive environment. We have developed processes to support these principles. 5

Real-World Learning: One way we’re working towards becoming a learning organization is to document and share our learnings by blogging on the CAPTURE website. All team members and CAPTURE supporters are encouraged to share their reflections and “great big ideas” around the Project’s activities and progress. To date, the CAPTURE team has posted 21 blog entries on topics ranging from evaluation to practitioner engagement.

Practicing what we preach As a champion of ongoing evaluation in the real world, the CAPTURE Project needs to practice what it preaches. In order to track our own progress, we developed a Performance Monitoring and Evaluation Framework that identified the key metrics that will be monitored as the platform develops. For this first phase of CAPTURE, our evaluation focuses on client/user engagement and satisfaction with products and services. Real-World Learning: A key objective for the CAPTURE Project is to encourage users to reflect on and document what works and what doesn’t. Our own attempts at “walking the talk” have shown us how easily other priorities can crowd out reflective practice. It’s critical that we find tools and approaches that make this process automatic and easy.

Planning for Sustainability CAPTURE is addressing sustainability in a number of ways. The first is to undertake client-based development so we can demonstrate our utility to users and clients whose collective support can carry us forward. Our consultation and engagement activities – where we actively involve users, clients and partners in the planning, building and testing of the platform – will help us ensure the CAPTURE platform meets their needs. We are also working with other organizations with similar mandates, so we can mutually support each other’s work. We have begun to explore different business models that enable us to leverage our core CPAC funding to activities that benefit the entire primary prevention community across Canada. This could include adapting an approach commonly used by service providers, where open access is offered for a range of online services and products, while specialized services and product development are provided to individual organizations or initiatives on a cost-recovery basis.

ENGAGING BUILDERS, USERS, SUPPORTERS, AND CLIENTS Consultation The CAPTURE Project arose from a “great big idea” to create a useful and easy-to-use platform that helps chronic disease prevention practitioners and program managers learn from what they do, and share that knowledge. Achieving this goal requires truly understanding the real-world challenges and needs of the people and organizations who could benefit from this platform. That’s why consultation is such a key part of the CAPTURE Project’s operations. To date, the CAPTURE Project has gathered information and consulted with more than 500 people across the country, including practitioners, program managers, policymakers, researchers and knowledge exchange and IT experts. Our activities have included expert panel meetings, a concept mapping exercise, key information interviews, surveys, and focus groups (Appendix A). Real-World Learning Consulting with the people responsible for community-based primary prevention programs brought a clear focus to their most pressing challenges for evaluating, learning and sharing. We heard that: • Program managers and policymakers want to be able to compare outcomes across programs. • Setting up evaluation systems is expensive and time consuming. • Doing evaluation is difﬁcult for many practitioners. • Finding other practitioners who are working on the same health promotion issues is challenging. • There is no place to share the results of evaluations for others to use. • There is no place to learn how people implemented their health promotion programs.

Building Awareness & Seeking Engagement Because the CAPTURE Project depends on the ongoing input of many stakeholders, effective communication is a cornerstone of our activities – not only to build awareness about the Project, but also to encourage the engagement that this type of initiative requires. In our first two years, we have made significant efforts to reach as many potential users as possible. An important approach is face-to-face interaction with the people who can contribute to and benefit 7

from the Platform. We have presented to more than 750 people at peer-reviewed conferences and organization-sponsored meetings (Appendix A), and we’ve distributed information sheets at three events. We have been very pleased by the universally positive response. We’ve also created two promotional videos to help describe the CAPTURE Project’s goals and mandate. For ongoing communications, we rely heavily on online tools. We have 170 subscribers to our e-bulletin, which alerts stakeholders to the Project’s progress. Our website is also an important communications tool, housing information about the CAPTURE Project, news articles, CAPTURE reports, and supporting communications materials. Real-World Learning The high-level vision of the CAPTURE Project, and what it seeks to achieve, has been met with widespread and enthusiastic buy-in from lots of stakeholders. However, we have found that different groups have differing expectations around what the platform will offer. We continue to define and refine our communications around what the platform is, and how its tools and supports will contribute to an integrated, learning system.

International Advisory Board The CAPTURE Project is fortunate for the expert guidance of its knowledgeable and engaged International Advisory board, which was formed in spring 2010. This 15-member group includes health promotion funders, health promotion practitioners, evaluators, senior decision makers in health service organizations, knowledge translation specialists, intervention researchers, and social enterprise leaders. To date, the IAB has met twice (April 2010 and November 2010) to provide strategic advice on the development of CAPTURE, including input around communications, demonstrating value, platform design and project evaluation. Real-World Learning IAB members have advised us to focus on high-touch supports just as much as – or even more than – our high-tech features. In order for CAPTURE to be more than “just another website” it will need to develop features and services that facilitate relationship building. We were also urged to provide a range of supports for different levels of evaluation capacity, to focus on clients and not just users, and to continue to work on our messaging.

PLATFORM DEVELOPMENT An End-to-End Solution: Knowledge to Action and Action to Knowledge While development of the CAPTURE platform is a highly iterative process, features that support the entire knowledge to action and action to knowledge cycle are planned.

High Tech Features

High Touch Supports

Share your work

Connect with colleagues through E-networking

Plan your evaluation Access evaluation expertise and mentors Manage your intervention Attend evaluation workshops and webinars Find information and resources on chronic disease prevention interventions Find data collection tools for healthy

Access the CAPTURE E-help line Participate in on-line tutorials

eating, physical activity, and smoking cessation and control

View CAPTURE how-to webinars

Collect, analyze, and manage your data

Read how-to popups

Create evaluation reports Reflect on your practice

The platform will house a series of relational databases that will enable practitioners to search: • Intervention summaries • Practitioner learnings and reﬂections • Unpublished evaluation reports • CAPTURE generated evaluation reports • Practitioner proﬁles • Data collection tools • Other Grey literature

Foundational Work During our first two years of operation, the CAPTURE Project focused on key planning and consultation activities that would guide future platform development. This has included: • Developing the ﬁrst CAPTURE platform prototype; • Conducting a review of legislation and best practices related to privacy, security and data sharing as the basis for developing a framework for safeguarding the information housed on the CAPTURE platform; • Commissioning an environmental scan and literature review of the protocols used by existing evidence repositories to ensure the validity and reliability of the information they contain; • Contracting the development of a web crawler to systematically search, retrieve and index content from existing evidence repositories and websites onto the CAPTURE platform. • Contracting the expansion of the Tobacco Information Monitoring System (TIMS) to include measures of physical activity and eating behaviour. CAPTURE will link to the new TIMS platform, allowing CAPTURE users to seamlessly access this data source. Real-World Learning Our preliminary work in getting the CAPTURE Platform ready for development has proven to be of interest to other organizations and initiatives. We’ve written and posted a number of reports on our website to share what we’ve learned during this planning phase (Appendix B).

Knowing our niche and adding value It’s important for us to understand how the CAPTURE Project fits with other knowledge transfer initiatives within Canada. To learn from what others are doing, CAPTURE routinely scans other initiatives that facilitate practice-based evidence, evaluation, shared measurement, reflective practice, knowledge transfer and exchange, communities of practice, and connecting. This allows us to adapt and build upon the good work of others. For example, CAPTURE’s initial thinking about its evaluation planning features drew from user-testing of the Ontario-based Online Health Program Planner. The unique value-add of the platform is its user-friendly integration of resources, tools and support. The platform can provide a link to the valuable work that already exists, and help users find the resources that meet their needs.

Upcoming Feature Release CAPTURE is currently in the process of user testing its first set of features: • Share your Work – allowing users to post summaries of interventions, upload evaluation reports and other intervention resources, and reflect on their learnings; • Connect with Others – enabling users to search for practitioners and organizations with similar interests; • Find Resources – allowing users to search for information on chronic disease prevention programs and policies within the CAPTURE database and from other websites. Real-World Learning Do users look to us to validate the quality of content that is contributed to the platform, or do they want unrestricted access to absolutely everything out there? In consideration of the balance between time, budget and philosophy, we’ve arrived somewhere in the middle – our plan is to tag content with “evidence assists” to help users understand what they find.

PROVIDE TRAINING, SUPPORT & CAPACITY BUILDING CONNECTIONS Client Support The CAPTURE Project is grateful for the opportunity to work with its first client, the CPACfunded Coalitions Linking Action and Science for Prevention (CLASP Initiative). In September 2009, CAPTURE was enlisted to develop a Cross-CLASP evaluation framework. The resulting framework includes data collection tools for tracking partnering, sustainability, and knowledge transfer and exchange. This work expanded further in October 2010 when CPAC received funding from the Public Health Agency of Canada to engage organizations working on chronic disease prevention in Canada’s Northern and Remote Regions through the CLASP initiative. The purpose of this engagement is to expand the high touch support CAPTURE provides to its clients, and understand their evaluation needs and develop a strategy for meeting these needs through CAPTURE. Not only have these projects helped us develop our client service model, they have helped us to populate our tool repository. The tools developed by CAPTURE for assessing partnerships and knowledge transfer and exchange will be made available on the platform for use by the CLASPs and other initiatives.

Real-World Learning Working on the CLASP Project has enabled us to create tools that are of interest to many, if not all, health promotion initiatives and programs. The tools and support we develop as part of the CLASP evaluation have broad application to other organizations seeking to learn about and evaluate partnering, KTE, and sustainability. By making these tools available on the platform, we will be creating a shared measurement platform which will make it easier for programs to do evaluations and for others to compare across programs. This will increase the knowledge base of what works for whom in what contexts.

Partnering for Learning Since its inception, CAPTURE has made many connections with like-minded organizations that are advancing the knowledge transfer and exchange agenda. These relationships allow CAPTURE to benefit from, and contribute to, work that is occurring across the primary prevention community. In March 2010, CAPTURE co-funded a Knowledge Exchange Forum with the Knowledge Development Division of the Public Health Agency of Canada and the Propel Centre for Population Health Impact. The forum brought together 40 practitioners to discuss how organizations can facilitate evaluation for learning.

Real-World Learning Organizers of the evaluation for learning forum remarked that the CAPTURE team’s ability to translate “systems thinking” into a practical framework for building evaluation capacity was especially novel and well-received.

SEEKING FEEDBACK IN ORDER TO CONTINOUSLY IMPROVE User Preferences With the first three features now on the Platform, CAPTURE is heavily focused on user testing of its products. Our early efforts in relationship-building consultation and engagement are paying off for us now, as many practitioners have stepped forward to help with the user testing. Real-World Learning The Project has learned how engaging in user testing of tools offered by others can inform its approach to what is available on the platform. In the spring of 2010, CAPTURE worked with the Towards Evidence-Informed Practice (TEIP) program of the Ontario Public Health Association to engage practitioners in the user-testing of OPHAâ&#x20AC;&#x2122;s On-line Health Promotion Planner. The results of this process, in conjunction with what users have told us about the need for simple, easy-to-use tools, had a major impact on the development of the platform.

ACKNOWLEDGMENTS The CAPTURE Project cannot achieve its mission alone, nor does it want to duplicate existing initiatives. Our goal is to work with other organizations and initiatives in order to mutually enhance our shared objectives. CAPTURE would like to thank the following individuals and organizations for their contributions during our first years:

K Friends of The CAPTURE Project

A Alison Osborne, Monkey Hill Communications B Barb Riley, Propel Centre for Population Health Impact Brian Hyndman, Heather Manson and Melody Roberts, Ontario Agency for Health Protection and Promotion C Canadian Partnership Against Cancer Catherine McNeill Cathexis Consulting Chronic Health Promotion and Chronic Disease Prevention Initiatives Database D Dayna Albert, Manager of Towards Evidence Informed Practice, Ontario Public Health Association Dinalex Technology Solutions E Ed McIvor H The Health Communication Unit and the Ontario Health Program Planner Heather Epstein I InSource Research Group

Kerry Robinson, Public Health Agency of Canada and The Canadian Best Practices Portal Kylie Hutchinson, Community Solutions Planning and Evaluation L Larry HershďŹ eld and Associates M Megan Ward, Peel Regional Public Health Members of the International Advisory Board N National Collaborating Centre for Methods and Tools P Participants in the 2009 Expert Panel Participants in the 2009 Stakeholder Workshop Phase 5 R Raincity Studios Rainer Beyleveldt Ruth Yeo, Hourglass Consulting Rob Schwartz, Ontario Tobacco Research Unit S Steve Manske, Propel Centre for Population Health Impact Stringbean Design Switch Marketing and Communications

Appendix A:

CONSULTATIONS AND COMMUNICATIONS

Consultation March 2009

Expert Panel

March – May 2009

Concept Mapping

July 2009

National Collaborating Centres Consultations

May – October 2009

Key Informant Interviews

October 2009

Stakeholder Workshop

December 2009

Peel Regional Health Department Consultations

Jan – March 2010

Ontario Focus Groups

February – May 2010

Usability Testing

February – April 2010

Ontario Evaluation Needs Assessment Survey

April 2010

1st International Advisory Board Meeting

June 2010

MEND (Mind, Exercise, Nutrition…Do it!) Consultations

August 2010

National Needs Assessment

November 2010

2nd International Advisory Board Meeting

Presentations March 2010

Population Health Evaluators Network (Public Health Agency of Canada)

March 2010

3rd Annual Conference on the Science of Implementation and Dissemination

April 2010

Canadian Best Practices Portal

May 2010

BC Early Childhood Development Evaluation Project

May 2010

Canadian Evaluation Society Annual Conference

June 2010

Vancouver Evaluation Consortium (United Way of the Lower Mainland, Vancouver Foundation, City of Vancouver, & Vancouver Coastal Health)

June 2010

Canadian Public Health Association Annual Conference

June 2010

Public Health Agency of Canada (National Office)

July 2010

BC Provincial Government (Ministry of Child and Family Development, Ministry of Education, Ministry of Health Services, and Ministry of Healthy Living and Sport)

October 2010

British Columbia Ministry of Health Services Policy Rounds

October 2010

Canadian Evaluation Society Ontario Chapter Annual Conference

November 2010

American Evaluation Association Annual Conference

November 2010

BC Health Evaluation Community of Practice, Vancouver Coastal Health “Just in Time” Evaluation Coaching Series 15

Appendix B:

CAPTURE REPORTS

Publish Date

Title

September 7, 2009

CAPTURE Sought guidance from expert panel

September 17, 2009

Consultations and collobarations central to CAPTURE’s Development Lessons learned from the concept mapping exercise

October 7, 2009

Summer session acts as sounding board with practitioners Interviews describe CAPTURE’s critical success factors

November 20, 2009

A snapshot of practitioners’ point of view Consultations culminate at productive fall workshop

February 1, 2010

CAPTURE forms International Advisory Board

February 8, 2010

Think Tank: How can we CAPTURE practice-based evidence?

February 10, 2010

TEIP helps CAPTURE learn about practitioners’ needs

February 15, 2010

Forum: Evaluation for learning and improvement

February 24, 2010

Peel Region practitioners share their perspectives

April 9, 2010

If we build it will they come?

April 21, 2010

Think tank attendees give feedback on reflective practice Summary of initial consultations and information gathering

June 7, 2010

Report on Inaugural IAB meeting

October 14, 2010

Important sources of evidence not easy to access

October 18, 2010

CAPTURE Project gains deeper understanding of evaluation needs

October 26, 2010

CAPTURE and the Canadian Partnership Against Cancer partnering to learn about the evaluation needs of Northern and Remote Communities

December 9, 2010

Demand is high for real-world evidence

January 27, 2011

Creating a system to support reflection and learning

February 1, 2011

Report on the second IAB meeting

Available upon Request

Review of evidence repositories CAPTURE Evaluation Plan Privacy, Security and Data Sharing Framework Cross-CLASP Evaluation Framework Review of tools for assessing partnerships Review of tools for assessing sustainability Review of tools for assessing knowledge transfer and exchange