Evaluating the Kamloops P.I.E.C.E.S ™ Demonstration Project Carol Ward, MD FRCPC a,b, Caili Wu, PhD a, Sandy DaSilva a, Shelley Mitchell a & Alicia Miehling a a. Tertiary Mental Health and Substance Use Services, Interior Health, BC, Canada b. Department of Psychiatry, University of British Columbia, BC, Canada
Introduction
Results
The demand for coordinated, high-quality psycho-geriatric care is increasing. There are gaps in knowledge of dementia care in the province. In order to fulfil the present and emerging demands, the P.I.E.C.E.S ™ Education Program was introduced to Interior Health to improve staff knowledge and skills and system characteristics in 2012. P.I.E.C.E.S ™ is a best practice learning and development initiative that provides an approach to understanding and enhancing care for individuals with complex physical and cognitive/mental health needs and behavioral changes. The P.I.E.C.E.S ™ approach considers the person’s Physical, Intellectual, and Emotional health, supports strategies to maximize Capabilities, the individual’s social and physical Environment, and his/her Social self (cultural, spiritual, life story, family). The P.I.E.C.E.S ™ framework provides a common set of values, a common language for communicating across the system, and a common yet comprehensive approach for thinking through problems to enhance the capacity of those providing care, services, and support to older adults with complex needs and associated behaviors. With Kamloops as a demonstration site, two P.I.E.C.E.S ™ training programs, the P.I.E.C.E.S ™ 24-hour trainer program, and the P.I.E.C.E.S ™ Leadership & Performance Improvement program, were delivered in February 2012. Program evaluation was proposed in the planning stage to assess the outcomes and to monitor further rolling out of the P.I.E.C.E.S ™ program.
Methods A logic model was developed at the planning stage. Critical Indicators of output and outcomes were identified. Pre-, post-, and follow up surveys were designed to capture relevant knowledge, skill, attitude, and behaviour ratings and changes. All surveys included both closed- (5-point Likert scale) and open- ended items to measure the changes of knowledge etc . and suggestions/recommendations for the P.I.E.C.E.S ™ implementation.
Results (Continue)
• Related activities
• Longer- term outcome
Several networks such as P.I.E.C.E.S ™ Steering, Leadership, and Participant Committees have been established. Members have met on a regular ongoing basis to guide, discuss, and review issues related to the implementation of P.I.E.C.E.S ™. Training sessions such as tailored mini P.I.E.C.E.S ™ education has been offered to staff members regularly. A one-day Physician session and a Refresher day have been delivered to physicians and original participants. Original trainers have facilitated three formal 24-hour training sessions to staff members and will continue to deliver the program. Most importantly, the P.I.E.C.E.S ™ templates have been integrated into various clinical settings such as rounds, safety huddles, care planning, Kardex, and additional documentation to translate the knowledge into daily practice.
• Immediate outcome For the P.I.E.C.E.S ™ Leadership & Performance Improvement program, the response rate to the questionnaire was 87.5%. Overall, the results showed respondents had a very high rating for the P.I.E.C.E.S ™ training sessions, education team, information relevance, and knowledge of learning principles. For the P.I.E.C.E.S ™ 24-hour trainer program, the response rate was 100% for both the pre- and post- surveys. More respondents rated their knowledge/confidence/skills level as “Good” or “Excellent” after the training. And most respondents rated their knowledge change as “Improved” or “Very much improved” after the training. Wilcoxon tests were conducted to compare the pre- and post- response. The results showed that the improvement on most items reached statistical significance. For example, when asked to rate current knowledge of major dementia and other mental health disorders in older adults, the results of pre- and post- training comparison is shown in Figure 1. More respondents (87% vs. 56%) rated their knowledge as “Good” or “Excellent” after the P.I.E.C.E.S ™ training. Wilcoxon test showed the improvement reached statistical significance, z = -3.75, p <.01.
For the P.I.E.C.E.S ™ 24- hour Trainer program, a pre- and a post- program survey was designed. The main purpose of the two surveys was to assess the immediate outcome of the P.I.E.C.E.S ™ program, i.e., the knowledge, skill, and confidence changes right after the training. Thirty participants, including front line care givers and educators, were selected to attend the training and were asked to complete the surveys at the time of registration and at the end of the training. Follow-up post-surveys were mailed to the participants about every 6 months after the training to evaluate relatively longer-term outcomes and gather feedback/suggestions for future implementation.
Conclusions
The results of the post- surveys showed greater knowledge, confidence, and skill improvement after the training by participants. The results of the follow-up surveys showed overall positive changes/trends towards the P.I.E.C.E.S ™ project’s longer objectives. A number of education/training sessions have been delivered to staff members including front-line care givers and physicians. Networks have continued to monitor the implementation of this framework. P.I.E.C.E.S ™ templates/tools are being utilized in daily practice to prioritize care, enhance communication, and promote interdisciplinary collaboration.
Recommendation The recommendations are mainly based on the information collected from the openended survey items at different time frames.
Percentage
Knowledge of dementia For the P.I.E.C.E.S ™ Leadership & Performance Improvement program, a postprogram survey was designed to assess the satisfaction level towards the training team and session, as well as whether the goal of knowledge and attitude changes was achieved right after the training. Sixteen leaders (managers and team leaders) were selected and attended the one-day training and were given the post-evaluation survey at the end of the training.
Follow-up surveys for the 24-hour trainer program that evaluate a relatively longerterm outcome were conducted every 6 months after the initial training. Most participants have used P.I.E.C.E.S ™ knowledge in their work after the training in multiple occasions. Results from both the post- 6 month and one year surveys (response rate were 73% and 45% respectively) showed a general trend of improvement towards the longer-term goals of the P.I.E.C.E.S ™ project such as decreased stress level at work and increased motivation towards professional development. The difference between the pre- and follow-up rating on the item that assessed the work place implementation of a systematic and comprehensive approach to solve complex issues (P.I.E.C.E.S ™) reached statistical significance.
90% 80% 70% 60% 50% 40% 30% 20% 10% 0%
77% 53% 33%
3% 0% 1-Poor
13%
7%
3%
0% 2
3
4
10%
Recognize that the 3-question framework and accompanying assessment tools are key to understanding psycho-geriatric and complex care issues Utilize a common language across sectors as developed by P.I.E.C.E.S ™ Apply P.I.E.C.E.S ™ knowledge to daily practice such as rounds, safety huddles, care planning, Kardex and additional documentation Implement P.I.E.C.E.S ™ education and in-services Maintain directory of assessment tools applicable to this practice area
5-Excellent
Response Pre
Post
Figure 1. Percentage of respondents on each level of knowledge of dementia before (pre) and after (post) the P.I.E.C.E.S ™ training.
Key References Putting P.I.E.C.E.S ™ Together (http://www.piecescanada.com/). The P.I.E.C.E.S.™ Consult Group (2012). P.I.E.C.E.S.™ Learning Options: Interior Health, British Columbia. Wholey, J.S, Hatry, H.P., & Newcomer, K. E. (2004). Handbook of practical program evaluation. San Francisco: Jossey-Bass Publishers.
This poster is prepared for the InspireNet's Fall Conference Connect 2013: Health Services Research at Work: Using Evidence to Transform Care , September 16 th, 2013, Vancouver, BC, Canada.