Evaluation of safe boating course by Karen Kelly

Evaluation of The Safe Boating Course Prepared for: Prepared by: Date: Project #:

Boise State University - Ed Tech 505 - Dr. John Thompson Karen Kelly - Evaluator (Student) July 30, 2010 24389

Abstract This report will examine whether the safe boating course offered in Nelson, B.C. is meeting expectations and whether the program should remain as is, or be modified. In this evaluation process the major framework of the decision making model is used, to survey and test participants before and after a 3-hour boat safety course. The evaluation paper then uses the results of the surveys and the tests to confirm that the training is meeting the expectations and also makes considerable recommendations to the stakeholders on the following: 1) Increase the length of the evaluation project to allow for a larger sampling size, 2) Conduct a follow-up survey, of the participants initially surveyed, after the boating season to determine how ‘long-term’ the effectiveness of the course is, 3) Conduct a further evaluation to determine the feasibility of modifying the existing program by breaking it into separate courses (and licenses) for a) freshwater, and b) saltwater boating, 4) Offer the course through several means and different times including late winter, early spring, and online, 5) Offer supplemental easy to access boat safety information on the web including information on cold water shock, PFD/Life Jackets, and Boozing/Cruising, and 6) Offer frequent ‘refresher courses’ to all boaters to support the learner in remembering boat safety content long term.

Table of Contents Page

Introduction

Description of the Program

Program Goal and Objectives

Program Components

Evaluation Method

Participants, Procedures, and Data Sources

Surveys

Tests

Study Limitations

Results

Discussion

Invoice

References

!"#$%#&'()*&+!*,#-!*./#&'()*&0#'('()*10/)0#2** 3!

! Evaluation of The Safe Boating Course Introduction Ideally, all persons operating a boat on any waters are highly trained and competent in boat safety and usage. In fact, the reality is that this is not the case. As of September 15, 2009 due to an increase in the amount of people recreating in boats and the high rate of accidents on the water, it has been federally mandated, in Canada, that all boat operators require a boating license (Mandatory Watercraft Operator License in Canada Effective Sept. 15, 2009, 2009). This also means that all boaters are required to take a safe boating course. A $250.00 fine will be issued to any boater that fails to produce their Pleasure Craft Operator Card, which is proof that they have become a qualified Pleasure Craft Operator. For this new requirement, many boat safety courses are been offered, as a means to train boat operators in boat safety. Little is known about the effect of existing programs on the boat operator. In fact, “There is no data available to demonstrate whether the programs are working to increase knowledge and awareness of safe boating practices” (Evaluation of Transport Canada's Contribution to the Canadian Safe Boating Council and the Canadian Red Cross, 2006, p. 15). The purpose of this document is to report, after conducting an evaluation, on the feasibility of offering the safe boating course in the community of Nelson, B.C.. It is to determine if the training program, in place, provides the learner with adequate safe boating skills, more specifically with effective training, with quality information and with a positive impact. This report contains a brief description of the program, evaluation methods, results, and recommendations made to the provider of the program. Description of the Program The safe boating course is a 3-hour classroom course. It was created to teach novice to experienced boaters the safety of boating. Each person, that has completed this training program, is required to take a boaters examination. After passing the examination (With a minimum of 75%), the learner will obtain a B.C. Boating License or Pleasure Craft Operator Card, which authorizes the individual to operate a pleasure craft. There are age limits for the operation of specific size motorboats. These details can be found at http://www.boaterexam.com/canada/. Program Goal and Objectives The goal of the safe boater-training program is to promote boating safety awareness by encouraging boaters to focus on the importance of following safe boating practices including: • Evaluating their capabilities and limits, • Preparing themselves adequately before heading out, and • Being responsible on the water.

!"#$%#&'()*&+!*,#-!*./#&'()*&0#'('()*10/)0#2** 4!

! The objectives of the safe boater-training program are to: • Raise public awareness of boating safety issues, • Increase the level of pleasure craft (or small boat) operators following safe boating practices, • Improve national boating incident data quality and collection systems to support evidence-based awareness and education initiatives, and • Contribute to the reduction in loss of life, injuries and property damage due to boating accidents. Program Components This evaluation’s focus uses the decision-making model as a means to determine the effectiveness, quality and impact of the training on boaters of all ages. The safe boater-training program is designed to promote boating safety awareness by encouraging boaters to focus on the importance of following safe boating practices. At a cost of $75.00 + tax (early online purchase) or $85.00 + tax (at the door), the learner receives a 49 page booklet. It is believed, by all stakeholders, that completion of the training program will foster attitudes and promote practices of safe boating. This information will be used in the following ways: 1) Provide valuable decision making information on the current training program (To keep program as is, or to modify it accordingly.) and, 2) Indicate, whether or not, the training program is meeting expectations. The training program consists of approximately 3.0 hours of training. There are 8 modules. Each module is broken down into smaller units. The modules are completely self-contained. They are taken in chronological order. All of the training is done, using a checklist of met objectives. If learning transfer has not been completed, the learner can re-do specific modules and take practice-tests (https://www.boatsmartexam.com/en/practiceQuiz/practiceQuiz2.html) for each individual module. To complete the course, each student is required to take a 36multiple choice question test. The cost of this test is included in the course fee. The test is taken when all of the modules have been completed. The test may be repeatedly taken, as many times as is necessary to pass it with a minimum of 75%. Evaluation Method Participants, Procedures, and Data Sources This evaluation was conducted between the dates of July 22nd - July 25th, 2010 in Nelson, B.C., Canada. The stakeholders are the program review team, and the instructor of the course. For this evaluation, 30% of the participants, that took this particular course, were sampled. This involved 4 participants, ranging in age from 15 – 51. The focus of the evaluation was to determine if the safe boater-training program offered, positively impacts the learner by providing effective training and quality information, and if the course should continued to be offered in the community. This evaluation used 2 methods to evaluate the safe boater-training program. To assess the impact, quality and effectiveness of the training, qualitative data was collected through pre and post learner surveys. To further assess these 3 intended

!"#$%#&'()*&+!*,#-!*./#&'()*&0#'('()*10/)0#2** 5!

! outcomes, quantitative data was collected through test results. These results were compared to the results of a pre-test that had been taken previous to the course offering. The learner surveys were given in order to check for a willingness to participate in the program, as well as to implement the training program into real safe boating practices. They were also completed in order to check for any behavior changes due to the training and to check for any preconceptions about the training. Lastly, the survey checked for any specific needs that were either addressed or not addressed in the training. This would be helpful as well for future course developments. The survey consisted of ten Likert-type questions where options for choices were (1) excellent (2) good (3) fair (4) poor and had one or two open ended questions. The survey was completed by 4 of the participants in the course, both before and after the training had been completed. The pre-tests were given before the course started and then compared to the final summative results of the post-test. The test results will aid in determining the impact and the quality of the course on the learner. Results Surveys Table 1 (Before Course)

Questions

1 Excellent

2 Good

3 Fair

4 Poor

MEAN

1. My reaction as to how effective this course will be 2. My overall impression of this course is 3. My attitude toward taking this course is 4. My reaction as to what the quality of this course will be 5. The time that this course if offered is 6. My motivation for taking this course is 7. My comfort level with the rules & regulations 8. My comfort level with operating a power boat is 9. I am aware of the equipment required 10. I am aware of the nautical rules Totals

2.25

3.5

1.5

3.5

1.5

4.0

2.5

2.75

3.25

2.7

!"#$%#&'()*&+!*,#-!*./#&'()*&0#'('()*10/)0#2** 6!

! Responses to the two surveys using an online survey tool are summarized above in Table 1 (before course), and below in Table 2 (after course), and Figures 1 – 4 respectively. Each table shows the number of learners selecting each response, the mean for individual questions, and the overall mean for the survey. Each figure graphically represents the answers that participants gave in response to openended questions. The first survey focused on the perceived effectiveness, quality, impressions and attitudes of the participants before their training. It also addressed issues of the time frame of the course, motivation for taking the course, and included four questions based on the participants knowledge of boating and boat safety. Pre-course questions about behaviors towards the course indicated a negative attitude to taking the course, as well as reported ‘bad timing’ for when it is offered. (Even though, all of the participants were motivated to take the course so that they could receive their license.) Overall the participants expected that the quality of the training would be high, and had a fair-good impression of the course and it’s effectiveness. Participants unanimously agreed that they required training in boating safety rules and regulations, rules that govern the water, and awareness of the equipment required. All of the participants were comfortable, before the course, with operating a boat with a motor. Figure 1 Age Ranges of Participants "#$%&!'(! )(*!

0-!,!(+! )(*! .-!,!0+! )(*!

'+!,!)+! )(*!

)-!,!.+! /*!

!"#$%#&'()*&+!*,#-!*./#&'()*&0#'('()*10/)0#2** 7!

! Figure 2 Is this a 'good time' to offer the course? 1%23!45!426! 789%3!45!42!#856! :;<=%!<8>!?8>@$!8AA%&!45!$>&4#B! 5C%!D%%E!4#25%;$F! G5!$8%2#H5!I;55%&!58!I%!

The figures reported the results of the open-ended questions. For figure 1, the age range of the participants ranged from 15 â&#x20AC;&#x201C; 51 with no representatives in the 27 â&#x20AC;&#x201C; 36 age range. For figure 2, a suggestion to offer the course during the week was made by half of the respondents. Table 2 (After Course) Questions 1. My reaction to this courses effectiveness 2. My overall impression of 3. My attitude towards 4. My reaction to the quality 5. My comfort level with the rules & regulations of 6. My comfort level with operating a power boat 7. My knowledge of the equipment required 8. My awareness of the nautical rules 9. The pace of the course was appropriate 10. My reaction to the cost Totals

1 Excellent 2

2 Good 2

3 Fair 0

4 Poor 0

MEAN

1.25

1.0

1.5

1.0

1.5

1.0

1.5

1.0

1.225

1.5

!"#$%#&'()*&+!*,#-!*./#&'()*&0#'('()*10/)0#2** 8!

! The second survey followed up with 8 similar questions to the first survey. This time views were surveyed ‘after’ the course had been completed. Overwhelmingly, all post-training participants experienced ‘attitude improvement’ after taking the course. Unanimously it was concluded that the course was of high quality, effective and that the overall impression was good- excellent. In addition all participants agreed with the pace of the course and the cost of taking it. In the areas of knowledge (four questions), all four areas saw improvements in their comfort levels of rules and regulations of boat safety, of the equipment required, and of the rules that govern the water improved. Confidence, although not lacking before taking the course, in operating a boat with a motor, also increased. The tables reveal that overall the participants are satisfied with the safe boating course. The overall mean is 1.225 indicating a rate of “good - excellent”. Responses to general attitudes and satisfaction of training ranged from 1.0 to 1.5. Responses to questions about cost and pace of the course ranged from 1.0 -1.5. Finally responses to knowledge obtained on boating safety ranged from 1.0 – 1.5. Figure 3 (After Course) (Open-Ended Question Summary) What would you change about the course? J%9;&;5%!A&%2C!D;5%&! ;#$!2;@5!D;5%&!=8;5%&! 5&;4#4#B! :;E%!9%89@%!5;E%! &%A&%2C%&!?8>&2%2F! KAA%&!45!.!I8#5C2! %;&@4%&! KAA%&!4#A8&I;548#! 8#@4#%!

Figure 4 (After Course) (Open-Ended Question Summary)

!"#$%&'()**+,*(-.#$()(/$%00( 1,2+%3,()&4"35#$%"&("&( L8@$!M;5%&!JC8?E! NOPQR4A%!S;?E%5! T88U%!V!L&>42%!

!"#$%#&'()*&+!*,#-!*./#&'()*&0#'('()*10/)0#2** 9!

! Figure 3 summarizes the surveyed participants desire to change the course design so that there is a separate fresh water component and a separate salt water component, to force participants to take refresher courses, to possibly offer the course ‘online’, and finally to ensure that the course is offered in the late winter (or early spring) rather than the summer. Figure 4 summarizes that participants, after taking the boat safety course, still feel a need to learn more about the topics of coldwater shock, PFD/Life jackets, and Boozing and Cruising. Tests Responses to the two tests (pre and post) using 1) The interactive practice exam at https://www.boatsmartexam.com/en/practiceQuiz/practiceQuiz2.html, and 2) the final online test at http://www.boaterexam.com/canada/overview-en.aspx are summarized below in Table 3. A series of 36 multiple-choice questions were asked in both tests. Table 3 (Results of Tests: Pre & Post) Participant #

Age of Participant

Pre - Test %

Post – Test % #1

2 3 4

20 46 51

65 42 70

99 73 95

N/A N/A 80 N/A

Overall the test results indicate that for all the participants, taking the safe boating course improved their test scores. Only 1 person was required to ‘re-do’ the test a second time, as they did not pass with the expected 75% the first time. This person’s weakness was in the area of salt water, and since he/she lives on a fresh water lake, he/she thought that the questions did not pertain to their boating needs. Study Limitations This evaluation was conducted as an Educational Technology 505 assignment during the summer of July 2010 for Dr. John Thompson. The scope of the project was limited in time, so keeping that in mind, to further legitimize the evaluation, the program studied requires a larger and a random sample size to ensure the study is rigorous and that the collection is valid. Also, ideally the results of the post-tests should be compared to data results from learners that have taken the course through various other means. Given the time frame available, the evaluator was not able to find this necessary previous data to make such a comparison.

!"#$%#&'()*&+!*,#-!*./#&'()*&0#'('()*10/)0#2** :!

Discussion

This evaluation was undertaken to determine if the safe boating course is meeting Ministry of Transport (MOT) expectations and if the program should be kept as is or modified to make improvements. The areas that were evaluated included determining if the course positively impacts the participants, offers quality information regarding boat safety, and is effectively administered to the participants. The surveys of the participants revealed an overwhelmingly majority of positive aspects after taking the boat safety course. All post-training participants experienced ‘attitude improvement’, considered the pace and cost adequate, considered the course ‘high quality’ and ‘effective’, and completed the course with a ‘good- excellent’ impression of it. The participants reported an increase in knowledge in the areas of rules and regulations, equipment required, rules that govern the water, and operation of a boat with a motor. The test results of the participants revealed, overall, that taking the safe boating course improved participant test scores. This means that taking the safe boating course positively impacts the learner in that it provides effective training and quality information to pass the test. The evaluator recommends the following: 1) Increase the length of the evaluation project to allow for a larger sampling size, 2) Conduct a follow-up survey, of the participants initially surveyed, after the boating season to determine how ‘long-term’ the effectiveness of the course is, 3) Conduct a further evaluation to determine the feasibility of modifying the existing program by breaking it into separate courses (and licenses) for a) freshwater, and b) saltwater boating, 4) Offer the course through several means and different times including late winter, early spring, and online, 5) Offer supplemental easy to access boat safety information on the web including information on cold water shock, PFD/Life Jackets, and Boozing/Cruising, and 6) Offer frequent ‘refresher courses’ to all boaters to support the learner in remembering boat safety content long term. In summary, the boat safety-training program that was evaluated is meeting the goals and objectives of the MOT. Invoice

Evaluation of The Safe Boating Course ITEMIZED

Invoice # 24389 COST

DESCRIPTION

Personnel Karen Kelly Travel and per Diem Printing and Communication

5 days (5 x $500) 4 trips to course facility at .35 / km (4@ 15km round trip x .35)

$ 2500.00 $ 21.00 $

30.00

Balance Due: $2551.00 Thank you for your business, please make payment within 30 days.

!"#$%#&'()*&+!*,#-!*./#&'()*&0#'('()*10/)0#2** ; <!

! References Boulmetis, J., & Dutwin, P. (2005). The ABCs of evaluation: Timeless techniques for program and project managers (2nd ed.). Jossey - Bass. Evaluation of Transport Canada's Contribution to the Canadian Safe Boating Council and the Canadian Red Cross. (2006, March). . The Department Evaluation Services Branch. Retrieved from http://www.tc.gc.ca/media/documents/corporate-services/obs.pdf Mandatory Watercraft Operator License in Canada Effective Sept. 15, 2009. (2009, September 16). Marine Web News. Retrieved from http://boatworktimes.com/library-mainmenu110/license-mm-/2223-mandatory-watercraft-operator-license-in-canada-effectivesept15-2009.html Safe Boating Guide (2009). Canada. Retrieved from http://www.tc.gc.ca/publications/en/tp511/pdf/hr/tp511e.pdf

!"#$%#&'()*&+!*,#-!*./#&'()*&0#'('()*10/)0#2** ; ;!