Formative Evaluation

Formative Evaluation Plan Transporting Chemicals

Expert Review

Pilot Studies

Editorial Review

Review Phase

# of Reviewers

Main Purpose

1. Expert Review

2-3 Subject Matter Experts Content Accuracy

2. Pilot Study (1 to 1)

1 Trainee, 1 instructor*

Program Usability

3. Pilot Study (group)

6 Trainees, 1 instructor*

Program Usability

4. Editorial Review

2-3 Editors

Program Readability

*The same trainers for both 1 to 1 and group pilot studies would be ideal.  

1

Formative Evaluation

Expert Review

Decision-Making to Date

   

A needs assessment was conducted in October, 2012.

At this time I spoke with glove box heads from three separate research groups at McGill University.

A pre-meeting questionnaire was also distributed. (See results below) It was decided by the feedback received and in conjunction with the project sponsor (Jeffrey Quesnel) to make the training accessible to beginner level frequent and beginner level infrequent users and to target four main training areas: a. b. c. d.

Using the glove box computer.

Bringing chemicals into the glove box.

Bringing chemicals out of the glove box.

Preparing the workspace for the next user.

Identified Performance Gaps from Questionnaire

# of People who Identified this

Not preparing the workspace for the next user

3

Using solvents and/or chemicals inappropriately

3

Increasing O2 or H2O levels

2

Not repairing holes in the gloves

2

Inability to do maintenance

1

Tightening valves and lids too tightly or too loosely

1

Mistakes in using the vacuum pump

1

Issues related to users overlapping

1

Identified Training Issues from Questionnaire

# of People who Identified this

Students lack a basic understanding of the central concepts of glove boxes The glove box is a common space used by multiple users

4

Not all techniques translate across labs & equipment

1

No consistent training protocol exists for glove box training

1

Not sure of the student’s prior knowledge

1

The trainer lacks knowledge himself or herself in some areas Some techniques have very complex procedures

1

There is lots of information to cover in a limited timeframe

1

2

2

1

Formative Evaluation

Expert Review

Background Information

   

In order to ensure that the content being prepared for the glove box training program is accurate, logical, and safe. I will be asking several subject matter experts (SMEs) to review the content produced to date. A pilot study & editorial review will also be conducted.

For the purpose of this review a small questionnaire has been created to direct your attention to important aspects of the project, with a place for general feedback at the end of sections 2 & 4.

Your honest and full feedback is welcomed! The more information you provide the more likely your comments will translate into a better training experience for everyone involved, so please don’t be shy!

Should you have any questions, or should you like to follow up further on a particular point, please contact Jessica Haak by e-mail, jessica.a.haak@gmail.com or by phone, 514.589.4251.

Section One: Learner Package Accuracy Instructions: Please review the materials making sure to assess each of the elements below. It is asked that you annotate the document using the “review comments” function in PowerPoint, located within the tools menu.

1. Safety information in the introductory section.

2. “4 important glove box pointers” & labeled glove box schematic.

3. Glove box computer usage.

4. Sequence of steps relating to bringing a chemical into the glove box.

5. The 8 steps listed in relation to bringing a chemical into the glove box.

6. Bringing chemicals into the glove box safety review.

7. The 6 steps listed in relation to bringing a chemical out of the glove box.

8. Bringing chemical out of the glove box safety review.

9. The 6 steps listed in relation to preparing the glove box for the next user.

10. Worksheets 1, 2 & 3 11. Videos 1, 2, 3 & 4 12. Evaluation Section 13. Glossary 14. Job Aid

3

Formative Evaluation

Expert Review

Section Two: Learning Package Overall Impressions Instructions: Please answer the questions below in a separate document & submit them to the instructional designer.

1. What 3 words would you use to describe the learning package you just viewed?

2. Name the elements that would think are most important to preserve in the final draft.

3. Name the elements that you would most like to see improved.

4. Name the visuals that you most appreciated.

5. Name the visuals that you would like to see improved.

6. What topic do you think was best articulated in the package?

7. What topic do you think was worst articulated in the package?

8. Were there any safety issues that you noted? If so, what advice would you give to improve this issue?

9. How did you find the sequence of the program? Was it logical? If not, what advice would you give to improve this issue?

10. Did you find it to navigate through the content? If not, what advice would you give to improve these issues?

11. Please provide your general feedback on the learning package.

Section Three: Instructional Package Accuracy Instructions: Please review the materials making sure to assess each of the elements below.

It is asked that you annotate the document using the “review comments” function in PowerPoint, located within the tools menu.

1. Instructions for the teaching the glove box computer.

2. Instructions for teaching how to bring a chemical into the glove box.

3. Instructions for teaching how to bring a chemical out of the glove box.

4. Instructions for teaching how to prepare the glove box for the next user.

5. Instructors for conducting the evaluations.

4

Formative Evaluation

Expert Review

Section Four: Instructional Package Overall Impressions Instructions: Please answer the questions below in a separate document and submit them to the instructional designer.

1. What 3 words would you use to describe the learning package you just viewed?

2. Which instructions do you think would be most successful in a training setting?

3. What instructions do you think would be least successful in a training setting?

4. Is there any content that you would like to see added to the instructional package?

5. Is there any content that you would like to see removed from the instructional package?

6. Were there any safety issues that you noted? If so, what advice would you give to improve this issue?

7. How did you find the sequence of the instructor package? Was it logical? If not, what advice would you give to improve this issue?

8. Please provide your general feedback on the instructional package.

Thank you very much for your feedback!

5

Formative Evaluation

Pilot Study (Trainee & Trainer Information)

Background Information



The purpose of this pilot study is to assess how easy it is to read through and use the content in the glove box training program.



Through this process it is hoped that any procedural gaps, poor labeling, logistical problems, and inappropriate teaching strategies written into the program are identified.



To best accomplish this, it is asked that you follow the steps in the training to identify problem areas (in the event that you notice any safety-related issues please always air on the side of logic & caution).



Your honest and full feedback is welcomed! The more information you provide the more likely your comments will translate into a better training experience for everyone involved, so please don’t be shy!



Lastly, it is important that trainees in the pilot study are able to complete the entry level tasks prior to participating.



Task

✔

Name the major parts of the glove box.

✔

Label a basic diagram of a glove box.

✔

Explain how the value on the vacuum gauge relates to the pressure inside the antechamber.

✔

Explain how the glove box removes oxygen and water from its atmosphere.

✔

Explain what types of chemicals are incompatible with the O2 purifier in the glove box.

✔

Explain what each of the four circular icons on the glove box computer main menu represent in relationship to the glove box.

✔

Explain what types of reactions are unsafe or impossible to conduct in a glove box.

✔

Should you have any questions, or should you like to follow up further on a particular point, please contact Jessica Haak by e-mail, jessica.a.haak@gmail.com or by phone, 514.589.4251.

6

Formative Rate YourEvaluation Training

Pilot Study (Trainee)

Instructions: Marking an ‘X’ in the appropriate column indicate whether you strongly disagree (SD), disagree (D), are neutral to (N), agree (A) or strongly agree (SA) with the following statements.

Statement

SD

D

N

A

SA

1. The learner package was easy to follow.

SD

D

N

A

SA

2. The visuals added to the experience of the program.

SD

D

N

A

SA

3. I believe this program could be completed in 30 minutes.

SD

D

N

A

SA

4. I found the order of the sections logical.

SD

D

N

A

SA

5. I like the red highlighting of safety points.

SD

D

N

A

SA

6. I found the large job aid poster useful.

SD

D

N

A

SA

7. I thought the activities in the program engaging.

SD

D

N

A

SA

8. I found the general layout of this program unappealing.

SD

D

N

A

SA

9. The videos in the program were useful.

SD

D

N

A

SA

10. The order of the elements in this program were not logical.

SD

D

N

A

SA

11. The tone of the writing in the program was appropriate.

SD

D

N

A

SA

12. I found the introduction prepared me for the training.

SD

D

N

A

SA

13. This program was hard to follow.

SD

D

N

A

SA

14. The evaluation preparation section was useful.

SD

D

N

A

SA

15. The material in this program was hard to follow.

SD

D

N

A

SA

16. I found the glossary was useful.

SD

D

N

A

SA

17. I would use the materials in the learner package after the training was done to help me in my work.

SD

D

N

A

SA

7

Formative Evaluation

Pilot Study (Trainee)

Instructions: Please check off the boxes that apply, and provide additional feedback in the ‘other’ section.

1. Which sections of the program did you find confusing?  Table of Contents

 Section 1

 Evaluation Prep Page

 Course Info Page

 Section 2

 Worksheets

 Training Prep Page

 Section 3

 Videos

 Safety & Materials Page

 Section 4

 Glossary

Why did you find these sections confusing? Feel free to include other sections not listed above. ___________________________________________________________________ ___________________________________________________________________ ___________________________________________________________________ ___________________________________________________________________ ___________________________________________________________________ ___________________________________________________________________ 2. Which sections of the program did you find well-articulated?  Table of Contents

 Section 1

 Evaluation Prep Page

 Course Info Page

 Section 2

 Worksheets

 Training Prep Page

 Section 3

 Videos

 Safety & Materials Page

 Section 4

 Glossary

Why did you feel that these sections were well-articulated? Feel free to include other sections not listed above. ___________________________________________________________________ ___________________________________________________________________ ___________________________________________________________________ ___________________________________________________________________ ___________________________________________________________________ ___________________________________________________________________

8

Formative Evaluation

Pilot Study (Trainee)

Instructions: Please check off the boxes that apply, and provide additional feedback the ‘other’ section.

3. What three words would you use to describe the program?  _________________

 _________________

 _________________

4. What were your favorite parts of the program? _______________________________________________________________________ _______________________________________________________________________ _______________________________________________________________________ 5. What was your least favorite parts of the program? ________________________________________________________________________ ________________________________________________________________________ ________________________________________________________________________ 6. Did you notice any problems with the sequencing of the program? If so, please explain. ________________________________________________________________________ ________________________________________________________________________ ________________________________________________________________________ 7. Please provide any other feedback you feel would be useful. (You may use the back of this sheet as well). ________________________________________________________________________ ________________________________________________________________________ ________________________________________________________________________

9

Formative Rate YourEvaluation Training

Pilot Study (Instructor)

Instructions: Marking an ‘X’ in the appropriate column indicate whether you strongly disagree (SD), disagree (D), are neutral to (N), agree (A) or strongly agree (SA) with the following statements.

Statement

SD

D

N

A

SA

The one-on-one training notes were logical.

SD

D

N

A

SA

The in-class portion of this learning program could be completed in 30 minutes.

SD

D

N

A

SA

The layout of the instructor package was logical.

SD

D

N

A

SA

I found the quick notes useful.

SD

D

N

A

SA

I visuals for this program were difficult to read.

SD

D

N

A

SA

The large job aid for this program was confusing.

SD

D

N

A

SA

The group adaptation notes were useful.

SD

D

N

A

SA

The program worked better 1 on 1 than in groups.

SD

D

N

A

SA

The layout of this guide was user friendly.

SD

D

N

A

SA

I was confused by the layout of the instructor guide.

SD

D

N

A

SA

The order of sections in this guide was logical.

SD

D

N

A

SA

I found the evaluation descriptions hard to follow.

SD

D

N

A

SA

The program worked better in groups than in 1 to 1.

SD

D

N

A

SA

This guide appeared complete to me.

SD

D

N

A

SA

I was confused by some of the terms in the instructor guide.

SD

D

N

A

SA

The introduction helped prepare me to teach the program.

SD

D

N

A

SA

This guide was missing critical sections.

SD

D

N

A

SA

10

Formative Evaluation

Pilot Study (Trainer)

Instructions: Please check off the boxes that apply, and provide additional feedback in the ‘other’ section.

1. Which sections of the program did you find confusing?  Table of Contents

 Section 1

 Evaluation Prep Page

 Course Info Page

 Section 2

 Worksheets

 Training Prep Page

 Section 3

 Videos

 Safety & Materials Page

 Section 4

 Glossary

Other: ___________________________________________________________________ ___________________________________________________________________ ___________________________________________________________________ ___________________________________________________________________ ___________________________________________________________________ ___________________________________________________________________

2. Which sections of the program did you find well-articulated?  Table of Contents

 Section 1

 Evaluation Prep Page

 Course Info Page

 Section 2

 Worksheets

 Training Prep Page

 Section 3

 Videos

 Safety & Materials Page

 Section 4

 Glossary

Other: ___________________________________________________________________ ___________________________________________________________________ ___________________________________________________________________ ___________________________________________________________________ ___________________________________________________________________ ___________________________________________________________________

11

Formative Evaluation

Pilot Study (Trainer)

Instructions: Please check off the boxes that apply, and provide additional feedback the ‘other’ section.

3. Were there any steps that took more time to complete than stated on the instructor package? Please indicate the page numbers. _______________________________________________________________________ _______________________________________________________________________ _______________________________________________________________________ 4. Where there any sections that took less time to complete than stated on the instructor package? Please indicate the page numbers. _______________________________________________________________________ _______________________________________________________________________ _______________________________________________________________________ 5. Did you notice any issues with the sequencing of the program? If so, please explain. ________________________________________________________________________ ________________________________________________________________________ ________________________________________________________________________ 6. Please provide any other feedback you feel would be useful, including what you liked and did not like about the program. (You may use the back of this sheet as well). ________________________________________________________________________ ________________________________________________________________________ ________________________________________________________________________

12

Formative Evaluation

Editorial Review

What is a Glove Box?

   

A glove box is a large piece of equipment roughly 7’ deep, 14’ long and 6’ tall.

A glove box is used to conduct chemical reactions that cannot be done in an oxygen or water saturated environment.

An antechamber with an inner and outer door helps to transport chemicals and equipment in and out of the box.

While in the antechamber, the oxygen and water can be “cycled out” and the chemicals or equipment can then be transported into the box.

Background Information



Although you may not be familiar with chemistry and/or glove box equipment your observations will help to ensure that the layout, editing and readability of the program are logical and accurate.



Your attention to detail is very important for the success of the program. Please note all anomalies to the best of your abilities.



Should you have any questions, or should you like to follow up further on a particular point, please contact Jessica Haak by e-mail, jessica.a.haak@gmail.com or by phone, 514.589.4251.

13

Formative Evaluation

Editorial Review

Section One: Instructor & Learner Package Accuracy Instructions: Please review the copy making sure to assess each of the points listed below on all of the pages of the program. It is asked that you annotate the document using the “review comments” function in PowerPoint, located within the tools menu.

Additionally please review videos 1, 2, 3 & 4 including the credit sections.

1. Heading mistakes 2. Inconsistent terminology 3. Spelling errors 4. Sentence errors 5. Missing steps 6. Inconsistent procedures 7. Issues with jargon 8. Issues with definitions 9. Issues with poor word choices 10. Cultural issues or biases 11. Issues with pictorial labels 12. Issues with image placement 13. Issues with fonts 14. Issues with layout 15. Issue with colors 16. Issues with legibility 17. Issues with bullets 18. Issues with spacing 19. Other noted problems

14

Formative Evaluation

Editorial Review

Section Two: General Issues Instructions: Please answer the questions below.

1.  Did you find any elements of the program to be incomplete? _________________________________________________________________________ _________________________________________________________________________ _________________________________________________________________________ _________________________________________________________________________ 2. Did you find any major inconsistencies in style or layout? _________________________________________________________________________ _________________________________________________________________________ _________________________________________________________________________ _________________________________________________________________________ 3. Did you find any major issues with the tone of the writing? __________________________________________________________________________ _________________________________________________________________________ _________________________________________________________________________ _________________________________________________________________________ 4. Were there any other critical issues you noted? __________________________________________________________________________ _________________________________________________________________________ _________________________________________________________________________ _________________________________________________________________________

15