Formative Evaluations by Jessica Haak

FORMATIVE EVALUATION

EXPERT REVIEW

USABILITY TEST

EDITORAL REVIEW

Review Phase

# of Reviewers

Main Purpose

1. Expert Review

2-3 Subject Matter Experts

Message Accuracy

2. Usability Test

3 users, 1 facilitator

Program Usability/Reactions

4. Editorial Review

2-3 Editors

Program Readability

Arndtsen Group Performance Campaign

Formative Evaluation

Expert Review

Decision-Making to Date

        

A performance needs assessment was conducted in the Winter of 2013 in the Arndtsen research group. At this time I administered a survey to group members that looked at their personal views related to glove box use, policies, systems, supports, incentivizations, training and feedback. In subsequent discussions with the project sponsor (Jeffrey Quesnel) it was decided that many of the issues being faced fell under the umbrella of glove box contamination, both in terms of physical surfaces and atmospheric conditions. From this a campaign that focused on a collaborative approaches to tackling the problem was created. The job one and performance objectives for this campaign are listed below:

JOB ONE To prevent major safety incidents in the glove box, by encouraging an open and cooperative environment for dealing with contamination issues in a continuous and timely manner. MAIN TASKS 1.  Diagnose glove box contamination cooperatively. 2.  Correct glove box conditions using teamwork. 3.  Report glove box experiences openly. 4.  Recognize the benefits of cooperatively, communally, and openly working to prevent contamination issues.

Background Information

  In order to ensure that the content presents an accurate depiction of scientific principles and safety       

standards this expert review has been developed. For the purpose of this review a small questionnaire has been created to direct your attention to important aspects of the project. There is a roadmap for the campaign for you to learn more about the dissemination strategy, nine posters, a video, and an online skill testing question. Your honest and full feedback is welcomed! The more information you provide the more likely your comments will translate into a better campaign for everyone involved, so please don’t be shy! Should you have any questions, or should you like to follow up further on a particular point, please contact Jessica Haak by e-mail, jessica.a.haak@gmail.com or by phone, 514.589.4251.

Arndtsen Group Performance Campaign

Formative Evaluation

Expert Review

I. POSTERS Instructions: Please answer the questions below: 1) Do you think that the images presented on the posters are appropriate for a safety campaign on the topic of glove box contamination? Why or why not? ____________________________________________________________________________________ ____________________________________________________________________________________ ____________________________________________________________________________________ ____________________________________________________________________________________ 2) Do you feel that the information presented on the posters is relevant to the issue of glove box contamination? Why or why not? ____________________________________________________________________________________ ____________________________________________________________________________________ ____________________________________________________________________________________ ____________________________________________________________________________________ 3) Are there any specific graphics or terms used on the posters that you feel are factually inaccurate? ____________________________________________________________________________________ ____________________________________________________________________________________ ____________________________________________________________________________________ ____________________________________________________________________________________ 4) Are there any specific graphics or terms used on the posters that you feel promote a dangerous safety practice in the lab? ____________________________________________________________________________________ ____________________________________________________________________________________ ____________________________________________________________________________________ ____________________________________________________________________________________

Arndtsen Group Performance Campaign

Formative Evaluation

Expert Review

II. VIDEO Instructions: Please answer the questions below: 1) Do you think that the message presented in the video is appropriate for a safety campaign on the topic of glove box contamination? Why or why not? ____________________________________________________________________________________ ____________________________________________________________________________________ ____________________________________________________________________________________ ____________________________________________________________________________________ 2) Do you feel that the information presented in the video is relevant to the issue of glove box contamination? Why or why not? ____________________________________________________________________________________ ____________________________________________________________________________________ ____________________________________________________________________________________ ____________________________________________________________________________________ 3) Are there any specific terms or phrases used in the video that you feel are factually inaccurate? ____________________________________________________________________________________ ____________________________________________________________________________________ ____________________________________________________________________________________ ____________________________________________________________________________________ 4) Are there any specific terms or phrases used in the video that you feel promote a dangerous safety practice in the lab? ____________________________________________________________________________________ ____________________________________________________________________________________ ____________________________________________________________________________________ ____________________________________________________________________________________

Arndtsen Group Performance Campaign

Formative Evaluation

Expert Review

III. GENERAL FEEDBACK Please provide general feedback about what you would keep the same or change about the campaign. ____________________________________________________________________________________ ____________________________________________________________________________________ ____________________________________________________________________________________ ____________________________________________________________________________________ _____________________________________________________________________________________ ____________________________________________________________________________________ ____________________________________________________________________________________ ____________________________________________________________________________________ ____________________________________________________________________________________ _____________________________________________________________________________________ ____________________________________________________________________________________ ____________________________________________________________________________________ ____________________________________________________________________________________ ____________________________________________________________________________________ _____________________________________________________________________________________ ____________________________________________________________________________________ ____________________________________________________________________________________ ____________________________________________________________________________________ ____________________________________________________________________________________ _____________________________________________________________________________________ ____________________________________________________________________________________ ____________________________________________________________________________________ ____________________________________________________________________________________ ____________________________________________________________________________________ _____________________________________________________________________________________ ____________________________________________________________________________________ ____________________________________________________________________________________ ____________________________________________________________________________________ ____________________________________________________________________________________ _____________________________________________________________________________________

Arndtsen Group Performance Campaign

Formative Evaluation

Usability Test

Instructions for the Facilitator   This usability test is designed to have the participants experience the campaign in a simulated format. All three stages of the campaign (exposure to “the problem” posters, “the solution” posters, the video, and the incentivization program) will be simulated over a course of the day in a real-life lab. After each simulated stage, a questionnaire will be provided to the participants.   The remainder of this documents includes schedules of the days events, prompts to be read to the participants and the questionnaires that will need to be distributed. Time

Activity

Materials

9:00-9:15am

Brief participants on the purpose of the test

Prompt 1: Test Overview

9:15-11:30am

Participants enter their lab. They go about their work and observe “the problem” posters throughout the space

“Problem” poster printed and posted in high traffic areas (e.g. by the coffee maker, in the office and/or by the glove box equipment).

11:30-11:45am

Participants fill out questionnaire 1 about “the problem” posters

Questionnaire 1

12:00-2:30pm

“The solution” posters are place beside “the problem” posters

“Solution” posters printed and posted in high traffic areas

2:30-2:45pm

Participants fill out questionnaire 2 about “the solution” posters. Explain to participants that for the next phase they should try to access the online content.

Questionnaire 2

3:00-4:15pm

Giveaway poster put up in the lab with details on how to go online

“Giveaway” posters and live online giveaway with skill testing question.

4:15-4:30pm

Provide participants with prizes and ask them to fill out questionnaire 3

Questionnaire 3

4:30-5:00pm

Debriefing & Refreshments

Prompt 3: Participant Debriefing

Arndtsen Group Performance Campaign

Prompt 2: Instructions on accessing giveaway.

Formative Evaluation

Usability Test

PROMPT 1: Test Overview

  Read to the paragraph below to the participants: “Hello everyone! I am happy to welcome you to the Arndtsen Group Performance Campaign usability test. We are so happy that you are here. This usability test will take place over the course of the day and will involve your viewing content around the lab, going online at one point to view some content and answering three 15 minute questionnaires. Please be at your desk at from 11:30-11:45, 2:30-2:45 and 4:15-4:30 for the questionnaire sessions; otherwise please go about your day as you normally would. Upon the completion of the test there will be refreshments and a debriefing session in the office, from 4:30-5:00.”

  Answer questions, see the FAQs below: -If the participants asks about the content, please only divulge that it is about glove box performance. -If the participant is concerned about the timeframe, you may arrange with them to set change the times, if you are available to administer the questionnaire earlier or later than specified. -If the participants would like to know more info about what a usability test is, let them know that it involves the process of ensuring the content and the execution of a campaign are done without error. -If the participant would like to know more about performance campaigns, let them know that it is like a marketing campaign that encourages and supports works to improve their work performance.

Arndtsen Group Performance Campaign

Formative Evaluation

Usability Test

PROMPT 2: Accessing Giveaway

  Read to the paragraph below to the participants “Thank you so much for filling out questionnaire 2. For the next phase please be aware that you will be given information in the lab about accessing some online content. Please attempt to try to go online and access this content.” PROMPT 3: Debriefing

  Read to the paragraph below to the participants “Again we would like to thank you for your participation in this test! For the purpose of this project we are attempting to improve glove box contamination issues resulting in safety incidents and loss of work.”

Arndtsen Group Performance Campaign

Formative Rate YourEvaluation Training

Usability Test Questionnaire 1

Instructions: Marking an â&#x20AC;&#x2DC;Xâ&#x20AC;&#x2122; in the appropriate column indicate whether you strongly disagree (SD), disagree (D), are neutral to (N), agree (A) or strongly agree (SA) with the following statements. Statement

1. The posters were easy to spot in the lab.

2. I noticed that there were four different posters in the lab.

3. The poster was easy to read.

4. I liked the visuals in the posters.

5. I thought the posters were fun.

6. The message in the posters was clear.

Instructions: Answer the questions below using the space provided. 1) What did you like most about the posters? ____________________________________________________________________________________ ____________________________________________________________________________________ ____________________________________________________________________________________ 2) What did you like least about the posters? ____________________________________________________________________________________ ____________________________________________________________________________________ ____________________________________________________________________________________ 3) Would you change anything about the posters? ____________________________________________________________________________________ ____________________________________________________________________________________ ____________________________________________________________________________________

Arndtsen Group Performance Campaign

Formative Rate YourEvaluation Training

Usability Test Questionnaire 2

1. The second set of posters complimented the other set.

2. I saw a connection between the first and second set of posters.

3. The second set of posters were easy to read.

4. I liked the visuals in the second set of posters.

5. I thought the posters were fun.

6. The message in the posters was clear.

Arndtsen Group Performance Campaign

Formative Rate YourEvaluation Training

Usability Test Questionnaire 3

1. I was able to connect to the online giveaway.

2. I was able to view the online video.

3. I was able to submit my online question.

4. I did not receive details on how to collect my prize.

5. I thought the video was informative.

6. I thought the campaign as a whole was effective.

Instructions: Answer the questions below using the space provided. 1) What did you like most about the video? ____________________________________________________________________________________ ____________________________________________________________________________________ ____________________________________________________________________________________ 2) What did you like least about the video? ____________________________________________________________________________________ ____________________________________________________________________________________ ____________________________________________________________________________________ 3) Did the video change your opinion about any aspects of the campaign? ____________________________________________________________________________________ ____________________________________________________________________________________ ____________________________________________________________________________________

Arndtsen Group Performance Campaign

Formative Rate YourEvaluation Training

Usability Test Questionnaire 3

Instructions: Answer the questions below using the space provided. 1) What three words would you use to describe the campaign? ____________________________________________________________________________________ ____________________________________________________________________________________ ____________________________________________________________________________________ 2) Would you make any changes to the delivery of the campaign. ____________________________________________________________________________________ ____________________________________________________________________________________ ____________________________________________________________________________________ ____________________________________________________________________________________ ____________________________________________________________________________________ ____________________________________________________________________________________

3) Did you have anything else you would like to add? ____________________________________________________________________________________ ____________________________________________________________________________________ ____________________________________________________________________________________ ____________________________________________________________________________________ ____________________________________________________________________________________ ____________________________________________________________________________________ ____________________________________________________________________________________ ____________________________________________________________________________________ ____________________________________________________________________________________ ____________________________________________________________________________________ ____________________________________________________________________________________ ____________________________________________________________________________________ ____________________________________________________________________________________ ____________________________________________________________________________________ ____________________________________________________________________________________ ____________________________________________________________________________________ ____________________________________________________________________________________ ____________________________________________________________________________________

Arndtsen Group Performance Campaign

Formative Evaluation

Editorial Review

What is a Glove Box?

    

A glove box is a piece of equipment used to conduct chemical reactions that cannot be done in an oxygen or water containing environment. Contamination may occur when chemicals are spilled, mishandled, or improperly transported. Contamination can lead to serious safety incidents such as exposure to dangerous chemicals and given that much of the lab’s work is stored in the glove box it can impact research results for the entire group.

Background Information



Although you may not be familiar with chemistry and/or glove box equipment your observations will help to ensure that the layout, editing, and readability of the campaign are logical and accurate.



Your attention to detail is very important for the success of the program. Please note all anomalies to the best of your abilities.



Should you have any questions, or should you like to follow up further on a particular point, please contact Jessica Haak by e-mail, jessica.a.haak@gmail.com or by phone, 514.589.4251.

Arndtsen Group Performance Campaign

Formative Evaluation

Editorial Review

Points to Examine Instructions: Please review all of the materials including the posters, videos, and online content. In a word document submit the errors that you see and reference them by poster number or by video or online headers.

POSTERS 1)  Grammar errors 2)  Inconsistent terminology 3)  Spelling errors 4)  Issues with layout 5)  Issues with colours 6)  Issues with readability 7)  Issues with fonts 8)  Issues with pictures 9)  Issues with spacing 10)  Other noted problems VIDEO 1)  Grammar errors 2)  Inconsistent terminology 3)  Incomprehensibility errors 4)  Issues with music 5)  Issues with cuts 6)  Issues with text 7)  Issues with lighting 8)  Issues with tone 9)  Issues with jargon 10)  Other noted problems ONLINE 1)  Issues with access 2)  Issues with layout 3)  Issues with spelling 4)  Issues with colours 5)  Issues with fonts 6)  Issues with usability 7)  Issues with file sizes 8)  Other noted problems

Arndtsen Group Performance Campaign