0HV100 Human Factors Assignment 1.2:
The coffee vending machine in Metaforum Group number: 1 F.R.A. van Tilburg – s144400 A.L.M. Verhoijsen - s20170185 G.Chen - s20175115
Introduction The purpose of our usability study is to survey users’ satisfaction about the payment and ordering system of the coffee vending machine when ordering multiple drinks in Metaforum TU/e. The problem is that when ordering multiple drinks, the person had to pay and order the drinks separately. This could be irritating and frustrating. To discover the reactions of users on the system, a study was performed. With this study we want to know what students think about the usability of the system, emotions of the students and the time consuming when ordering multiple drinks. Further, a conclusion will be made out if the results.
Method System The coffee vending machines inside all buildings of the TU/e can only sell coffee one by one. Along with this, you pay for each cup separately (for example if a person buy 2 cups of drink he has to pay 2 times instead of once). But many students would like to order several cups of coffee for friends at one time. To test this problem, we used the Wizard of Oz methodology: A cardboard prototype was made to represent the present vending interface. We would use role play to test how end users will interact with the system.
Tasks Participants are asked to order coffee or chocolate with the wizard of oz prototype. The main instruction will be to order some drinks for himself/herself and fellow students during a study day in Metaforum. There are two tasks that participants could perform. Each participant is asked to perform one task, otherwise the ‘Usability Test’ takes too long. This was also done to ensure people were willing to participate in our study.
The tasks purpose is to order a minimum of two drinks to discover the reaction of the participants on the product. The participants need to perform one of the two tasks: 1. Order three coffee on the template of the Coffee Vending Machine 2. Order one coffee and one hot chocolate on the template of the Coffee Vending Machine The tasks were given randomly to each participant. No distinction was made between different participants.
Measurements Our Usability Test uses three tools for measurement (1) a think-aloud protocol, (2) a (standardized) questionnaire (See Appendix 1) and (3) time notation. With the think-aloud protocol we try to check the emotions and behaviour of the participants during the performance of the tasks. The (standardized) questionnaire consist of questions about demographics (gender, age, study etc), a self made question about emotional responses and a standardized question, The System Usability Scale (SUS). The self made question in the questionnaire is a Radio Button choice question about the the emotions of the participants after performing the tasks. With the time notation the time is measured how long it takes to order the amount of drinks on the Coffee Vending Machine. Time notation is a way to measure the efficiency of the machine.
Participants For the usability test conducted we asked 7 people to participate. Since our aim was to look at students during their daily rituals of studying, we went to the library and spoke to the people there. During the selection of participants no distinction was made between age, gender, level of education or ethnicity. Overall, we found a distribution of age with a mean of 22.7 years. Participants were between the age of 19 and 25, so all sorts of students were represented. As seen from the demographic questions, 4 out of 7 (See Appendix 2) were in their masters. This might have influenced the results with regard to studying/working hours. We, however, made no distinction between the level of education when asking the participant to perform a specified task.
Results On the think-aloud protocol and the question about the emotions after tasks, several problems came forward: 1. Coffee-making process waiting too long - The coffee making process takes approximately twenty seconds. One participant felt quite irritated while waiting. The person kept complaining how much time was wasted here. One participant didn’t feel annoyed by the system at all and just waited very patiently. Five participants thought the coffee was made too slow but they said it’s still acceptable. Most of them tried to channel their attention to something else, such as checking cell phones or talking to fellow students. 2. Payment process waiting too long - The payment process takes approximately five seconds. All participants said that this is okay when paying just once. But when the number of ordered-coffee increased to three, they started to lose patience while waiting for another five seconds. Also, the people had to pay multiple times when
ordering the drinks. With TA protocol, we observed that they made weary face while waiting. 3. People felt annoyed after ordering more than 1 coffee- Most people said they felt irritated and annoyed after finishing the task. To be more extreme, some of them even felt anger and rage from the waiting process. Only few of participants felt joy and serenity from the process.
Conclusion Overall the participants thought the usability of the system was good (See Appendix 3: Results SUS). The average of the SUS is 70,2, this means that the systems usability is sufficient for the users. However, the people get annoyed and irritated when ordering multiple drinks. The primary problem about the tasks is that it takes a long time to order multiple drinks. The second problem is that people felt annoyed and irritated after the tasks. In conclusion, the system of the coffee vending machine should not be changed majorly. A minor implementation should be added that people can order and pay once and then receive the drinks.
Appendix 1. Questionnaire Questionnaire Usability Test – The Coffeemachine in Metaforum Hello, what nice you want to participate to our Usability Test. For our course “Human Factors”, we need to perform an Usability Test about a bad Human Facto Design. Our test is about the actions you perform when getting coffee for you and your friends during a long learning day at the Metaforum. First you have to fill in these questions, than the Usability Test starts. After the ‘Usability Test’, a couple questions are asked about the performance. 1.
What is your gender? o Male o Female
2.
What is your age?
……………………………………………………………………………………………………………………… 3.
What is your ethnicity? o White o Hispanic or Latino o Black or African American o Native American or American Indian o Asian / Pacific Islander o Other………………………………………………………………………………………
4.
What is your employment status? o Employed for wages o Self – employed o A homemaker o A student o Other………………………………………………………………………………………
5.
What is your level of education?
……………………………………………………………………………………………………………………… 6.
On a scale of 1 - 5 , how do you feel after the tasks you performed?
1
2
3
4
5
Strongly
Strongly
Disagree
Agree
I feel irritated
Ο
Ο
Ο
Ο
Ο
I feel joy
Ο
Ο
Ο
Ο
Ο
I feel frustrated
Ο
Ο
Ο
Ο
Ο
I feel serenity
Ο
Ο
Ο
Ο
Ο
I feel annoyance
Ο
Ο
Ο
Ο
Ο
I feel anger
Ο
Ο
Ο
Ο
Ο
I feel ecstasy
Ο
Ο
Ο
Ο
Ο
I feel rage
Ο
Ο
Ο
Ο
Ο
7.
The system usability scale (SUS) is a simple, ten-item attitude Likert scale giving a global
view of subjective assessments of usability. What did you think about the system after you using it?
References Questionnaire 1. Bevan, N, Kirakowski, J and Maissel, J, 1991, What is Usability?, in H.-J. Bullinger, (Ed.). Human Aspects in Computing: Design and use of interactive systems and work with terminals, Amsterdam: Elsevier. 2. Kirakowski, J and Corbett, M, 1988, Measuring User Satisfaction, in D M Jones and R Winder (Eds.) People and Computers IV. Cambridge: Cambridge University Press. 3. Pratchett, T., 1990 Moving Pictures. London: Gollancz 4. SUS: A Quick and Dirty Usability Scale by John Brooks 5. Measuring Usability with the System Usability Scale (SUS) by Jeff Sauro 6. SUS: A Retrospective by John Brooke 7. Determining What Individual SUS Scores Mean: Adding an Adjective Rating Scale by Kortum Bangor and Miller May
2. Results
The system usability scale (SUS) is a simple, ten-item attitude Likert scale giving a global view of subjective assessments of usability. What did you think about the system after you using it?
3. Results question System Usability Scale
Using SUS The SU scale is generally used after the respondent has had an opportunity to use the system being evaluated, but before any debriefing or discussion takes place. Respondents should be asked to record their immediate response to each item, rather than thinking about items for a long time. All items should be checked. If a respondent feels that they cannot respond to a particular item, they should mark the centre point of the scale. Scoring SUS SUS yields a single number representing a composite measure of the overall usability of the system being studied. Note that scores for individual items are not meaningful on their own. To calculate the SUS score, first sum the score contributions from each item. Each item's score contribution will range from 0 to 4. For items 1,3,5,7,and 9 the score contribution is the scale position minus 1. For items 2,4,6,8 and 10, the contribution is 5 minus the scale position. Multiply the sum of the scores by 2.5 to obtain the overall value of SU. SUS scores have a range of 0 to 100.