Usability Study & Findings HCID 530 A Megan Wilson / Siyi Kou / Luis Gamboa-Sierra
Table of Contents 01 TASKS OVERVIEW 02 PARTICIPANTS PROFILE 03 GENERAL FINDINGS 04 ISSUES & RECOMMENDATIONS 08 ANALYSIS OF POST-QUESTIONNAIRE 12 NEXT STEPS
HCID 530 A Megan Wilson / Siyi Kou / Luis Gamboa-Sierra
Tasks Overview For this particular study we intended to uncover specific problems with the way users interact with Siri and the information they receive from the weather app in relation to the three different tasks we asked them to complete.
TASK 1
TASK 2
TASK 3
The first task was focused on prompting the participants to ask Siri about the weather in their non-preferred unit of temperature (Celsius or Fahrenheit). This task is included, because different cultures use different measurement of temperature. When gardening together, gardeners need to communicate the temperature in a way that makes sense to the both of them, regardless of their preferred unit of temperature. We wanted to find out how participants will ask Siri to receive that specific information and how comfortable they felt about completing the task.
For the second task, participants asked Siri about the weather three hours from now. This task was included because gardeners need this information to plan their gardening activities. For example, gardeners need to know when it is more appropriate to water their plants during the day. We wanted to see how people will ask this question, and if they think the answer they received was satisfactory to their inquiries.
For the third task, participants asked Siri about the wind. This task was included because the wind speed greatly affects plant growth, such as pollination and general health. We wanted to see how people asked Siri to perform this task and how satisfied they were towards the answers that they received.
To succeed in this task, participants need to receive the weather forecast for three hours into the future for a specific day.
To succeed in this task, participants get Siri to display the wind information or say it aloud.
To succeed in this task, participants will be provided with the accurate weather forecast using the unit of temperature they are not familiar with.
HCID 530 A Megan Wilson / Siyi Kou / Luis Gamboa-Sierra
PAGE 01
Participants Profile We required five participants for our study. Ideal participants needed to be Apple iPhone literate. It was imperative that participants had experience using voice recognition softwares (such as Siri, Amazon Alexa, and etc.), although it was not required that they had regularly used one in the past. Participants were also required to have previous experience checking weather with their smartphone.
HCID 530 A Megan Wilson / Siyi Kou / Luis Gamboa-Sierra
PAGE 02
General Findings All of our participants are familiar with Siri, whether they use it frequently or not. Most of them use the weather app for their personal use rather than checking the weather for their plant. Siri seems to be designed in a similar fashion; it only provides information for daily use.
TASK 1
TASK 2
TASK 3
All participants successfully completed this task and received information about the weather.
All participants were able to successfully complete this task. Many expressed that they had not done this before, but they generally did not expect much difficulty in asking this question.
Participants felt a little strange when we inquired them to ask Siri about this particular question. However, all participants were able to finish the task successfully. Most of them were surprised to find out that Siri provided them with an answer about the wind.
We identified that a majority of the participants were comfortable with the question. They thought it was a common question and Siri should know the answer. Most participantes mentioned that Siri was able to provide a quick and clear answer.
“
I can think of what I am going to say, seem like a easy command
They were mostly happy with the result Siri pulls out. However, 3 out of 5 people said that they did not think the result produced was as precise as they would expect. It only provided a “quick answer.”
” “
HCID 530 A Megan Wilson / Siyi Kou / Luis Gamboa-Sierra
I don’t think it is precise on the answer, but it is there
Many expressed that they expected Siri to provide a wikipedia page about the wind and its definition. Many expressed concern that if they were looking for a more general understanding on the term wind, they would be disappointed.
” “
I think siri can grasp the keyword very easy. The problem is the discrepancy between my cognition and Siri. We don’t think the same thing
”
PAGE 03
Issues & Recommendations This report utilizes color coding to indicate the severity of the issues:
CRITICAL ISSUE MODERATE ISSUE
The severity of the issue was determined based on two factors: 1. How many participants experienced the issue. 2. How difficult it was for partcipants to recover from the issue.
HCID 530 A Megan Wilson / Siyi Kou / Luis Gamboa-Sierra
PAGE 04
Issues & Recommendations TASK 1 Siri failed to respond to participants, didn’t hear their complete answers, or couldn’t pick up the their voices if there was external noise. This created very unnatural and uncomfortable interactions for our participants when trying to find out information about the weather in the opposite prefered unit of temperature. Participants generally expected difficulties when it was the first time they asked Siri a question, or if they did not have a clear question formulated in their mind.
RECOMMENDATIONS: 1.1 One recommendation for this issue is to fix issues within Siri’s programming. 1.2 The second recommendation is to allow users to customize information from the weather app that they want to use for gardening to insure more detailed information is offered by Siri. 1.3 Another recommendation is improving Siri to hear better, understand inarticulate words, and hear incomplete sentences. Siri should also be able to remember past questions and make inferences from these past questions. Siri should also ask the user for conversation direction. When in confusion, she should ask the user, “Did you mean x or y?” or “Are you referring to x?”
HCID 530 A Megan Wilson / Siyi Kou / Luis Gamboa-Sierra
PAGE 05
Issues & Recommendations TASK 2 Many participants had higher expectations for the information they would receive about a specific weather condition. Participants wanted more defined answers rather than simplified ones. Participants hope for more visual representation of the weather condition and for these displays to be dynamic.
RECOMMENDATIONS: 2.1 The weather app needs to be reformed to give more complex answers. Siri should explain in words a more complex weather report, rather than have the user look it up in the weather app. This could also be a Siri/Weather app preference. 2.2 The visual data that Siri provides could be more graphical. For example, when asking about rain, it could show the possibility of raining through an infographic, rather than only through symbols.
HCID 530 A Megan Wilson / Siyi Kou / Luis Gamboa-Sierra
PAGE 06
Issues & Recommendations TASK 3 Participants found that the third task was worded incorrectly. They also were not satisfied with the answers given. The third task asks users to “Ask Siri about the wind.” The answer to this question is usually a response of “It’s going to be windy today.” This issue asking about “the wind” is improper English and uncomfortable for speakers. When asking about the wind, Siri will only give the current wind speed, and display the temperature for the following hours. However, no other wind information is provided verbally or visually.
RECOMMENDATIONS: 3.1 A recommendation for the this usability issue has to do with Siri’s programming. She must be programmed to recognize natural English sentences regarding the wind. She does not often recognize when you say windchill in a sentence. She interprets this word incorrectly. She also does not recognize the words windchill or wind speed with the same frequency as wind. 3.2 Another recommendation is to provide more detail voice information along with some graphics, or more descriptive icons to make it clearer for users.
HCID 530 A Megan Wilson / Siyi Kou / Luis Gamboa-Sierra
PAGE 07
Analysis of Post-questionnaire We were able to identify that most participants are satisfied with how Siri provided information that they asked and how this can assist them while gardening. All participants were satisfied with how easy it was to complete all the tasks.
All participants were satisfied with the amount of time it took to complete each task in this usability test.
Strongly Disa...
0 (0%)
Strongly Disa...
0 (0%)
Disagree
0 (0%)
Disagree
0 (0%)
Neutral
0 (0%)
Neutral
0 (0%)
Agree
4 (80%)
Strongly agree 0.5
1
3 (60%)
Strongly agree
1 (20%)
0
Agree
1.5
2
2.5
3
3.5
4
60% of the participants found that Siri is intuitive and easy to use.
2 (40%)
0
0.5
1
1.5
0 (0%)
Strongly Disa...
Disagree
0 (0%)
Disagree
1 (20%)
Neutral
1 (20%)
2 (20%)
Agree Strongly agree 0
0.5
1
1.5
2
2.5
HCID 530 A Megan Wilson / Siyi Kou / Luis Gamboa-Sierra
3
3.5
4
3.5
4
2 (40%)
Strongly agree
0 (0%)
3
0 (0%)
Agree
3 (60%)
2.5
60% of the participants felt very confident determining the voice commands to find information from the weather app.
Strongly Disa...
Neutral
2
0
1 (20%) 0.5
1
1.5
2
2.5
3
3.5
PAGE 08
4
Analysis of Post-questionnaire 60% of participants think that most users would quickly learn to find weather information with Siri.
80% of participants agree that the voice interface of Siri is pleasant.
Strongly Disa...
0 (0%)
Strongly Disa...
0 (0%)
Disagree
0 (0%)
Disagree
0 (0%)
Neutral
0 (0%)
Neutral
1 (20%)
Agree
Agree
3 (60%)
Strongly agree
Strongly agree
1 (20%)
0
0.5
1
1.5
2
2.5
3
3.5
0 (0%)
Strongly Disa...
Disagree
0 (0%)
Disagree Neutral
1 (20%)
0
1
1.5
2
1.5
2
2.5
HCID 530 A Megan Wilson / Siyi Kou / Luis Gamboa-Sierra
3
3.5
2.5
3
3.5
4
0 (0%) 1 (20%) 0 (0%) 2 (40%)
Strongly agree
1 (20%) 0.5
1
Agree
3 (60%)
Strongly agree
0.5
Most participants (80%) would use or recommend Siri to other gardeners in the future to get weather information.
Strongly Disa...
Agree
1 (20%)
0
4
60% of participants agree the natural language interface that was clear and precise.
Neutral
4 (80%)
4
0
2 (40%) 0.5
1
1.5
2
2.5
3
3.5
PAGE 09
4
Analysis of Post-questionnaire We were able to use the post-questionnaire answers to confirm the issues we uncovered during our analysis of the data obtained from the three different tasks. The following is a list of participants’ answers from the post-questionnaire survey. We have highlighted words of phrases that we believe are key points in solving the issues or describe things that went well. Overall, how was your experience with using Siri?
If you could change a feature of Siri, what would you change?
• “Went well, incomplete information” • “Surprisingly good, Siri understood the commands” • “Awesome, Frustrated at the beginning but Siri provided what I needed” • “Pleasant” • “Not bad”
• “Offered Keywords to help users - confirm with users” • “Stop interrupting while talking” • “Include more graphs or images. E.g. it could show icons that combines clouds and wind to be more visual” • “Offer more information than just the weather” • “More filters before it provides me with an answer”
Was Siri useful in assisting you with weather information before or after any gardening activity?
What aspects of Siri do you find satisfying or what works well and why?
• “Useful to get the weather while gardening” • “Faster than typing “Weather” in google” • “Yes, I need to know information before I go outside to work on my garden” • “More specific information” • “It is quick to get weather info before gardening”
• “The time of response is quick” • “It understands any commands and presents relevant information” • “Very precise even when I’m not an native English speaker” • “The ability of easily understand me” • “The precision of recognizing what I asked”
HCID 530 A Megan Wilson / Siyi Kou / Luis Gamboa-Sierra
PAGE 10
Analysis of Post-questionnaire What are you impressions of the voice interface, the weather information feedback, and the time of response based on of your interaction with Siri? • “Voice interface is pleasant, the weather info is easy to understand, and the time of response is fast” • “Overall the interaction went well” • “Voice interface is precise--but lacks human touch, the feedback was ok but I would’ve expected more detail information, and the time of response was fast” • “Great voice interface, the feedback is based on a third party app, and the time of response is convenient” • “All good”
HCID 530 A Megan Wilson / Siyi Kou / Luis Gamboa-Sierra
PAGE 11
Next Steps Improve the voice-recognition function of Siri, especially in situations where assimilation and coarticulation are common. Develop natural language software with the ability to remember previous questions/statements and make inferences from those previous questions/statements. Develop customizable add-on functions for Siri’s weather forecast. Allow Siri to provide more vocal information. Siri should be able to speak the answers to the specific question asked, rather than just give graphical information. For example, when asked about the future temperature in the day, it should at least give the temperature verbally. Design more explicit infographics for the weather information.
HCID 530 A Megan Wilson / Siyi Kou / Luis Gamboa-Sierra
PAGE 12