Telstra Assistant – Usability Testing Report by #cursed_baos

Usability Testing Report

Project: Telstra Assistant on Google Assistant Date of Report: 15 May 2019 Location of Test: Melbourne CBD Prepared by: #cursed_baos

Executive Summary

7 participants take part in testing the demo for the Telstra Voice Assistant on Adobe XD. All of the participants were asked to finish 2 activities: check data with Telstra Voice Assistant and pay phone bill through Telstra Voice Assistant. 80% of the participants managed to finish both tasks. Their overall feedbacks are that the Voice Assistant is easy to understand and it is easy to use. But we also ran into some problems, for the first 2 participants, there were some technical problems but we managed to finish it on time.

Methodology

Who we tested 7 participants, with ages, ranging from 18 to 25, evaluated the demo of Telstra Voice Assistant.

What participants did Participants meet and discuss with the developers for roughly 15 minutes each. They were asked to complete 2 tasks with the Telstra Voice Assistant: check data and pay bills. After that, they engaged in casual conversations with the developers and answered questions.

What data we collected We mostly collected the information through visual presentation while the participants were testing. In addition to that, we also asked questions after the sessions to collect more data.

Major findings & recommendations

Major issues The low-fidelity prototype did not include enough voice trigger from users People tend to choose to use the buttons instead of talking Telstra Voice Assistant misheard or takes too long to response Identify solutions Anticipate and add more user inputs and triggers Find alternative payment methods for credit card Improve Telstra Assistant ability to pick up user’s voice

Activity 1 Check data

“You are using a Telstra post-paid plan and want to check how many data you have left on your number: 0400 987 654. You want to try using Telstra Assistant to check your data usage.”

Findings Most of the participants need some kind of prompting.

Though it is mostly due to the limitation of Adobe XD. Dannie thought that it is finished at the step when Telstra Assistant offered to set a reminder.

Quynh’s voice wasn’t picked up by Telstra Assistant so she needs to switch to use buttons, tend to choose to use the buttons.

Activity 2 Pay bills

“It’s almost time for you to pay the bill of your monthly post-paid number: 0400 987 654. You want to try using Telstra Assistant to pay the bill.”

Findings Most of the participants need some kind of prompting.

Though it is mostly due to the limitation of Adobe XD. Truc’s voice did not get any response when transferring from Check data to Pay bill so she needs to tap on the suggestion chips instead.

Dannie tried to press ‘Use password’ when pay bill.

Shazad was confused when he tried to pay the bill through Google Pay using his fingerprint because the Adobe XD system lagged.

Interview Questions Is it easy enough for you to understand what you should do in every step?

All of the participants think that it is easy to use.

They think that it is pretty straightforward, quick to use and user-friendly. A lot of them take likings for the suggestion chips.

Were the answers what you expected?

All of the participants state that it is what they expect. The answer is obvious, clear, concise and pretty throughout in explanation. They also like how the important answers go first.

Did the conversation feel natural?

Even though the Telstra Voice Assistant misunderstands sometimes, it is still generally quite clear. Though a participant says it is too monotone.

Is anything confusing/unclear?

Some participants find the part when you need to take fingerprints to pay the bill through Google Pay confusing. They don’t understand whether they should use their fingers or use passcode.

What is your overall impression of the experience?

Most of the participants think that the experience is good. They think that the Voice Assistant is quite easy and convenient to use. One even gives it 8/10. Though some still say that it is a bit clunky but they think it is still good for a prototype.

What did you like best about the site?

Most of the participants love the suggestion chips because it is much faster than listening and talking.

Participants also say that the Voice Assistant is useful to interact with, the information given is precise and straight to the point.

What did you like least about the site?

Three of the participants think that the Telstra Voice Assistant is talking too slow.

One, in particular, has a significant worry about privacy and thinks that the Voice Assistant’s voice is too monotone.

One thinks that we should make the visual graphics and the texts bigger The rest say that they are satisfied.

Interview Questions If you were the developer, what would be the first thing you would do to improve the experience?

Developers should pay more attention to the privacy issue, make the Telstra Voice Assistant talk faster, add more functions, etc.

Is there anything that you feel is missing?

Most of the participants don’t have anything more to add. They think the Telstra Voice Assistant fulfils all the thing that the Telstra app provides.

Would you be willing to use VA for these activities in the future?

One participant was sure that he would use it as an alternative. According to his opinion, the Voice Assistant is much easier to use, he didn’t have to type in too much information, he can click buttons and automatically link to the account, and sometimes when he doesn’t feel like pressing the button he can use Voice.

The rest of the participants are still considering but are not so sure if they will use it right away. But they still see some convenience in it (faster, take less data, more on the go, etc.)

Are there any final questions?

All of the participants don’t have any more information to add.

Users’ Impressions

King Positive – the low data reminder function Shazad Positive – It is useful, precise, straight to the point.

Negative – Telstra Voice Assistant talks too slow.

Improve – The Telstra Voice Assistant is talking a bit slow. Developers should add more paying options aside from Google Pays.

Dannie Positive – The suggestion chips.

Improve – Should make the texts bigger.

Quynh Positive – The suggestion chips.

Negative – Doesn’t feel real.

Improve – Should consider customer’s privacy.

Truc Positive – Fun and different from the traditional way Negative – The experience feels kind of slow.

Improve – Should be able to answer more questions and pick up customer's voice better.

Mitch Positive – Fulfils everything needed to fulfil and managed to understand what he wants to say.

Improve – Make the whole interface smoother and more efficient.

Add more features (ie. book an appointment with a Telstra technician) Talks a bit too much, should let users choose options immediately to save time.