Usability Study Report

IAT432 : Design Evaluation

USABILITY STUDY REPORT Assignment 2

jonathan tsang melody chan sung hwa lee simon bong

Executive Summary The Usability Inc. project team hired by Ms. Smith conducted a usability study for the product, which is a service that provides private, secure, online networks of support. The purpose of this study was to find usability problems in the product and to see if the product is able to support the task scenarios given by Ms. Smith. A total of four usability researchers participated in the study. Based on our findings collected from the participant’s feedback and observational usability tests (think-aloud and co-discovery), problems concerning information architecture, consistency of information and visual design, system status, and error prevention are some of the factors where all participants have experienced. For the better usability of the product, some major changes need to be done. Firstly, we recommend making the vault more readily accessible and visible. Secondly, the ‘My Networks’ option needs to be changed to provide a functional use outside of being able to switch networks. Thirdly, the ‘File Drop’ function needs to be given a distinction from other methods of file access. Fourthly, descriptive sub-headings should be added to the functions offered by the product. Lastly, the diction of the product should be re-labeled or redefined to convey information accurately and to reduce navigational errors.

Section 1: Scenario This product is a website that is designed to allow seniors or people who need special care to set up support network with family members, friends, and health care personnel. This product allows users to create schedules, send messages and share documents with other users in a private network.

The purpose of product evaluation is to prove that the system is able to support the user scenarios and ensure the system is sufficient and correct for its usage. Product evaluation reveals the issues and identifies fatal errors within the system, which helps to improve user experience and satisfaction. It also helps the owner to understand whether the system is able to achieve the company’s online goals and create an user-centered system. A successful product evaluation can maximize user satisfaction and help to reduce time and cost in product development.

Section 2: Methodology RESEARCHERS A team of four usability researchers was recruited by Usability Inc. to conduct the usability study. The four usability researchers recruited are senior undergraduate students in Interactive Arts & Technology at Simon Fraser University.

1

METHODS USED Two different methods of observational usability test were conducted in this usability study: Think Aloud and Co-Discovery. Firstly, think aloud is a method for studying mental processes that reflect the user’s perception of task and device representations. In our study, the think-aloud was conducted with one subject, one facilitator and one silent observer/note taker per session. The users were asked to verbalize their thoughts, feelings and opinions while performing a scenario of tasks. Secondly, co-discovery is a method where two users work together to perform a scenario of tasks while being observed. In our study, the co-discovery was conducted with two different subjects (one semi-knowledgeable person and one new user being in charge of interacting with the product), one facilitator and one silent observer/note taker per session. As opposed to using a think-aloud method, users talked about what they are doing naturally and helped each other through difficulties.

SUBJECTS Eight subjects were recruited to participate this usability study. Four of them were our team members (T1, T2, T3, T4) and another four were naïve participants (N1, N2, N3, N4). To better understand our subjects, we gave them a pre-questionnaire. The pre-questionnaire asked the participants on gender, age, education, language preferences, distributed family communication, and their experience with social networking.

According to the pre-questionnaires, five subjects were males and three subjects were females. In terms of age range, all of them were between age 15 to 34. All participants were university students, and their language preference was English. In terms of a number of people in each subject’s household who use social networks, four subjects answered as “less than 2 people” and the rest four answered as “between 3 to 5 people”. In terms of a usage of social networks, six of them answered as “everyday” while one person answered as “more than 3 times per week” and another person answered as “less than 2 times per week” – most of the participants use social networks often. All participants use social networks to keep in touch with family and friends, and some of them also use them to share interesting links. All participants use the social network Facebook the most and some of them commented on Facebook that it is “easy to use” and “easy to navigate”. In terms of communication tools they use, all subjects checked “Mobile” and “E-mail” as their main communication tools.

MATERIALS AND INSTRUMENTS 1.

2.

3.

4.

5.

6.

7.

8.

9.

Product to be tested (with a temporary login) Laptops Papers and pencils Timer (phones) Instruction scripts Pre-questionnaires Observational data sheets Post-task analysis questionnaires Post-questionnaires

2

PROCEDURE The usability test was conducted in a study room at Simon Fraser University in Surrey on February 6th, 2012. In the beginning, we welcomed each participant, introduced ourselves, and told briefly about the product. We provided the participants with the product to be tested and scenarios of tasks (Network Admin, New Member, and Health Care) to perform. We asked the participants to perform the tasks using the product, and explain what they are thinking about while working with the product's interface. While conducting two different observational usability methods (talk-aloud and co-discovery), the facilitator and the silent observer/note taker took notes of the participant’s behavior and key comments, and were required to record the completion of each task as a success, success with prompting or failure. The silent observer also timed the duration of each task. During the test, the facilitator was not allowed to interfere the user while he or she was allowed to give a hint to get going. When each participant had a difficulty in completing a certain task, the facilitator gave the participant a post-task analysis questionnaire after the task is finished. The participants described their thought and experience in completing the task in this questionnaire. They also rated in integers (from 1 to 7) on how easy to complete the task was. When each session is done, the participants were provided with a post-test questionnaire to talk about their overall experience on the product they tested.

Section 3: Summary of Findings NEW MEMBER Post-Task Analysis Summary: The post-task analysis questionnaire for the role of the new member showed that participants found difficulty finding a ‘sign-up’ option on the Company website. Participants commented, also, on the absence of familiar visual graphics, or icons, that they were accustomed to.

Data Sheet Summary: 1.

Find Company invite 2.

Accept the invitation 3.

Participate in the network All of our participants were able to complete the initial task with ease. The second task we assigned had similar success when users were presented with the correct sign-up page.

However, there were cases that arose where participants attempted to create an account through the Company web site, independent of invite, where problems were discovered.

During the third task, the network administration button was easy to locate and users managed to successfully post a public message. The visual cue of having the message window on the bottom of the screen was reported as a positive reinforcement of having successfully posted the network message.

3

SCREENSHOT ILLUSTRATIONS

HEALTH CARE Post-Task Analysis Summary: The post-task analysis questionnaire for the role of the health care provider showed that participants found there was a lack of confirmation routines. Some specific examples include the file uploading process where one participant was more familiar with a subsequent ‘send’ button after choosing a file to upload. In one case, the web site showed errors that deterred the participant from continuing to finish their task.

Data Sheet Summary: 1.

Accept the invitation to contribute to User’s network 2.

Send two documents to User’s Company network 3.

Check if the documents are delivered The first task was simple to accomplish and participants did not encounter any errors.

Participants reported a lack of understanding of the ‘pending’ term used during the file upload phase of the second task. They reported being unsure if their file had been successfully uploaded. One participant gave up on verifying the success of the upload. For the final task, one participant reported confusion due to the lack of email confirmation of files being successfully uploaded, of our participants.

SCREENSHOT ILLUSTRATIONS

4

NETWORK ADMINISTRATOR Post-Task Analysis Summary: Our participants reported some difficulty in understanding the functions of the different option tabs within the Company website. A number of participants required further instructions to complete given tasks. The ‘vault’ and ‘make network admin’ options were reported as being difficult to locate.

Data Sheet Summary: 1.

Login to network 2.

Update profile – Add photo, content, and change account settings 3.

Invite a family member into User’s network 4.

Invite someone to use the File Drop to send files into a Company network 5.

Ensure documents are received into the network The first task was completed, problem free. In the second task, majority of participants defaulted to thinking that the photo upload option would be located within the ‘edit profile menu. The actual option is located under the ‘account’ menu, which was where participants defaulted to next. The third task showed all participants being able to successfully invite a new member to their network. There were comments regarding the language used and how it created confusion as participants confused ‘add’ and ‘invite’ in some cases. Invitation confirmation was another issue of concern, as participants were unable to learn whether their invitations have been successfully sent. The fourth task of file drop had participants reporting difficulty in locating the proper link. Participants navigated to the ‘share file’ buttons. One participant invited the doctor account as a new member. Only one participant successfully completed the task. In the fifth and final task, participants were unable to locate the ‘vault’ area to find their uploaded files. Many participants proceeded to randomly navigate throughout the site in hopes of discovering the ‘vault’.

SCREENSHOT ILLUSTRATIONS

5

Section 4: Problems and Recommendations Subsequent to Section 3 where the findings of the usability study are summarized, this section will outline some of the common problems that the participants encounter and some issues that might require immediate attention in order to increase the usability of the product. The problems outlined in this section are based on the concluded data obtained through participants’ feedback and also through observations made during the study. In addition, general recommendations to solve these problems are also provided in order to improve the user experience while utilizing the product.

1

INFORMATION ARCHITECTURE The Company profile page has a straightforward interface that utilizes tabs on the top for the primary menu and a side menu on the right of the website. The categorization of information helps users navigate through the product by giving them a limited number of choices to select depending on their goals. However, one of the more common problems that users faced during the study was the lack of familiar visual cues or guides that are present in other similar products. Many users pointed out that some of the important links pertaining to the given tasks and functions were not visible right away and experienced difficulty locating them.

This suggests an apparent issue in terms of information architecture, specifically the positioning of the visual elements on the product. For example, as users whose task was to accept the invitation to join as new members, many had problem locating the sign-up page.

Most of the users also experienced difficulty when they were asked to upload a profile photo because intuitively, they would go to the ‘edit profile’ page but failed to locate the option because it was located in the ‘account’ page. Similarly, users who performed the role of network administrators had difficulty locating the ‘Vault’ page where they could check the files that are being shared on their network. It seems that members who are not authorized as network administrators when they are invited do not have access to the ‘Vault’ page. This information could have been conveyed more successfully to avoid confusion for the users.

Our recommendation to resolve these issues is to reconsider the visual hierarchy of the product by prioritizing the important elements by positioning them in a hierarchical fashion and also regroup some of these functions in a way that is familiar to most users. Instead of using text for all the links within the product, consider using icons or graphics to represent some basic functions to enable users to identify and distinguish them from other non-primary functions. Grouping options that are similar together in an intuitive way would also be useful in helping to avoid some of the confusion experienced by users. Example, having a central page for everything profile-related including editing profile photos would be more effective than having the function in the ‘account’ page.

6

SCREENSHOT ILLUSTRATIONS

2

CONSISTENCY OF INFORMATION AND VISUAL DESIGN In terms of visual design, the product appears to be streamlined and has a playful quality to it with the chosen color scheme and the corporate logo. From the study, some users had issues with the tabs on the top of the page mainly regarding their respective functionalities. The most common behavior that we observed was that users repeatedly go to the ‘My Networks’ page when they could not locate something. In some instances, users went directly to that page for tasks that had to do with inviting people to join or to send an invite for someone to share files. However, the only function of that page is to allow users to change to a different network. This is likely due to the fact that the tab for ‘My Networks’ is slightly apart from the others and therefore might suggests to users that it is a central page that contains all the necessary links pertaining to networking and that includes inviting new members or sharing files. Similarly, the ‘Share Files’ option was also confusing to most users because it does not contain the File Drop option, the function to invite non-members to contribute and share files on the network, on the page. Users had the most difficulty locating the File Drop function during the study as reflected on the task rating average.

In addition, some users also pointed the inconsistency in the side menu where clicking on the ‘Help and Videos’ link would activate a drop-down menu as opposed to the other links which direct users to new pages. Moreover, we noticed that users were searching for help and guidance when they were unable to complete a task but most of them did not notice the link for ‘Help and Videos’ immediately but rather searched elsewhere for help. This might suggest that grouping the help link with the other functions could be an issue because the common practice of web design is to set the help option apart from the main menu. While using colors to create contrast thus establishing emphasis on certain elements such as menu is generally a good practice, the colors chosen for the side menu feel a bit arbitrary and lack of consistency. In this case, it created a hierarchy within the list of menu options and users’ attention are drawn to the option boxes with colors that stand out from the others. This could also be the reason that most users were more drawn towards the tabs on top of the page rather than sorting through the side menu.

7

Some recommendations to solve these issues include removing the ‘My Networks’ tab and create a different button or link for it since it is not a function that is used often. Reducing the number of colors used on the side menu bar would increase the consistency and avoid creating visual hierarchy where you do not want it. Using less colors on the website layout would also make the colorful logo stands out more. Also, repositioning the ‘Help and Videos’ link so that it is distinguished from the other menu options would help users locate it more easily when they need help.

SCREENSHOT ILLUSTRATIONS

3 SYSTEM STATUS The current product provides a good amount of feedback to users and it is one of the many important features of the site. Due to the technicality of some of the functions of the product, it is essential that users are aware of that status of the product at all time. Whether they are sending out invitations for family members or inviting health care personnel to share documents, the users should always feel ensured that their actions have been registered in order to avoid delays or miscommunication. In this case, not being able to schedule correctly might have serious repercussions to the users who are expecting help from different sources.

From the study, we found that users experienced problems because of the lack of confirmation for their actions. Some users showed concern about the lack of confirmation when they are sending out invitations for people to join the network. They were mostly uncertain about the status of their action whether the invitations have been successfully transmitted. For the role of health care personnel, when performing the task of uploading and sharing files on a network, users questioned the ‘pending’ status that was shown for files that have been uploaded. Most of them were looking for a ‘Send’ button because they commented that the common practice for uploading files with other similar products is to allow users to select the preferred files to upload and follow by a confirmation action such as ‘Send’ or ‘Done’. This suggests that there is a lack of secondary confirmation that users are familiar with from their own practices using different products. Users also noted that the email confirmation function did not work, as they never received any email confirmation about the status of the files that they had uploaded.

8

We recommend that a standard confirmation message be implemented and displayed after invitations have been successfully sent out so that users are reassured that their actions have been registered and avoid any uncertainty in the process. As for uploading files, users could really benefit from a secondary confirmation so that it allows for error capturing, in case the users selected the wrong file and wanted to change, by simply creating a two-step process; select files to upload and confirming the action. Another possible change is to amend the status message that shows ‘files are pending’ to something more intuitive and easy to understand for users such as ‘Two files have been successfully added to the network’ followed by ‘Your files will appear on the network within an hour’. Users will feel more confident about their actions and reassured that the files will appear on the network.

SCREENSHOT ILLUSTRATIONS

4 ERROR PREVENTION In general, the Company’s product has a high tolerance of errors or mistakes being made by users. When users make an error, the product would deal with the situation by directing the users to a page that explains the type of error and suggesting the users to return to the previous page. However, we found that although the product does a good job in capturing errors, it could benefit from taking steps in preventing errors in the first place. For example, during the study, a user experienced frustration because of making too many errors while trying to sign-up as a new member and one user gave up on completing the task of sending documents to the network because of the errors that the user encountered such as the inability to locate the specific function or selecting the incorrect links. The reason that most users had difficulty completing the task correctly without running into errors could possibly be related to the myriad factors previously stated in this section. Factors such as lack of confirmation, inconsistencies in the visual design and issues related to the information architecture exacerbated the problems that users experience in terms of encountering errors while using the product.

In order to address this issue, we recommend taking necessary steps to prevent or minimize the chances of users making common and fatal errors. Providing help and guidance in an effective way could help users navigate smoothly through the product. Also, implementing a way for users to quickly correct any mistake can also increase the level of satisfaction while using the product. For example, incorporating a ‘go back’ or a ‘do over’ option that would enable users to return to their previous state before making the mistake can help users identify the problems and avoid repeating the same mistakes.

9

Section 5: Suggested Improvements As stated in the previous section, some of the observed issues are more severe than others in terms of affecting the overall usability of the Product. In this section, we are going to suggest 5 improvements that would address some of the more pressing issues based on the study findings. We believe that by implementing these changes, the usability of the product would be significantly improved and this would ultimately enhance the user experience.

1

Restructure the ‘Vault’ by making it more readily accessible and visually dominant. The term vault conveys a sense of security that can be vital to ensuring the users feel safe having their personal, and private, documents stored in that given space. This secured file sharing space is a fundamental tool that gives the product practical functionality. It should simultaneously give users a sense of security, and be obvious and intuitive enough in its usage to make it efficient.

2

The ‘My Networks’ option needs to be changed to provide a functional use outside of just being able to switch networks. A proposed central page where users could find various options, such as profile management, network settings, and other such things would make the rationale of the ‘My Networks’ existence much stronger. It would also eliminate the kind of confusion that participants experienced during the study.

3

The File Drop function is the next important change that would need to be addressed. It is again, difficult to locate, despite being an essential function of the product. It also needs to be given distinction from the other methods of file access, storage, and sharing that is offered by the product. We propose that a everything-file related central page be created where users could find files that are uploaded to the network, share files, and also invite non-members to upload files.

4

Descriptive sub-headings should be added to many of the functions. This would aid greatly in the efficient reduction of navigational-based errors regarding the product. A simple description of what the function does, or what it is commonly used for, could be strategically placed beneath the function itself. We also suggest implementing the existing ‘Help and Video’ option more efficiently to ensure that users utilize it to help them complete their tasks.

5

Product diction should be adjusted to better convey information accurately to users. This would improve the understanding by users of many of the products promoted functions. It is important for users to fully understand the benefits that are provided to them by the product, otherwise they will not be able to fully use and maximize its capabilities. By stream lining and consolidating the language used, user experience and efficiently can be improved. This can be done through investigating and researching what are the most common key terms and words associated with a particular task or function. By having the right tool for the job, errors and confusion can be minimized. Using the term ‘sign-up’ to register new members over ‘free trial’ is one such example. Consolidation of actual product functions can also be optimized, as currently there are a number of differing functions that, at glance, can appear ambiguous in use and function, for example, the file handling tools of the product.

10

Appendix 1: Comparison of Different Techniques Two different methods, think-aloud and co-discovery, were utilized for this usability study that our team conducted. The think aloud method required users to verbalize their thoughts as they completing the given set of tasks and observers took note and observed users’ actions and their thoughts about the system. In contrast, the co-discovery method required users to work as team of two where 1 is a naïve user and the other a user who is familiar with the system. The reason behind this particular method is to create a scenario where the naïve user would feel more comfortable under the impression that they are working with another user thus encouraging discussions and conversation. The observers would take notes of the conversations between the two users. Although our team is unable to recommend one method over the other without knowing the context and nature of the study, we believe that each method has its own advantages and disadvantages.

The former method, think aloud, is inherent less natural because people usually do not speak aloud in their daily lives. Encouraging participants to verbalize their thought process could be challenging and requires practices and experiences on the part of the facilitator to be able to do so effectively. Since most of the data are obtained through the participants’ willingness to do so, it is essential that the facilitator keep encouraging the participants to speak out loud.

One of the cons of this method is that participants might feel unnecessary pressured and refused to actively participate. The data obtained through this method of study is usually more qualitative as the participants give their opinions and their feelings about the system while they are interacting with it. The data is also more subjective as most participants would describe their thoughts when they encounter problems or if they feel like some elements within the system are ambiguous or that they could be implemented differently. With this method, I find that it is very dependent on the participants to give us valuable data because most participants for our study were speaking aloud on the process and not so much the problems that they encountered. This could possibly be because the participants wanted to find the solutions to the problems that they face rather than revealing them.

In contrast, the co-discovery plays on the psychological aspect of two people interacting with each other and reducing the presence of the testers. This is important because it reduces the stress and anxiety that are caused by the notion that the study is a “test” for the users. By pairing an expert with the naïve participant, we create a natural and less formal setting as if the participants are friends having a discussion about the system that is being tested. The data obtained with method is also generally more qualitative rather than quantitative because most the observations are focused on the prompted conversation by the expert counterpart.

However, we believe that this method could produce a richer data because of the ability to probe the naïve participants right away when the experts notice some difficulties experienced by the participants.

Compared to the doing a heuristic evaluation, a usability study is definitely requires more effort to prepare and design the study. The results obtained through user participation are also more reflective of its intended use context since the selected participants are representative of the target users. With that being said, working with users introduces a range of issues because of the unpredictability of participants’ behavior. The testers would have to be flexible and accommodate the participants’ needs and to make sure that the study adheres to standard ethical protocol.

11