Usability Evaluation Report

Final Report – Usability Test Plan & Analysis August 8th, 2019 ────

Cynthia Chong Jay Kietzman Phil Traini Sahana Shanthigrama 1

Table of Content I. II.

Page No.

Executive Summary

3

Method

4

A.

Procedure

4

B.

Evaluation Measures

5

C.

Participants

7

III.

Test Result

8

IV.

Findings &Recommendations

9

V.

A.

Findings

11

B.

Recommendations

12

Consent form/ Information Sheet Script (Moderator’s greeting) Task List Pre-test questionnaire Post-test questionnaire Evaluation Notes Data Logs and Recommendation mockups In-depth Tasks Log Sheet Member Contribution Working Schedule

13 16 18 18 19 21 22 32 58 59

Appendix A. B. C. D. E. F. G. H. I. J.

2

Executive Summary After initial heuristic evaluation of the website SVRN (https://svrn.com). Our group chose to create a test plan for the website SVRN (https://svrn.com). Our team created user profiles/personas, as well as brief explanations of our test design and evaluation measures. We formulated research questions with subsequent tasks that addressed the issues we found during heuristic evaluation of the website.

For our testing, we recruited 4 participants through personal connections, who were interested in online shopping or had prior online shopping experience. For our testing we provided various forms that were utilized during the testing process i.e. consent forms, scripts, questionnaires, and log sheets. Participants were asked to perform five tasks based on our research questions.

1. User accessibility (aka action items, utilizing features, ease of use) 2. Layout and flow (are the user able to navigate tasks without confusion or surprise) 3. Navigation accessibility and Error prevention (can the user “go back” is there a “way out “in case of a problem) 4. Search functionality (can the user find what they want when they want) 5. User affordances confirmed through actions (do they receive confirmation upon completion of an action) During testing, participants were asked to think aloud while answers were audio recorded and notes are taken down. The report results showed that all 4 participants completed the tasks successfully but had difficulty in performing a few tasks, as it showed that the site was easy to learn but lacks intuitive organization and clear messaging to make the user’s experience efficient and effective. Most performance errors in our usability testing were a result of a lack of navigation messaging, unclear buttons/placement of buttons, approach to content organization, and search bar functionality issues and newsletter signup confirmation issues. We looked at the process at which each task was completed and if the process could be accomplished more efficiently with fewer errors, be it from omission or commission. Usability testing helped us to find out these issues that users face while using the site and we came up with a list of recommendations. Our findings had multiple design implications for interface-based solutions that can resolve those issues to increase the overall user experience with the website. Henceforth, our recommendations included availability of size chart option, Item stock availability notifications, creating distinct back button and/or home button, optimized search engine results, item filter options, newsletter confirmation acknowledgment, clear buttons/placement of buttons and uniform button style.

Since this project was limited to demographics, the shortcomings were scenario-based observation, sample size, and time constraints. In future iterations, we would require responses from a large sample of participants who already are using svrn website (since all our tested participants are novice user for svrn website) to understand their perspective and thought processes when interacting with the website in their daily life for creating a greater interaction.

3

Method This usability study was a summative study on the effectiveness of the website SVRN.com, the online store and presence of SVRN, a Chicago-based fashion boutique. Each participant (4) will be prompted with five tasks, in differing orders, that aim to inform our core five research questions.

We will be collecting data on success rate, error, and qualitative data on participant’s experiences with the site.

Within-subjects design In this within-subjects design each participant (4) will be guided to perform five tasks, each in a different order to counterbalance a learning effect. We will be conducting a minimum of four individual 30-minute usability testing sessions. All participants will be performing the same tasks and responding to the same qualitative questionnaire. We will be using 15 minutes of the session to introduce the assessment, review basic background information, and administer a post-test debriefing interview. The middle 15 minutes will be utilized to facilitate and observe our users as they work through our five user tasks.

Procedure Session outline and timing The testing sessions will be 30 minutes long. We will be using 15 minutes for pre-test introductions and post-test debriefing interviews. Each session will be held in a different location, but all sessions will be conducted in a quiet, private space, free of distraction or influence on the participant.

Pre-test arrangements Have participants: •

Review and sign the consent form and recording permissions (appendix I)

•

Fill out the background questionnaire (with the same questions as the screener) (appendix II)

Introduction to the assessment (2 minutes) (appendix II) Discuss: •

Participants prior experience with usability testing

•

The value of the user’s participation in the usability testing

•

Moderator’s role

•

The use of recording systems, set up of the testing space, etc.

•

Ask participants to think out loud throughout the entire testing 4

Background Interview (3 minutes) (appendix IV) Discuss with participants: •

Previous experience with online shopping

•

Motivations for and potential concerns with online shopping

Tasks (15 minutes) (appendix III) Each participant will be prompted with five tasks, in different orders, to inform us on key research questions about the usability of the site.

Post-test debriefing (10 minutes) (appendix V) •

We will be asking broad questions to collect qualitative and preference data about the user’s experience

•

Follow up on unique problems the participant experienced during their testing

Evaluation Measures We will use both qualitative and quantitative measures to assess the effectiveness and usability of our website, svrn.com. We have compiled five core research questions, stated below, and metrics of performance and preference (listed below the research questions) to focus our usability testing.

This user testing data will be collected, analyzed, and distilled to inform our recommendations for design alteration and improvement. They will also be reported as benchmarks for future iterations of validation testing.

To answer these questions: 1. Is there a better navigation that could be accessible for the user? Does the user face any difficulty when navigating the page, does website action items/ features help user clearly navigate the website, in other words can the user do what they want to do easily?

2. Does the layout and flow of the website reflect the user’s expectations. Is there anything regarding the interface that surprised/ confused the user?

3. Does the user have the proper means to navigate back to the home page, if not what methods did, they use to get back to this page?

4. Does the search bar function adequately enough that the user is able to specifically search for an item of their choice? If not, how did they find the item they were looking for?

5. Does the website provide proper validation when performing a multi-step task? Do they receive notifications upon completion, do they receive notifications if something goes wrong?

5

We will collect both performance and preference data during our sessions.

Performance: •

Number of errors o

Errors of Omission

o

Errors of Commission

•

Number and percentage of tasks completed correctly with and without prompts or assistance

•

Time taken to complete the task

•

Setting Benchmarks for future site analysis

Preference: •

Usefulness of labeling and messaging from site to user

•

Appropriateness of sites function’s to user’s tasks

•

Ease of use overall

•

User Satisfaction with the site

•

User Goal achievement

•

User suggestions and feedback

During our user testing we will be using two data logs (appendix VI.). Log I will be use to take note of issues, observations, and notes on preference and qualitative insights. Log II will be used to record performance metrics. Together, logs I and II will be used, with our audio recordings, screen recordings, and questionnaires to inform our data analysis, reporting, and final recommendations.

Participants For our testing, we recruited 4 participants through personal connections. Our four participants were 18 or older who had online shopping experiences and shop at least once per month. All of them like to shop for shoes, clothing, or accessories through online websites, such as Zara, Nordstrom, Nike, and Amazon. None of them heard of the website svrn.com before, so they can help us evaluate the site objectively.

6

Table: information of interviewed participants No.

Age

Sex

Occupation

Shop Online/month

1

34

Female

Children’s Center Director

11-20 times

2

27

Female

Student

1-10 times

3

32

Male

Software Engineer

7-8 times

4

26

Female

Bartender

1-10 times

Figure 1: participants’ performance table

7

Test Results Based on our usability testing results we found that, although the site was easy to learn, it lacks intuitive organization and clear messaging to make the user’s experience efficient and effective.

Most performance errors in our usability testing were a result of a lack of navigation messaging, clear buttons/placement of buttons, approach to content organization, and search bar functionality.

(Refer : Raw Data ) Results by task 1. Is there a better navigation that could be accessible for the user? Does the user face any difficulty when navigating the page, does website action items/ features help user clearly navigate the website, in other words can the user do what they want to do easily?

Associated task: Add selected item to their cart and remove selected an item from their cart Results: The time it took our participants (4 users) to complete task 1 was between 15 and 50 seconds, with an average of 27.25 seconds. Three of our four participants completed this task in under 30 seconds, with participant 3 finishing in 50 seconds.

Participant 4 had two performance errors of commission and participant 3 had one performance errors of omission. This task (1) had an error-free rate of 50%. The error of omission resulted from a lack of clear size chart and availability options prior to adding an item to the cart. The errors of commission were a result of a lack of clarity in where to click to remove an item from the user’s cart. The user attempted two wrong locations before clicking the correct place on the page.

User Quote: “How should I know If M size will fit me or not, when they don’t have a proper size chart Information” - Participant 3 2) Does the layout and flow of the website reflect the user’s expectations. Is there anything regarding the interface that surprised/ confused the user?

Associated task: Navigate the website to find a particular designer’s brand lineup.

8

Results: The time it took our participants (4 users) to complete task 2 was between 9 and 31 seconds, with an average of 18.5 seconds. Three of our four participants completed this task in 15 seconds or less, with participant 3 finishing in 31 seconds.

There were no performance errors from our participants while navigating this task. One user did first attempt to use the search bar to complete the task. This task (2) had an error-free rate of 100%.

User Quote: “It feels like the organization of the site is hurting its branding. Like I don’t really know what they are trying to do. I can’t see their niche.” -Participant 4 3) Does the user have the proper means to navigate back to the home page, if not what methods did, they use to get back to this page?

Associated task: After viewing a product navigate back to the home page.

Results: The time it took our participants (4 users) to complete task 3 was between 6 and 33 seconds, with an average of 17.41 seconds.

There were three performance errors (of omission) from our participants while navigating this task. Participant 1 used the back button to navigate to the main page. Later she noticed that the banner returns to the main page. Participant 2 couldn't find a back button on the website but was able to navigate back to the main page by clicking the logo. Participant 3 clicked the website name (logo) because there was not a home button available. This task (3) had an errorfree rate of 25% User Quote: “I don’t find any back button to go back!” -Participant 2 4) Does the search bar function adequately enough that the user is able to specifically search for an item of their choice? If not, how did they find the item they were looking for?

Associated task: Use the search function to search for a specific item of your choice.

Results: The time it took our participants (4 users) to complete task 4 was between 21 and 55 seconds, with an average of 35.5 seconds.

9

Two out of our four participants (50%) could not complete this task without assistance.

Participant 1 had one error of commission and one error of omission. Participant 2 had one error of commission and two errors of omissions. This task (4) had an error-free rate of 50%.

Although 50% of the participants did complete the task without error, 75% expressed frustration with the search bar’s functionality.

User Quote: “I just searched for Vans and all that came up was one pair of shoes and a jacket.” -Participant 4

5) Does the website provide proper validation when performing a multi-step task? Do they receive notifications upon completion, do they receive notifications if something goes wrong?

Associated task: Sign up for the website’s newsletter.

Results: The time it took our participants (4 users) to complete task 5 was between 15 seconds and 1 minute and 34 seconds, with an average of 39 seconds. Three of our four participants finished this task in 30 seconds or less, with participant 1 finishing in 1 minute and 34 seconds.

One participant (2) had one error of omission because the participant assumed that the signup section would be located in the top right, so she looked in the wrong location. It took time for her to scroll down and find the newsletter section. Two users also pointed out that the newsletter sign up did not provide confirmation messaging for the user. This task (5) had an error-free rate of 75%.

User Quote: “I think I did it. Should I try again? I think it worked, I’m not sure” -Participant 1 For additional information on task performance and metrics see our in-depth tasks log sheets in the appendix (section I). We stored the data on a google drive with spreadsheets then we refined and edited the data for presentation in excel.

10

Findings & Recommendations Findings Based on our test results, we have found 4 high priority issues and also listed our recommendations based on high to low priorities.

1. Due to lack of clear size chart and availability of product notification option prior to adding an item to the cart made two participants click on non-active tab i.e., (size not available tab).

Participant 4 was confused on where to click to remove an item from the user’s cart. The user attempted two wrong locations before clicking the correct place on the page.

2. All 4 participants had trouble finding a back button or home button to navigate back to the home page. Participant 1 used the back button on the browser to navigate to the main page. Later the user realized that the banner returns to the main page. Participant 2 and 3 couldn't find the back button on the website but was able to navigate back to the main page by clicking the website name (logo).

3. Three of our four participants found the search engine not showing relevant results for the search they made. Users were also frustrated because there were no filter options available to find specific items. While Participant 1 was searching for a shirt they expressed frustration because it seemed to only bring up men’s shirt and the user had to clarify women shirt in the search. Participant 2 searched “dress “in the search engine but there were no search results and showed only a skirt section. Participant 4 searched vans but a jacket showed up in the results, as there was no option to filter items for what the user was looking for.

4. During the newsletter signup task (5) participant 4 pointed out that the newsletter sign up did not provide a confirmation message to show if the sign up was successful or not. Participant 1 was frustrated as user was provided with captcha after entering email address and the site did not provide any acknowledgment to show if was successful. Participant 2 wasn’t able to find the newsletter sign up in the first place because the location of the newsletter signup was not clearly stated on the site.

11

Recommendations Based on participant’s feedback and the issues found during website svrn testing, we ranked our recommendations based on the issue severity. (Figure: Recommendation Mockup) 1. Search engines needs to be optimized to provide relevant search results.

2. Make important messaging (i.e. stock availability notifications) pronounced and located appropriately so users are made instantly aware.

3. Pick up the right font size and standard design format that matches brand identity and follow platform conventions.

4. System should provide item filter options for user to navigate site and find a specific item.

5. Create distinct back button and/or home button to navigate back to the home page.

6. System needs to optimize newsletter confirmation acknowledge in a better visual way.

7. Provide a better remove buttons to remove an item from the cart.

8. Add navigation messaging (i.e. confirmation) like toast messages to inform users on their actions and errors 9. Create uniform style of buttons (i.e. raised edges) on the site so users know when something is clickable.

10. Provide size chart for users to check their size measurements (inches, centimeters) before clicking on standard sizes (S, M, L)

12

Appendix A.

Consent form/ Information Sheet ADULT CONSENT TO PARTICIPATE IN RESEARCH Usability Evaluation of www.svrn.com Principal Investigators: Sahana Shanthigrama, Phil Traini, Cynthia Chong, Jay Kietzman (Graduate Students) Institution: DePaul University, Chicago, Illinois, USA Department (School, College): Computing and Digital Media (“CDM”) Faculty Advisor: Bob Konow, CDM, DePaul University What is the purpose of this research?

We are asking you to be in our research study because we are evaluating the usability of svrn.com website for its target users. This study is being conducted by Sahana Shanthigrama, Phil Traini, Cynthia Chong, Jay Kietzman at DePaul University as a requirement to obtain their Master’s degree. This research is being supervised by their faculty advisor, Bob Konow.

We hope to include about 4 people in the research.

Why are you being asked to be in the research?

You must be age 18 or older to be in this study. This study is not approved for the enrollment of people under the age of 18.

What is involved in being in the research study?

If you agree to be in this study, being in the research involves 15-20 minutes of contextual evaluation of tasks performed by you (participants). The interview will be audio recorded and the screen will be screen recorded as you perform the tasks. We will transcribe the 13

interview questions and responses into written notes to get an accurate record of what was said. The screen recording will give us an accurate record of the session that we can review later in our research process.

How much time will this take?

This study will take about 30 minutes of your time.

Are there any risks involved in participating in this study?

Being in this study does not involve any risks other than what you would encounter in daily life.

You may feel uncomfortable or embarrassed about answering certain questions. You do not have to answer any questions if you do not want to.

Are there any benefits to participating in this study?

You will not personally benefit from being in this study.

Can you decide not to participate?

Your participation is voluntary, which means you can choose not to participate. There will be no negative consequences, penalties, or loss of benefits if you decide not to participate or change your mind later and withdraw from the research after you begin participating. You may stop at any time during the evaluation if you choose not to participate any further.

Who will see my study information and how will the confidentiality of the information collected for the research be protected?

The research records will be kept and stored securely. Your information will be combined with information from other people taking part in the study. When we write about the study or publish a paper to share the research with other researchers, we will write about the combined information we have gathered. We will not include your name or any information that will 14

directly identify you. We will make every effort to prevent anyone who is not on the research team from knowing that you gave us information, or what that information is. However, some people might review or copy our records that may identify you in order to make sure we are following the required rules, laws, and regulations. For example, the DePaul University Institutional Review Board may review your information. If they look at our records, they will keep your information confidential.

Who should be contacted for more information about the research?

Before you decide whether to accept this invitation to take part in the study, please ask any questions that might come to mind now. Later, if you have questions, suggestions, concerns, or complaints about the study or you want to get additional information or provide input about this research, you can contact the researchers: Sahana Shanthigrama Manjunath Sahana.shanthigrama@gmail.com Phil Traini pjtrai@gmail.com Cynthia Chong cynthiacchong@gmail.com Jay Kietzman JFKIETZMAN@gmail.com DePaul Faculty Advisor: Bob Konow

15

rkonow@depaul.edu You may also contact DePaul’s Office of Research Services if: ● Your questions, concerns, or complaints are not being answered by the research team. ● You cannot reach the research team. ● You want to talk to someone besides the research team.

You will be given a copy of this information to keep for your records. ❏ Please check the box to indicate your consent for the interview to be audio recorded.

Statement of Consent from the Subject: I have read the above information. I have had all my questions and concerns answered. By signing below, I indicate my consent to be in the research.

Signature: _______________________________________________ Printed name: ___________________________________________ Date: ______________________ B. Script (Moderator’s greeting) Screener Questions 1.

Are you at least 18 years old or older?

2.

Are you interested in online shopping?

If participant answers “no” to any of the above questions, thank the participants for their time and let them know they will not need to participate in the interview.

16

Step 1: Introduction Hi, my name is ___________. I am a graduate student working on a research project at DePaul University. Thank you for agreeing to be a part of this usability evaluation. In this project, our team is evaluating svrn.com website to check how well constructed the website is and how easily target customers can interact with this website. There are no right or wrong actions as the evaluation is on website and your actions will not be judged.

Explain the purpose of the project: The data which we collect today will only be used for this class assignment; only our team will know your identity. Usability testing will help us to find out any other issues that user can face while using svrn site. Report result explains what steps/path users take to complete the task and how much time it takes for them to finish it. Also, to find out if user face any issue during task completion and how we can articulate those errors to increase the user experience overall with the website.

Explain what will happen: There are no right or wrong ways to perform tasks or answer my questions. I will only watch what you do and listen to what you say. I will be taking notes and video/audio record the session. During the interview, let me know if you need a question to be clarified or explained.

I will ask you questions before and after the task. There are no right or wrong answers.

This session will take approximately 30 minutes.

Informed consent: This activity is voluntary. You have the right to not perform a task or answer any question, and to stop the observation at any time for any reason. Your actions and responses will be confidential and used only in connection with this class assignment. Only your first name will be used to identify you. If you wish, you can use a pseudonym rather than your real name.

[Give participants the informed consent form] Please read over this informed consent form and sign at the bottom if you consent to take part in this research. Feel free to ask me questions before you sign the form 17

C. Task List 1.

Participants will be asked to add and remove an item from the cart.

2.

Participants will be asked to navigate the website to find a particular designer’s brand lineup.

3.

Participants will be asked to view a product and navigate back to the home page.

4.

Participants will be asked to search for a specific item of their choice.

5.

Participants will be asked to Sign up for the website’s newsletter.

D. Pre-test questionnaire Demographic Data Name (optional): _______________ Age: ____ Gender: ___ 1.

Do you like to shop online?

2.

In a month, how often do you shop online?

a.

1-10 times

b.

11-20 times

c.

more than 20 times 3.

Do you shop for shoes, clothing, or accessories through online websites? a.

4.

If no, what items do you most commonly shop for online?

Name a few favorite online shopping sites you like?

18

5.

6.

Have you heard of the website svrn.com before? a.

If yes, when was the first time you heard of it?

b.

If not, can you take a guess, what the website svrn.com is browsed /used for?

Have you browsed the website svrn.com before? a.

If yes, how long have you been using the website?

b.

What do you like most about the website svrn.com?

c.

Did you have any issues with the website svrn.com?

d.

Have you purchased any products from the website svrn.com before?

If yes, what was it?

E. Post-test questionnaire Directions: Please indicate your level of agreement or disagreement with each of these statements regarding svrn website Strongly Agree

Agree

Neutral Disagree Strongly Disagree

1. I was confident to navigate around the website without the need of any help.

2. It was easy to find what I was looking for on website 3. It was easy to understand information on website.

19

4. Website pages contained necessary information I was looking for.

5. I felt the need of technical support to use this website 6. I found website design visually appealing 7.It is easy to learn this website 8. I would like to revisit the website again 9. Are you likely to recommend this website to your friends/family?

Notes: 1.

Did the website help you to achieve your goals? □ Yes

2.

Were you able to find the information you were looking for on our website? □ Yes

3.

□ No □ No

Would you recommend svrn website to your friend or colleague? □ Yes

□ No

4.

What do you like most about visiting our website?

5.

What do you like least about the website?

20

6.

7.

What was your favorite feature or portion of the product or service?

How would you rate your overall experience of svrn website? □ Highly satisfactory □ Satisfactory □ Neutral □ Unsatisfactory □ Highly Unsatisfactory

8.

What can we do to improve your experience?

Closing Those are all the questions I have for you. Do you have any questions? Is there anything else you’d like to share? [Pause for questions and comments.] Thank you for your participation in the study G. Evaluation Notes General Success/Failure chart https://docs.google.com/spreadsheets/d/1QZtR81TcUcbvX9PCc6tcy31lhZy6l0veqyIbPgKRik/edit?usp=sharing Post-Questionnaire https://docs.google.com/spreadsheets/d/1I7zZS038jSZcI7J7EsWP1LHuFWY0Uaqsywu6L2nIQo/edit?usp=sharing Pre-Questionnaire https://docs.google.com/spreadsheets/d/1rwHlciss1wwyYWYhzkFyIJWCmT6vYSSMWEQ4JgJ jBcg/edit?usp=sharing

21

H. Data Logs and Recommendation Mockups Pre-test Questions Response Table-

22

General

Success/Failure

Table-

Post- Test Questionnaire Response Table-

23

Back Button Recommendation Mockup (Original Page, Mockup #1, Mockup #2)

24

25

Confirmation Recommendation Mockup (Original, Mockup #1, Mockup #2)

26

27

28

Search Option Recommendation Mockup (Original, Mockup #1, Mockup #2)

29

30

31

I. In-depth Tasks Log Sheet Spreadsheets of raw data https://docs.google.com/spreadsheets/d/11ApC-zGzubWqUCyE5oiPsX6pXiD4hX9BUUwjWqhKy0/edit?usp=sharing Task 1

Task 2

Task 3

32

Task 4

Task 5

33

Participant 1 Demographic Data Name (optional): ____________L Age: ____34 Gender: ___F Occupation: Children’s Center Director 1.

Do you like to shop online?

Yes I do it often.

2.

In a month, how often do you shop online?

a.

1-10 times

b.

11-20 times

c.

more than 20 times 3.

Do you shop for shoes, clothing, or accessories through online websites?

Yes a.

4.

If no, what items do you most commonly shop for online?

Name a few favorite online shopping sites you like?

Amazon.com, Target.com, michaels.com, lakeshorelearning.com

5.

Have you heard of the website svrn.com before?

No a.

If yes, when was the first time you heard of it?

b.

If not, can you take a guess, what the website svrn.com is browsed /used for?

Not sure, maybe clothing or shoes?

6.

Have you browsed the website svrn.com before?

No, because I never heard of it a.

If yes, how long have you been using the website?

b.

What do you like most about the website svrn.com?

c.

Did you have any issues with the website svrn.com?

d.

Have you purchased any products from the website svrn.com before?

34

If yes, what was it?

Log I: Observation Notes Participant:

Date:

Time:

Tasks

Issues

Observations, Comments, Notes

Task 1-

none

the user promptly completed the task

none

She completed the task quickly and promptly with no issues

Participants will be asked to add and remove an item from the cart .

Task 2Participants will be asked to navigate the website to find a particular designer’s brand lineup.

Task 3Participants will be asked to view a product and navigate back to the home page.

user used the back user didn’t use the heading of the page to navigate back to the main page. The user button on the browser used the back button 4 times to get back to the page, she later noticed during task 4 that she could click the Svrn banner to get back to the home page

the search was not bringing up specifically Use the search function what she was looking to search for a and didn’t offer any specific item of assistance your choice.

Task 4-

she searched shirt, she expressed frustration after because it just brought up men’s shirts. She then retyped “women’s shirts” to get 3 results. She seemed slightly frustrated

35

didn’t receive proper scrolled to the bottom of the page entered email after submitting it she did not receive confirmation Participants will be confirmation that she submitted. She asked to Sign up for double clicked was brought to a captcha the website’s page to which she filled it out. newsletter.

Quote “I think I did it should I try again? I think it worked I’m not sure”

Task 5-

Log II: Performance Metrics

Time of completion

Task 1 Task 2 Task 3

Task 4

19 secs 15 secs 10.62

55 sec 1min 34

Number of tasks completed correctly without 1 assistance

1

1

1(4 clicks)

Errors of Omission

Errors of Commission Total Errors

V.

Task 5

1

1 1

1

1

2

1

Post-test questionnaire

Directions: Please indicate your level of agreement or disagreement with each of these statements regarding svrn website

36

Strongly Agree 1. I was confident to navigate around the website without the need of any help.

Agree

Neutral Disagree Strongly Disagree

X

2. It was easy to find what I was looking for on website

X

3. It was easy to understand information on website.

X

4. Website pages contained necessary information I was looking for.

X

5. I felt the need of technical support to use this website

X

6. I found website design visually appealing 7.It is easy to learn this website

X

X

8. I would like to revisit the website again

X

9. Are you likely to recommend this website to your friends/family?

X

37

Notes: I didn’t receive confirmation that I signed up for a newsletter. I didn’t realize that clicking the logo could bring me back to the home page. Also when I was searching it brought up men’s clothing. I felt like I had to do too much to find what I wanted clicking here and there.

1.

Did the website help you to achieve your goals? □ Yes

2.

Were you able to find the information you were looking for on our website? □ Yes

3.

□ No

Would you recommend svrn website to your friend or colleague? □ Yes

4.

□ No

□ No

What do you like most about visiting our website?

The design was clean

5.

What do you like least about the website?

I had to be too specific with the search I couldn’t do a general search

6.

What was your favorite feature or portion of the product or service?

The website looked clean it wasnt messy

7.

How would you rate your overall experience of svrn website? □ Highly satisfactory □ Satisfactory □ Neutral □ Unsatisfactory □ Highly Unsatisfactory

8.

What can we do to improve your experience?

-search improvement -confirmation of newsletter -If there were more filters I feel like my experience would have been better

Participant 2 Demographic Data Name (optional): Age: 27 38

Gender: F Occupation: Student 1.

Do you like to shop online?

Yes, very much

2.

In a month, how often do you shop online?

a.

1-10 times

b.

11-20 times

c.

more than 20 times 3.

Do you shop for shoes, clothing, or accessories through online websites?

Yes a.

4.

If no, what items do you most commonly shop for online?

Name a few favorite online shopping sites you like?

Zara/ Nordstrom

5.

Have you heard of the website svrn.com before?

No a.

If yes, when was the first time you heard of it?

b.

If not, can you take a guess, what the website svrn.com is browsed /used for?

I don’t know, maybe local treat

6.

Have you browsed the website svrn.com before?

No

Log I: Observation Notes Participant: 2

Date: 08/03/19

Time: 4:15pm - 4:30pm

Tasks

Issues

Observations, Comments, Notes

39

Task 1-

None

The participant completed the task without any issue

None

The participant was asked to find the Vans line up.

She first looked for the search bar and typed in the keyword.

None

The participant couldn't find the back button on the website but be able to navigate back to the home page by clicking on the logo

Participants will be asked to add and remove an item from the cart .

Task 2Participants will be asked to navigate the website to find a particular designer’s brand lineup.

Task 3Participants will be asked to view a product and navigate back to the home page.

Didn’t find the I asked the participant to search for a black dress, right item so she searched “dress” in the search engine. But Use the search function there are no search results, so she looked for the immediately to search for a filter in women’s collection, and there is only a specific item of your “skirt” section. There’s no color filters or specific choice. assistant for finding specific items.

Task 4-

Task 5-

None

Participants will be asked to Sign up for the website’s newsletter.

The participant assumed that the signup section is usually located on the top right, so she couldn't find it in the first place. It took time for her to scroll down and find the newsletter section

Log II: Performance Metrics

40

Task 1 Task 2 Task 3 Task 4 15 secs 10 secs 20 secs

Time of completion

Number of tasks completed correctly without 1 assistance

1

Errors of Omission

45 sec 30secs

1

1

Errors of Commision

2

1

1

Total Errors

V.

Task 5

0

0

2

3

1

Post-test questionnaire

Directions: Please indicate your level of agreement or disagreement with each of these statements regarding svrn website

Strongly Agree

Agree

Neutral

1. I was confident to navigate around the website without the need of any help.

X

2. It was easy to find what I was looking for on website

X

Disagree Strongly Disagree

41

3. It was easy to understand information on website.

X

4. Website pages contained necessary information I was looking for.

X

5. I felt the need of technical support to use this website

X

6. I found website design visually appealing

X

7.It is easy to learn this website

X

8. I would like to revisit the website again

X

9. Are you likely to recommend this website to your friends/family?

X

1.

Did the website help you to achieve your goals? □ Yes

2.

Were you able to find the information you were looking for on our website? □ Yes

3.

□ No

Would you recommend svrn website to your friend or colleague? □ Yes

4.

□ No

□ No

What do you like most about visiting our website?

The minimal layout design of the website

42

5.

What do you like least about the website?

I had a hard time finding a specific item, there is no guideline or hints

6.

What was your favorite feature or portion of the product or service?

High quality and different views of the images

7.

How would you rate your overall experience of svrn website? □ Highly satisfactory □ Satisfactory □ Neutral □ Unsatisfactory □ Highly Unsatisfactory

8.

What can we do to improve your experience?

•

Provide Assistant / instructions

•

Better navigation design

Participant 3 Name (optional): Vihal Age: 32 Gender: Male Occupation: Software Engineer 1. Do you like to shop online? Yes 2. In a month, how often do you shop online? a. 1-10 times - 7 to 8 times b. 11-20 times C. more than 20 times 3. Do you shop for shoes, clothing, or accessories through online websites? Yes a. If no, what items do you most commonly shop for online?

4. Name a few favorite online shopping sites you like? Amazon, Hollister, Nike 43

5. Have you heard of the website svrn.com before? No a. If yes, when was the first time you heard of it? b. If not, can you take a guess, what the website svrn.com is browsed /used for? Might be food Restaurant website 6. Have you browsed the website svrn.com before? No a. If yes, how long have you been using the website? b. What do you like most about the website svrn.com? c. Did you have any issues with the website svrn.com? d. Have you purchased any products from the website svrn.com before?

If yes, what was it?

Log I: Observation Notes

Participant: Vihal

Tasks

Time: 6pm – 6.28pm

Task Issues Completed

Date: 08/03/2019

Observations, Notes

Comments,

Status

44

Task 1 Participants will be asked to add and remove an item from the cart.

Add to Cart User was unable to guess if item size is available or not Success as he directly clicked on M size (as the item was not available).

He also complained about size chart (standard sizes) as there was no size information on the website.

· User scrolled down on an original page, clicked on an image. · Click on image – connected to “Helmut clothing list” · Clicked on “Helmut Lang” and then selected “M” size on size options · Since M size was not available, he later clicked on “L” size.

Initially user clicked on M · User scrolled down to size when adding to cart but search if size chart is since it was not available, he available since he could find later clicked on L size it. · He scrolled up and clicked on Add to cart Remove Cart · User Maximized add to cart screen. · Scrolled over “Remove X” but later scrolled over “cart icon” on left side and clicked on it. · System showed add to cart mini pop window with the added items list. · User then clicked on “–” button to remove the item

User Quote: “ How should I know If M size will fit me or not, when they don’t

45

have a proper size chart information?�

46

Task 2 Participants will be asked to navigate the website to find a particular designer’s brand lineup.

· User clicked on “Men” · Under “Designers” list, he clicked on “Alchemist” brand Success

· Screen showed a blank page with Alchemist brand info with no clothing list.

None

· He later clicked on “Marni” brand to check if there are any clothing available under the brand. · User then clicked on Marni sweater.

Quote: “This website is not well designed I guess; they just have given brand information but no info on clothing availability!” · User clicked on “Marni sweater”

Task 3 Participants will Success be asked to view a product and navigate back to the home page.

· Went through sweater User was trying to search images. back button and later clicked on SVRN text field to get · User scrolled up and back into the original page. searched for back button. · Later clicked on “SVRN” heading text to get back into the original page.

Task 4

· User clicked on search tab Success

Use the search function to search for a specific item of your choice.

None

· Types “pants” and pressed on “enter” button from keyboard. · Scrolled down to choose his favorite pants from clothing list. · Clicked on “clot split pant”

47

· Went through pant images

· In the original page, user scrolled down to bottom of page.

Task 5 Success

None

Participants will be asked to Sign up for the website’s newsletter.

· Clicked on newsletter text field · Typed his mail id · Clicked on join button · Once system showed confirmation notification, he scrolled up

Log II: Performance Metrics

Task 1

Time of completion

Task 2

50sec

Task 3

Task 4

Task 5

33sec

21sec

17sec

1

1

1

1(looking for 0 back button)

0

31sec Number of tasks 1 completed correctly without assistance Errors of Omission

1

1(clicked on M size as the 0 text was faded to show unavailability)

48

Errors of Commission

0

0

0

0

0

Total Errors

1

0

1

0

0

Post-test questionnaire Directions: Please indicate your level of agreement or disagreement with each of these statements regarding svrn website

Strongly Agree

Agree

1. I was confident to navigate around the website without the need of any help.

Disagree Strongly Disagree

✓

2. It was easy to find what I was looking for on website

3. It was easy to understand information on website.

Neutral

✓

✓

49

4. Website pages contained necessary information I was looking for.

✓

✓

5. I felt the need of technical support to use this website

✓

6. I found website design visually appealing

✓

7.It is easy to learn this website

✓

8. I would like to revisit the website again

✓

9. Are you likely to recommend this website to your friends/family?

Notes: 1.

Did the website help you to achieve your goals? đ&#x;”˜Yes

2.

Were you able to find the information you were looking for on our website? đ&#x;”˜ Yes

3.

â–Ą No

Would you recommend svrn website to your friend or colleague? đ&#x;”˜ Yes

4.

â–Ą No

â–Ą No

What do you like most about visiting our website? They got a lot of branded clothing.

50

5. What do you like least about the website? Website is not attractive and visually appealing, it can be well designed.

6. What was your favorite feature or portion of the product or service? No favorite feature but Search option was helpful 7.

How would you rate your overall experience of svrn website? â–Ą Highly satisfactory â–Ą Satisfactory đ&#x;”˜ Neutral

â–Ą Unsatisfactory â–Ą Highly Unsatisfactory 8.

What can we do to improve your experience?

Website is not visually attractive and menu, size chart options needs to be included.

Participant 4 Name (optional): Gabriella Age: 26 Gender: Female Occupation: Bartender 1. Do you like to shop online? Yes 2. In a month, how often do you shop online? a. 1-10 times b. 11-20 times C. more than 20 times 3. Do you shop for shoes, clothing, or accessories through online websites? Yes

51

a. If no, what items do you most commonly shop for online?

4. Name a few favorite online shopping sites you like? Pinksia, Akira, Amazon 5. Have you heard of the website svrn.com before? No a. If yes, when was the first time you heard of it? b. If not, can you take a guess, what the website svrn.com is browsed /used for? No 6. Have you browsed the website svrn.com before? No a. If yes, how long have you been using the website? b. What do you like most about the website svrn.com? c. Did you have any issues with the website svrn.com? d. Have you purchased any products from the website svrn.com before?

If yes, what was it?

Log I: Observation Notes Participant: Gabriella

Date: 08/04/19

Time: 3:00 PM

Tasks

Issues

Observations, Comments, Notes

User could not The user had no issue adding the item to her instantly find the card. To remove the item from her cart she correct button to first clicked on the word “Remove,” then remove the item from clicked on the “update cart” button. She then Participants will be tried clicking on the “X” under “Remove.” asked to add and her cart remove an item from the cart.

Task 1

52

Task 2

No issues

User had no issues navigating to a designer’s line up

No issues

User made the correct action on her first attempt.

Participants will be asked to navigate the website to find a particular designer’s brand lineup.

Task 3 Participants will be asked to view a product and navigate back to the home page.

User had to try to search two times to find what she was Use the search looking for and neither function to search for time produces good a specific item of your results choice.

Task 4

User used the search bar from the home page and searched “shoes.” One result came up.

She then searched for “Vans” and two results came up, one of which was a jacket. **Search function is major navigation issue

No issue navigating to User did not receive messaging to indicate that they were successful in their action of sign up signing up for the newsletter. User also looked on the left side bar at the bottom of Participants will be the page before seeing the newsletter sign up asked to Sign up for on the right side. the website’s newsletter.

Task 5

Log II: Performance Metrics Task 1

Task 2

Task 3

Task 4

Task 5

53

Time of completion

27 secs 9 secs

6 sec

21 sec 15 sec

Number of tasks completed correctly without 1 assistance

1

1

1

1

Errors of Omission

0

0

0

0

0

Errors of Commission

2

0

0

0

0

Total Errors

2

0

0

0

0

Post-test questionnaire Directions: Please indicate your level of agreement or disagreement with each of these statements regarding svrn website

Strongly Agree

1. I was confident to navigate around the website without the need of any help.

Agree

X

2. It was easy to find what I was looking for on website

3. It was easy to understand information on website.

Neutral Disagree Strongly Disagree

X

X

54

4. Website pages contained necessary information I was looking for.

X

5. I felt the need of technical support to use this website

X

6. I found website design visually appealing

7.It is easy to learn this website

X

X

8. I would like to revisit the website again

X

9. Are you likely to recommend this website to your friends/family?

X

Notes: User found the site to be a “hot mess.” She pointed out that there are significantly less options in the women’s section then the men. She also felt that the site lacked brand identity. She felt that organization of the site would greatly improve her experience. Overall, her biggest issues of navigation were in recognizing messaging, success of search bar use, and navigation organization.

1.

Did the website help you to achieve your goals?

Yes,

2.

Were you able to find the information you were looking for on our website?

Yes,

3.

No X

Would you recommend svrn website to your friend or colleague?

Yes,

4.

No X

No X

What do you like most about visiting our website?

The shoes, specifically the sneakers 55

5.

What do you like least about the website?

Lack of organization and structure of the site

6.

What was your favorite feature or portion of the product or service?

The easy navigation button to return to the home page (top center “SVRN� on every page) 7.

How would you rate your overall experience of svrn website?

Highly satisfactory Satisfactory Neutral X Unsatisfactory Highly Unsatisfactory

8. What can we do to improve your experience?

Organize the site by theme better and/or create a band for the site that makes them stick out.

J. Member Contribution Cynthia Chong

Interview, User Profile, Participants, Persona, Findings and Recommendations, Spreadsheet, Working Schedule

Jay Kietzman

Interview, Test Design, Evaluation Measures, Test Results, Recommendations

Phil Traini

Interview, Overview, Goals, Research Questions, Test Results, Spreadsheets, Mockups

Sahana Shanthigrama

Interview, Participants, The Consent Form, Scripts, and Pre and Post Test Questionnaires, Findings and Recommendations

56

Everyone

Executive Summary, Report Contents/Summary, Evaluation Notes, Data Log sheets, Spreadsheets, Finalizing and Formatting

K. Working Schedule 7/21

- First group meeting - Decide which application for our project - Work dividing

7/22

- First Draft of Goals and Research Questions, User Profile, Test Plan, Evaluation Measures, Consent Form, Scripts and Questionnaires

7/23

- Edited and revised the first draft of the report - Second group meeting on work discussion

7/24

- Finalized every section of the report - Report Contents/Summary - Third group meeting on work finalization

7/25

Initial Report

7/30

- Planning on Usability Testing - Forth group meeting

7/31 - - Conduct Usability Testing 8/3 8/4

- Generate Usability Testing Results Notes

8/5

- Edited and revised project 3 - Analyze Usability Testing Results

57

- Fifth group meeting 8/6

- Participants Information - analyze Usability Testing Results - Finding and Recommendations

8/7

- Finalized every section of the report - Executive Summary - Report Contents/Summary - Sixth group meeting

8/8

Final Report

58