User testing by Kristen Chon

July 29th, 2013

User Testing Research AGENDA ITEMS: Types of User Testings Procedures Sample size and Demographics Time frame of User Testing Recommendations

! ! ! ! !

.biz

User Testing Research

Thinking Aloud Protocol

2 Ă?

User Testing Research

3 Ð

Thinking Aloud Protocol  What A way of understanding the contents of working memory !

“In a thinking aloud test, you ask the test participants to use the system while continuously thinking out loud - that is, simply verbalizing their thoughts as they move through the user interface” ! ! ! !

Traditional Test administrator says as little as possible - Can only say, “Keep talking.” !

Speech-communication Test administrator can acknowledge the test participant’s verbalizations with “Mm-hmm” or “Uh-huh” in addition to “Keep talking.” !

Coaching Test administrator asks the participant for feedback and actively intervenes with probes such as “Did you noticed that link up here?” “You are doing great,” or “Can you explain why you clicked on that link?

When At any stage in the development state (Early prototypes to fully running systems) !

Demographics 2-8 users per session !

Time Frame However long it takes to complete the task; this method will increase task time ! Source: HowTo.gov, Austin Center for Design

! ! !

.biz

© 2010 AT&T. All rights reserved. AT&T, the AT&T logo and all other AT&T marks contained herein are trademarks of AT&T Intellectual Property and/or AT&T affiliated companies. All other marks contained herein are the property of their respective owners.

User Testing Research

4 Ð

Thinking Aloud Protocol  Methodology 1

Develop a prototype All designed screens, but they don’t have to be hot

Schedule sessions with users that match the target audience 2-8 users per session

! !

Develop tasks that present typical user goals Scenarios and paths

Organize and Record Video, batteries, tapes, pens, etc.

Get set up Physical arrangement, written consent ! !

Explain rules to the user Who you are, what you are doing, you are testing the interface not the user, they can quit any time, no help provided, continue talking, verbalize actions and verify that the user understands tasks ! Source: Austin Center for Design

.biz

User Testing Research

5 Ð

Thinking Aloud Protocol  Methodology

Run the session Take good notes, if the user is silent for more than 3 seconds, prompt them to keep talking !

Do not say “Please explain what you are doing,” “Note any design problems you see,” “Tell us if you have any suggestions,” “Why are you doing what you are doing.” These can disrupt the information state/memory and alter data

Identify critical incidents ! !9

Calculate a SUS score per user ! ! !

Calculate a System Usability Scale (SUS) score all users Odd items - Subtract one from the user response Even items - Subtract the user response from 5 This scales all values from 0-4 Average across all users ! Source: Nielsen Norman Group

! ! ! !

.biz

User Testing Research

6 Ð

Thinking Aloud Protocol

Pros and Cons

Pros •No introspection •Robust •Flexible •Convincing •Cheap

Cons •Unnatural situations •Filtered Statements •Biasing user behavior •No detailed statistics

! ! ! ! ! !

Source: Austin Center for Design

! ! ! ! ! ! ! !

.biz

User Testing Research

7 Ð

Thinking Aloud Protocol  Recommendations

•Coaching method may provide assistance that users would not get on their own !

•The traditional or speechcommunication methods will reflect an environment more similar to what users might experience “in the field,” where they will have no assistance ! ! ! ! ! !

.biz

•Usability practitioners should carefully consider which method to use for each usability test and discuss the implications of the decision with the test sponsors ! ! ! ! ! ! !

•When writing the documentation for the test, usability practitioners should accurately report and document the Think-Aloud method they used ! Source: Austin Center for Design

! ! ! ! ! !

User Testing Research

A/B Testing

8 Ă?

User Testing Research

9 Ð

A/B Testing  What A way of conducting an experiment where you compare a control group to the performance of one or more test groups by randomly assigning each group a specific single-variable treatment

Conversion Rate # of visitors who register _______________________ # of total visitors

Runs the experiment in parallel and randomly assigns a treatment each person who visits (Controls for any timesensitive variables and distributes the population proportionally) !

No behavioral insights !

Provides data only on the element you’re testing ! ! !

.biz

! !

•Two versions for an element •Metric that defines success •Subject both versions simultaneously Elements tested •The call to actions (Wording, size, color and placement) •Headline or product description •Form’s length and types of fields •Layout and style of website •Images on landing and product pages •Amount of text on the page (Short vs. Long)

When Test when there’s not much traffic and results are needed quickly ! ! ! ! ! ! !

! ! !

User Testing Research

10 Ð

A/B Testing

Methodology 1

Design/create the two elements to test against each other Two creatives, ads, landing pages, conversion processes, search engines, etc. ! !

2 Determine the success metric on which to judge the test Click Thru Rate (CTR), Conversion Rate, time spent on site, page views, position

! !

Determine any success/ 3 conversion latency How long does it take to see a success metric after initial exposure to the element? !

.biz

Determine the desired confidence level of the results 95% confidence level is suggested for the digital area ! !

5 Determine how long you think you should actively run the elements against each other Put latency aside. Run the test for at least 14 days

! !

Determine your sample size 6 How many people do you want to survey? Testing 10% of the total affected population is sufficient

The minimum active test has been defined Length of active test must meet your sample size requirements and cover the time for data stability ! !

8 Run the active test Check progress regularly through period

! !

End active test and wait average 9 latency period ! !

Calculate ‘Winner’ and associated confidence level of results Consider longer test if confidence level is not 95%

User Testing Research

11 Ð

A/B Testing

Pros and Cons

Pros •Gets clear evidence •Good for testing new ideas •Answers specific design questions •Tests reality, not theory •Quick to conduct •Provides actual numbers that can be compared •Accurate ! ! ! ! !

Cons •Overly reductive •Abdicates responsibility •Ignores emotional value •Only fitting for designs with one, clear goal •Goal has to be measured by a computer •Only works for fully implemented designs •Short-term focus ! Source: Experience Solutions, Useful Usability

! ! ! ! ! ! !

.biz

User Testing Research

12 Ð

A/B Testing  Recommendations

DO •Know how long to run a test before giving up; don’t give up too early •Show repeat visitors the same variations; this prevents mistakes •Make your A/B test consistent across the whole website •Do many A/B tests

DON’T •Wait to test the variation until after you’ve tested the control. Always test both version simultaneously •Conclude too early (Statistical confidence) •Surprise regular visitors •Let your gut feeling overrule test results

! ! ! ! !

.biz

Combine A/B testing with other methods Qualitative observation of user behavior is faster and generates deeper insights (traditional usability test) ! Source: 20 Bits, Wired, NN Group

! ! ! ! !

User Testing Research

Card Sorting

13 Ă?

User Testing Research

14 Ð

Card Sorting  What A user-centered designed method for increasing a system’s findability. THe process involves sorting a series of cards, each labeled with a piece of content or functionality, into groups that make sense to users !

It is an inexpensive method that can be used to determine optimal website layouts, navigation structure, wording of website copy, how to arrange blocks of text, and more ! ! !

Open Card Sorting •Participants are given cards that have the names of the website’s section on them. They are responsible for creating groups and organizing them •Good for new and existing websites •Can solve/prevent problems that arise from the information architecture !

Closed Card Sorting •Participants are asked to place content into preexisting groups, categories, and sections. The most efficient method is to provide participants with basic top-level groups and let them go from there •A mixture of open/closed card sorting is possible ! ! !

.biz

When •Designing a new site •Designing a new area of a site •Redesigning a site •Best tested at early stage of design !

Demographics •Someone who is online/uses computer a lot •Should be predefined intended audience is important !

Why •You learn how different people think about, organize and expect to access your content •You learn about the language used by a particular group !

Source: Usability.Gov, Six Revisions

! ! ! !

User Testing Research

15 Ð

Card Sorting  Methodology 1

Preparation •Select content •Select participants •Prepare the cards ! !

2 Selecting content •Existing online content •Descriptions of business groups and processes •Planned applications and processes •Potential future content

! !

Number of cards •Recommends 30-100 cards 3 •Can have more or less ! !

Granularity and sampling content •Content should be representative of the site •Ensure that the content has enough similarity to allow groupings to be formed

Preparing the cards •Each item on the list placed on card •May be printed on standard mailing labels or print by hand •Recommend using 3’’x5’’ index cards

! ! !

! !

Selecting participants 5 Exercise will be performed multiple times •Individuals: 7-10 •Groups: 5 groups of 3 per group (Total 15) ! ! ! ! !

Execution 7 •Facilitate - Observe and listen •Keep the momentum going without leading the participants ! !

Analyzing the results/next steps •Enter information into a 8 spreadsheet •Analyze/identify patterns !

.biz

User Testing Research

16 Ð

Card Sorting

Pros and Cons

Pros •Provides good information •Established •Involves users •Cheap •Simple •Quick to execute ! ! ! ! !

Cons •Does not consider users’ tasks •Results may vary •May capture “surface” characteristics only •Analysis can be time consuming ! ! !Source: Experience Solutions, Useful Usability ! ! ! !

.biz

User Testing Research

17 Ð

Card Sorting  Recommendations

Consider conducting a synonym analysis If you’re having trouble settling on a final label name, consider a synonym analysis or review search engine and site search data !

Make sure objects being sorted are themselves classifications Make the cards sortable ! ! ! ! !

Don’t expect the same results Discrepancies are good !

Look for more information in the conversations than in the results !

Be clear on your intentions Choose from closed (validating) and open sorting (learning) !

Don’t equate final card sort as your site structure Look at this as input and translation is still required

Run this session with the actual site visitors !

Run with individuals and with groups !

Be clear about labeling Use a variety of labels and descriptions, unless you are testing labels as well ! ! !Source: Sorting Things Out: An Introduction to Card Sorting !by Stephen Andersen (Slideshare) !

! ! ! ! ! ! !

.biz

User Testing Research

Eye Tracking

18 Ă?

User Testing Research

19 Ð

Eye Tracking  What Uses special equipment to detect exactly where people’s eyes are focused when they look at a computer screen !

What part of a design are users attracted to? Which do they over look? !

What draws attention and why? !

Augments traditional usability methods, providing additional information that the test participant cannot report and the researcher cannot observe •First glance •Search patterns •Failed search ! !

Demographics Gender •Male/Female ratio on Internet usage is equal •Males have a longer duration of fixation than females and react differently to images, which affects memory performance •Females have more detailed information processors-they notice more of the entire web page than males •Females focus on all information available while males focus on fewer areas •Males have higher level of self confidence in focusing on a new challenge and are more likely to apply general knowledge ! ! ! !

.biz

Demographics Age •Young computer savvy users Vs. Aging seniors who didn’t grow up with constant access to a computer •Older users rely on main page to direct them to where they want to go •Younger users want to navigate further within the website without constantly returning to the home page !

When •6 hour slots •Session lasts 1.5 hours ! Source: UX Matters, User Interface Engineering

! ! ! ! ! !

User Testing Research

20 Ð

Eye Tracking  Methodology 1

Communicate to the participant during the screening interview Typical points covered, “we will be video taping the session,” “This is in no way a test of you or your abilities.”

Questions to asks the user during the interview General target profile question/ is it a match?

Disclose to the user that his eye will be tracked during the session

If so, ask questions related to the eye Never disclose to potential participant the answer you are hoping for

Recruit over the phone than email ! !4

Number of test participants Generating heat maps: 39 Qualitative (Gaze replays): 6 ! !

Do not over-explain/emphasize Limit the number of questions eye tracking. Briefly mention the you ask about the eye to less eye-tracking technology than 8 ! !

.biz

User Testing Research

21 Ð

Eye Tracking

Pros and Cons

Pros •Proof of what people say •Were brand elements noticed? •Choosing a homepage design •Shows hard-to-articulate behaviors •Providing additional insights •Visualizing data for observers •No interruption (Like the talk aloud test) ! ! ! ! !

Cons •No proof/correlation on what attracts the eye/mind •Doesn’t capture peripheral vision •Fixation doesn’t represent attention or communicate meaning •Can interfere with a naturalness of a test session (Self consciousness) •Difficult to learn •More time consuming •Expense/technical difficulties •Less opportunity for user feedback •Often incompatible ! Source: Experience Solutions, Useful Usability

! ! ! ! ! ! ! !

.biz

User Testing Research

22 Ð

Eye-Tracking  Recommendations

Give people some openended tasks This will help you see what people look at rather than influencing or imposing behavior on them !

Incorporate quantitative measures It can provide the foundation for a more mature approach to measuring usability !

Start with a research question/ hypothesis Brainstorm questions and use one or more of them to form a hypothesis ! !

Create a panel from your customers or target market It will get your better quality data than using the general population

Don’t skip the playback video

Be very specific in your task requirements !

Choose at least one alternative design (radical redesign) to test to compare Look at existing pages and test challengers !

Don’t stop at the aggregate Insights are always hidden in aggregate data. Analyzing individual sessions will give you some context around each visit. ! ! !

.biz

Use participant feedback I t will help you unpack what’s going on on the behavior side Avoid drawing conclusions from a small sample This is qualitative, not quantitative data !

Track clicks and mouseovers This helps you identify which call to actions stand out, and which tasks were completed or no action taken !

Complement with A/B Test Quantitative data can confirm your hypothesis ! Source: The UX Factor, UX Magazine, Get Elastic

! ! !

User Testing Research

Heuristic Evaluation Usability Testing

23 Ă?

User Testing Research

24 Ð

Heuristic Evaluation  What A usability engineering method for finding the usability problems in a user interface design so that they can be attended to as part of an iterative design process !

Each individual evaluator inspects the interface alone. Multiple evaluators separately evaluate and are allowed to communicate and have their findings aggregated. !

This procedure is important in order to ensure independent and unbiased evaluations from each evaluator. ! ! !

The experimenter can assist the evaluators in operating the interface in case of problems, and help if the evaluators have limited domain expertise !

The experimenter has the responsibility of interpreting the user’s actions in order to infer how these actions are related to the usability issues in the design of the interface ! ! ! ! ! !

When •Beneficial at early stages of design •Session for an individual evaluator lasts one or two hours •Longer evaluation session might be necessary for more complicated/larger interfaces !

Demographics •Proportion of usability problems found increases as number of evaluators increase •Jacob Nielsen states that a single evaluator found only 35% while 15 evaluators found up to 90% ! ! ! ! ! ! !

.biz

User Testing Research

25 Ð

Nielsen’s Heuristics  Guidelines 1

Visibility of the system status The system should always keep users informed about what is going on, through appropriate feedback within reasonable time

User control and freedom Support undo and redo

! ! !

Match between system and the 2 real world The system should speak the user’s language. Follow realworld conventions, making information appear in a natural and logical order ! ! ! ! ! ! ! .biz

! !

4 Consistency and standards Follow platform conventions

Error prevention 5 Eliminate error-prone conditions or check them and present users with a confirmation option before they commit to the action ! ! ! ! ! !

Recognition rather than recall Instructions for use of the system should be visible or easily retrievable ! ! !

7 Flexibility and efficiency of use Allow users to tailor frequent actions

! ! ! ! ! ! ! !

User Testing Research

26 Ð

Nielsen’s Heuristics  Guidelines 8

Aesthetic and minimalist design Every extra unit of information in a dialogue competes with the relevant units of information and diminishes their relative visibility ! ! ! ! ! ! ! ! ! ! ! ! ! !

Help users recognize, diagnose, and recover form errors Error message should be expressed in plain language, precise indicate the problem, and constructively suggest a solution ! ! ! ! ! ! ! ! ! ! ! ! ! !

.biz

Help and documentation It may be necessary to provide help and documentation. Any such information should be easy to search, focused on the user’s task, list concrete steps to be carried out, and not be too large ! ! ! ! ! ! ! ! ! ! ! ! ! !

User Testing Research

27 Ð

Heuristic Evaluation

Pros and Cons

Pros •Assigning the correct heuristic can help the best corrective measures to designers •You can conduct usability testing to further examine potential issues •Quick and relative inexpensive •You can use it together with other usability testing methodologies ! ! ! !

Cons •It requires knowledge and experience to apply heuristics effectively •The evaluation may identify more minor issues and few major issues •Trained usability experts are sometimes hard to find and can be expensive •You should use multiple experts and aggregate their results ! ! ! ! ! ! ! ! !

.biz

User Testing Research

28 Ð

Heuristic Evaluation Usability Testing  Recommendations

Justify every problem with a heuristic “Too many choices on the home page” suggests aesthetic and minimalist design !

Can’t just say “I don’t like the colors” !

List every problem Even if an interface element has multiple problems

List every problem Even if an interface element has multiple problems !

Go through the interface at least twice Once to get the feel of the system !

Again to focus on particular interface elements ! !

Don’t have to limit to the 10 Nielsen heuristics They are good because it’s a short list that covers a wide spectrum of usability problems !

Can always be a guide ! ! Source: Heuristic Evaluation guide by MIT Computer Science and Artificial Intelligence Laboratory

! ! !

.biz

User Testing Research

Tree Testing

29 Ă?

User Testing Research

30 Ð

Tree Testing  What A variant of usability testing where the only variable to be tested is the site structure and participants are asked to find different pieces of information via a clickable sitemap.

Demographics Margin or Error (+/-) 10 users = 27% 39 users=15% 115 users=9% 263 users=6% 1064 users=3%

! ! ! ! ! ! !

Evaluates findability, labeling and organization of the structure of the website. !

Compliment to Card sorting ! ! !

.biz

When •15-20 minutes •Session lasts 1.5 hours !

Why •Detects navigational issues early prior to building a prototype or dynamic site •Analyzes first and all attempts where participants had trouble navigating before your site goes live ! ! ! ! ! ! !

User Testing Research

31 Ð

Tree Testing  Procedure 1

Organize and create your item hierarchy list Keep the number of items in your list to under 1000 items to prevent the user abandoning the task prematurely

.biz

Create a task by asking participants to locate a particular product using only the list of items Consider removing “help” and “contact us” topics from your tree test. Guide participants to work with your navigation structure

Configure options based on your tree testing tool

Launch your study and collect results

User Testing Research

32 Ð

Tree Testing

Pros and Cons

Pros •Research results are easy to analyze and react upon •Ideal for comparing alternative structures by measuring the relative effectiveness •Participants find it easier to provide feedback when testing a structure (tree testing) rather than creating one (card sorting) •Easy to conduct; It can be conducted online via a web application •Can create a test under 10 minutes •Sessions can be completed remote and face-to-face, moderated or unmoderated

Cons •With an unmoderated study, user’s thought process can’t be achieved •Relatively new technique ! ! ! ! ! ! ! !

! ! !

.biz

User Testing Research

33 Ă?

Tree Testingâ&#x20AC;¨ Recommendations

Test a few different alternatives Tree tests are quick and easy so this resolves opinion-based debates over what option is better. ! ! ! ! !

.biz

Test new against old Compare success rates to achieve a solid, quantitative improvement ! ! ! ! !

Practice and repeat Do two or three revision cycles for a given tree, using each set of results to progressively tweak and improve it ! Source: UX Alliance

ÂŠ 2010 AT&T. All rights reserved. AT&T, the AT&T logo and all other AT&T marks contained herein are trademarks of AT&T Intellectual Property and/or AT&T affiliated companies. All other marks contained herein are the property of their respective owners.

User Testing Research

Thank You

34 Ă?