Involved Parties Personal Details Name: Martijn Verschuuren Education: ICT & Media Design University: Fontys University of Applied Science City: Eindhoven, The Netherlands Student Number: 2055808 Company Details Name: Betterplace.org Address: Schlesische StraĂ&#x;e 26 Postal Code: D-10997 Berlin Phone: +49 30 7676 4488 44 Fax: +49 30 7676 4488 40 Email: change@betterplace.org Internet: www.betterplace.org Contact person Betterplace: Niklas Sum Skype: tehniklas E-Mail: Nsu@betterplace.org Graduation Teacher
Dhr. L. Maassen
Second Graduation Teacher
Dhr. F. van Alebeek
Foreword This research has been written to graduate at the studies ICT & Media Design at the Fontys University of Applied Science in Eindhoven. For this research I have done a twenty week graduation internship at Betterplace.org Berlin, Germany. During this research I searched for tools to monitor and improve the User Experience for the Betterplace website. Next to my research, I improved parts of the website, added new content and worked out concepts to keep the visitors satisfied. I would like to thank the people at Betterplace for the opportunity to graduate. In particular I would like to thank Niklas Sum for guiding me during this internship and my colleagues from development, with whom I worked together on daily bases. Martijn Verschuuren
3
Index Explaining words list 5 Problem analysis 6 Aim of research 6 Research questions 7 Research methodology 8 Chapter 1: About User Experience What is User Experience? 10 UX Honeycomb 14 Why is UX important? 15 Example of bad UX 16 Why create a UX process? 18 Chapter 2: Tools & Techniques A/B split testing 21 Multivariate testing 27 Heat maps 33 Personas 36 Heuristic Evaluation 38 Low Budget Testing 39 Other Companies 44 Conclusion 51 Chapter 3: Advice & Recommendations Advice & Recommendations 54 Chapter 4: Appendix
4
Explaining Words List Bounce Rate
Visitors who enter the site leave rather than continue viewing other pages within the same site
Conversion Rate
Goals divided by number of visitors
Core Content
Content that is impractical for competitors to replicate
Moderated Testing UX testing with live view of the users screen with the possibility to interact MVT Multivariate Test(ing) Pros & Cons
Arguments for (pro) or against (con) a particular issue
Remote Based Testing Testing where users complete pre-defined tasks from their home Response Time
The time a user takes to react to a given input
UX User Experience Unmoderated testing
UX testing without possibility of interaction with users
URL
Uniform Resource Locator - Link to webpage of file
WYSIWYG
What You See Is What You Get
5
Problem Analysis The ever changing expectations from users of the world wide web ask for continuous improvement of websites today. Betterplace.org, as one of the biggest donation websites in Germany, is under continuous development and is always evolving to meet the changing demands of their visitors. To keep up with the competition, Betterplace is constantly improving the usability and user experience of the website. This is achieved by incremental improvements in realtime. This way, the website is never finished and maintains up to date with the latest Internet trends. The problem is that there is no process concerning user experience involved in the creation or improvement of web pages. The product management and development teams have an eye for usability and user experience, but there’s no process if a page shows a low conversion rate.
Aim of Research Betteplace.org is depending on the donations of visitors. In order to keep these visitors satisfied, it’s critical to give them the best user experience while visiting the website. Therefore the objective of this research is to investigate which tools Betterplace can use to improve the user experience and thus improve the bond with their visitors.
6
Research Questions The main research question in this report is: Which User Experience tools and techniques are relevant to improve the User Experience process at Betterplace.org? My research will exist out of three parts: Part 1: User Experience Introduction What is User Experience? Why is User Experience important to a company? Why have a UX Process? Part 2: Overview of tools and techniques Which tools and techniques are popular? Which tools and techniques are other companies using? Which tools and techniques are relevant for Betterplace? Part 3: Advice & Recommendations
7
Research Methodology The following research methods have been chosen for the particular research which needs to be done. The mixture of the research methods will lead to the answers of the sub questions as well as the main question: Which User Experience tools and techniques are relevant to improve the User Experience process at Betterplace.org? Desk research In order to answer tvhe questions concerning, ”what is User Experience?”, “why is User Experience important for a company?”, and “why create a UX process?” I will first and foremost use secondary data. Qualitative research Qualitative research will take place in the form of meetings with my supervisor and the company’s development department. During these meetings I will collect data concerning Betterplace’ mission statement and vision and get an insight of the tools and techniques valuable to the company. Quantitative research The quantitative research in this document will be the testing of tools and techniques. During this part of my research I will compare price, compatibility, features and usability. The outcome of this research will give me an answer to the question “Which tools and techniques are relevant for Betterplace?”.
8
User Experience
What is User Experience (UX)? It’s pretty hard to get a good definition of User Experience. User Experience is basically a sum of series of interactions that deal with the way a person feels about using a product, system or service. It represents the perception left in someone’s mind following a series of interactions between people, devices, and events. In this day and age, User Experience has become a big deal in our daily lives, but only few people recognize the benefits of good UX. People only tend to see bad UX and consider it as unavoidable. Everyone has had encounters with bad UX: an MP3 player that doesn’t play music in a way that makes sense to you, an online form that’s confusing and takes more than one try to fill out or even a mobile phone that has so many features, making a phone call becomes a chore. These issues could have been avoided by testing the UX, instead of creating a product the way the designer would like it. Users notice these issues, but seldom take action to make a change. In their opinion, every device, system or service is bound to have a few flaws and you’ll just have to live with it. They will be left with a bad experience, which leads to a negative association with the company from which the product came. Ultimately, if these bad experiences keep up, they will stop using one company’s products and join the competition. Exemplary user experience requires to meet the exact needs of the customer, without fuss or bother. Next comes simplicity and elegance which make a product of service a joy to use and own. True user experience goes beyond giving the costumers what they say they want, or providing a checklist of features. In order to achieve the ultimate User Experience in a company’s offerings, there must be a seamless merging of the services of multiple disciplines, including engineering, marketing, graphical and industrial design and interface design.
10
Outdated design process
11
User Experience
User Experience isn’t part of a process. It is a process itself which needs to be integrated in everything you do. In order to create a good experience for your users, you can’t just design something people would like to use. Instead, it is an ongoing effort, a process of continually learning about users, responding to their behaviours, and evolving the product or service. User Experience is like fashion, in the way that it’s never done. It’s dynamic. As time goes by, circumstances change and you will have to iterate. It doesn’t have to be a rigid process, but it does need to exist. The term User Experience was originally used in reference to human-computer interactions and is still largely associated with those disciplines. But nowadays the term is used to refer to any specific human-design interaction, ranging from a digital device, to a sales process, to an entire conference. Perhaps due to its organic development and lack of formalization, User Experience may be defined by very different departments from organization to organization. In some organizations, it is owned by marketing, in others it falls under IT. Then, from a solutions perspective, some organizations base their User Experiences around the research and academic-based approaches of human-computer interaction, others treat interface and product design as the source for User Experience, while still others let marketing or IT drive it. A lot of companies still think they can do without User Experience Design and see good experience design as an extra, not a requirement. The biggest misconception these companies make is thinking they have a choice to invest in their user’s experience. They might be doing well now, but as time goes by, they won’t be able to survive.
12
UX Honeycomb
13
UX Honeycomb Many people confuse User Experience with Usability. Although usability is an important part, UX is so much more. Peter Morville’s UX Honeycomb explains this best. Useful
Valuable
As practitioners, we can’t be content to paint within the lines drawn by managers. We must have the courage and creativity to ask whether our products and systems are useful, and to apply our deep knowledge of craft and medium to define innovative solutions that are more useful.
Our sites must deliver value to our sponsors. For non-profits, the user experience must advance the mission. With for-profits, it must contribute to the bottom line and improve customer satisfaction.
Usable Ease of use remains vital, and yet the interface-centered methods and perspectives of human-computer interaction do not address all dimensions of web design. In short, usability is necessary but not sufficient. Accessible Just as our buildings have elevators and ramps, our web sites should be accessible to people with disabilities (more than 10% of the population). Today, it’s good business and the ethical thing to do. Eventually, it will become the law.
Findable We must strive to design navigable web sites and locatable objects, so users can find what they need. Credible Thanks to the Web Credibility Project, we’re beginning to understand the design elements that influence whether users trust and believe what we tell them. Desirable Our quest for efficiency must be tempered by an appreciation for the power and value of image, identity, brand, and other elements of emotional design.
14
Why is UX important to a company? Over the last decades, a product’s effectiveness, intuitiveness, and ease-of-use have become tremendously important competitive differentiators. Modern customers have become more likely to switch to the competition as more companies focus on user experience and as it is becoming easier to find and switch to the competition’s products. Therefore, a modern business will find it ever more difficult to compete without optimizing the user experience of its products or services. User experience practitioners can help to give products a competitive edge by incorporating features that will most effectively meet customers’ needs and by making them easy and enjoyable to use. For certain products such as web sites, user experience designers can formulate a strategy and design that will increase key user interactions such as online purchases, viral action, and page views. Like product managers, user experience professionals generate product concepts and strategy, screen product ideas, develop and design the products, and perform beta and market testing. However, user experience professionals perform all these tasks from a usability engineering and user-centered-design approach while product managers look at the same tasks from a business and marketing angle. That is why user experience designers should work closely with product management to help define, design, and refine the final product.
15
Bad UX
Example of bad UX A while ago, I was in a store to buy some new headphones. I found a set of Philips headphones that where in my price range and the cool thing was that you could change the print on the ear caps, or even design them yourself. The set included 4 pieces of pre printed cardboard designs, so people could change the look of the caps whenever they wanted. It also offered the option to design your own ear caps, and as a designer I couldn’t wait to print my first caps. As I opened the box, I was looking for the measurements for custom made prints. Philips forgot to supply any information about printing your own headphone caps, so I went to the Philips website to look for a template or something similar. The website recommend I’d register my product first for maximum service. So I did. The first thing I needed was a product registration number, which, according to the website, was on the back of the box. After examining the back of the box three times, I gave up and chose to look for the product manually. I later found out the code was on top of the box instead of the back. I clicked ‘browse for product’ in the registration menu and the next page showed me several different categories. Now I had to guess under which category my headphones were stored. I first checked ‘Sound and vision’, then ‘Other products’ and finally ‘Accessories’. Here, I found over 350 different headphones, but I couldn’t find my model.
16
After some setbacks I managed to find my model and proceeded to the customer service, only to find out they had no template or whatsoever for my caps. I ended up scanning my default caps and editing them into my own “cap templates”. After some online research I found out I wasn’t the only one who had this problem. For me this was a bad user experience with a Philips product (headset) and customer support (website). This could have been avoided if someone from Philips would have thought ahead. When your product says ‘Design your own headset!’ on the cover of the box, you might as well make sure you supply your customers with some information. A small book containing information or a link to a website would have provided. This doesn’t mean I won’t buy any more products from Philips, but it did damage the brand for me. In my opinion, this was a onetime incident, but it could have easily been avoided. If these kind of flaws keep coming in the future, I’ll eventually stop buying products from Philips and buy them from the competition.
17
UX Process
Why create a UX process? In the past, designers mainly focused on building products that fulfilled the functional requirements of the user. User experience was often considered late in the development process. But today the customer demands more than just a working product. Providing the right features is still the prerequisite for a good product, but to turn it into something extraordinary you need to provide a good user experience. Providing a rich user experience is not a thing of fortune. It needs to be planed, designed and integrated into the development of a product. Designing a rich user experience is not only about make up your user interface by some graphics and gradients - it’s a much broader concept. It’s about creating an emotional connection between the user and your product. It makes the user feel good and so he likes to continue using the products. This is where a UX process comes in to play. Involving a UX process from the beginning of a project helps build products people want and need. You’ll create a product that’s a good fit for the people who end up using it, instead of for the developer who built it or the CEO who envisioned it.This is particularly important if your users also spend their hard earned dollars to buy your product. Adding a UX process to the production process might seem like a time and money consuming business, but it will pay out in the long run. Your team will save valuable time and resources by getting it right, or mostly right, the first time. And they’ll be faster doing it.
18
Tools & Techniques
Market leading tools and techniques In this chapter, I’ll be testing tools and techniques used to measure and improve UX. For this research, I’ve tested a lot of different tools, but I will only display the top tools that are valuable to Betterplace.org. During my search for tools, I’ve focused on unmoderated testing tools, because betterplace.org doesn’t have the time, nor the budget for moderated testing. Starting my search for the best UX tools, I’ve split up my selection of tools into two groups: quantity- and quality based tools. Quantity based tools are designed to collect data from a large number of participants, with a focus on UX metrics such as task success, task completion times, click paths, abandonment rate, etc. These are the tools you need to run a benchmark usability study or compare subtle design treatments. The other group consists of Quality based tools. These tools are essentially a substitute for lab or remote-based usability testing. These tools emphasize collecting feedback from a small number of end users in a quick and dirty fashion. You may or may not be able to actually derive any metrics, but at the very least you will gain some insight into the most significant usability issues and hopefully make the right design decisions.
Note: The tools are changing everyday in terms of functionality and pricing structure.
20
A/B Split Testing Also known as Split testing, A/B/n testing, bucket testing The basics behind A/B testing are very simple: you have an A and a B version of a website or element and test them against each other to find out which one is more successful. You split up a group of users and choose a metric which defines the success of the experiment. To determine which version is better, you subject both versions to experimentation simultaneously. In the end, you measure which version was more successful and select that version for real-world use. What to test is entirely up to you. It depends on the goals you set for your experiment. If your goal is to test a form, you might test the length of the form, experiment with taxonomy or the display of privacy. The goal in this case is to figure out what prevents visitors from finishing the form. Is the form to long? Is the placement of the fields confusing? Are people concerned about their privacy? By testing different elements of your page, you will get an insight of the elements that prevent your visitors from finishing the form. The duration of the test is also entirely up to you. You can set a period of time or a number of visitors, but if you make some radical changes to your page, you might also get some radical results from your tests. If you set your test to 1000 unique visitors and after 400 visitors you see great differences between your A and B site, you might reconsider the setup of your test.
21
A/B Testing
There are two ways to set up an A/B test: - Replace the element to be tested before the page loads: If you are testing a single element on a Web page, you’ll need to create variations of that element in your testing tool. When the test is live, the A/B tool will randomly replace the original element on the page with one of the variations before displaying the page to the visitor. - Redirect to another page: If you want to A/B test an entire page, you’ll need to create and upload a new page on your website. When the test runs, your tool will redirect some visitors to one of your alternate URLs. Every A/B test is unique and has a different goal, but there are some elements that are frequently tested:
Headline or product description Images on landing and products pages Call to action buttons Layout and style of website Form length and types of field Product Pricing and promotional offers Amount of text on the page
Once you’ve decided what to test, the next step is to select a tool for the job. I have tested a lot of different A/B split tools during my research. The results of this research can be found in the appendix of this research.
22
Example of an A/B test
Version B is better 23
Tips
Tips - Always test both (or more) versions simultaneously. Split traffic equally between these versions and test them for the same amount of time. If you test one version one month and the other version the next, your results may be false. - Don’t confuse your regular visitors. If you are testing a basic page of your website, only include the new visitors in the test. If your new version turns out to be better, you can include your old visitors afterwards. - Know how long to run a test. If you give up too early, your results may not be reliable. If you give up too late, you will have a poorly running version, which can cost you conversions. To know how long to run a test, see page 32 of this report. - Show repeat visitors the same variations. If you change your website with every visit, your visitors will be confused and are less likely to stay. Most tools have a mechanism to show one version to each unique visitor. - Make sure you use your tests across the whole website. If you’re testing a sign-up button that appears in multiple locations, make sure your visitors see the same button everywhere. - Don’t stop after one or two bad results. Chances are your first tests will return negative results. The key to optimizing conversion rates is to do lots of A/B tests, so that all positive results add up to a huge boost to your goals.
24
Popular A/B test tools Bronze Silver Gold
$ 19,-/month $ 79,-/month $ 399,-/month
(2000 viewers/month) (20000 viewers/month) (200000 viewers/month)
Optimizely.com is a simple, fast and powerful AB-testing tool. It’s easy to use interface and WYSIWYG editor are great to test different lay-outs of your webpages. Free Trial Small Business Small Agencies Large Agencies Enterprises
1 month $ 49,-/month $ 129,-/month $ 249,-/month $ 729,-/month
(10000 viewers/month) (30000 viewers/month) (100000 viewers/month) (300000 viewers/month)
Visual Website Obtimizer is much like Optimizely, but the WYSIWYG-editor doesn’t work well with the Betterplace website. Although Google Web Optimizer (GWO) is free and widely used, I wouldn’t recommend it. For starters, it doesn’t offer a WYSIWYGeditor, which isn’t necessary, but it will save you time in the long run and can be used by colleagues with minimal programming experience. Also, the coding used in GWO has to be removed afterwards, while the other programs only use one simple code snippet to be added to your website.
25
Popular Tools
Best A/B tool During my research, I compared a lot of different A/B testing tools. I compared price, features, speed and usability. The results from this research can be found in the appendix of this document. The tools on the left scored the highest results. Optimizely seems to be the best A/B split tool for Betterplace.org. The WYSIWYG-editor works great with the lay-out and the price fits well within our budget. It’s also easy to implement: just copy one line of javascript and paste it on top of your header and your ready to start testing. The user interface is very easy to use and contains the many features Optimizely has to offer. As said before, you won’t have to edit any code while experimenting, so people without any HTML or PHP knowledge can still create experiments. For those who do know their programming, Optimizely offers some advanced options. Optimizely also offers multivariate test (MVT) options, which I will cover later during my research. Free Account After I concluded Optimizely is the best A/B tools for Betterplace.org, I send an email to the Optimizely developers. I informed them about our company and asked them for a discount, due to our low budget. They offered us a free account. Therefore I would like to thank the developers and everyone involved at Optimizely.
26
Multivariate Testing Also known as Multi-variable testing, MVT Over the years our attention span for websites has decreased dramatically. According to a research from the BBC, the window to grab a visitor’s attention is about 9 seconds. So, how can you gain the interest of your visitors? What headline will you use? How much text should I use on the landing page? For these questions and more, you have Multivariate testing. Taking a first glance at Multivariate testing, it looks a lot like A/B testing, but there are some crucial differences. In a Multivariate test, a website can be seen as a combination of elements that can affect a conversion rate. These elements can be headlines, images, buttons, texts, etcetera. During a test, you create different variations for these elements and test combinations of them. Every test has a goal against which the performance of different variations is measured. Types of goals may be sign-ups, purchases, clicks, leads, page views, or bounce rate. It is important to define the goal which is closest to your business objectives. For example, an eCommerce store optimizing for sales shouldn’t define clicking on the “Add to Basket” button as a goal. Rather, it should define the goal as a visit to the “Thank you” page after a purchase is completed.
27
Multivariate Testing
Here’s an example of a MVT. Let’s assume you make the following variations:
Two Headlines Two texts Two images
You will be able to make the following combinations:
1. Headline 1 + Text 1 + Image 1 2. Headline 1 + Text 1 + Image 2 3. Headline 1 + Text 2 + Image 1 4. Headline 1 + Text 2 + Image 2
5. Headline 2 + Text 1 + Image 1 6. Headline 2 + Text 1 + Image 2 7. Headline 2 + Text 2 + Image 1 8. Headline 2 + Text 2 + Image 2
Just like in A/B testing, you split traffic equally between these different versions of the page and see which combination produces the highest conversion rate. Where an A/B test usually has a couple of variations, MVT can produce as many different variations as you want. The example above has eight different variation (2*2*2), but if you, for example, add two different buttons, you’ll have sixteen different variations (2*2*2*2). t This way the number of variations can go up pretty quick. Therefore you’ll need more traffic to get meaningful results, which your website might not have. A good rule is to limit the total number of combinations to twenty five or fewer.
28
Multivariate Testing
29
Types of MVT
Types of MVT Full factorial testing This type is the kind people generally refer to when they talk about MVT. The traffic is distributed equally among all combinations. If there are sixteen variations, each one will receive one-sixteenth of the website traffic. This way the test provides all of the data needed to determine which variation performed the best. A test might reveal a certain image had no effect on the conversion rate, while a specific headline was very influential. Full factorial method makes no assumptions with regard to statistics or the mathematics of testing. Partial or fractional factorial testing As the name suggests, partial testing only tests a fraction of variations on your website. For example, if you have sixteen variations, website traffic is only split among eight of those. For the other eight, we don’t get any conversion data. Instead, partial testing uses complex mathematics to determine the outcome of those pages. This way, fewer traffic is required, but I wouldn’t recommend it. No matter how advanced the mathematics are, hard data is always better than inference. Taguchi testing Taguchi testing is another way to get results with fewer traffic. It was originally used in the manufacturing industry, where specific assumptions were made in order to decrease the number of combinations needing to be tested for quality assurance and other experiments. In my opinion, these assumptions are not applicable to online testing. Therefore, Taguchi testing is not a theoretically sound method.
30
A/B or Multivariate test? A lot of people ask; should I run A/B test or Multivariate test? The short answer is yes. You should use both methods to test your website, but you should know how to use both of them together. Setting up new page Whenever you test a new webpage, start with A/B testing of the major elements. Run these tests early on in the design process. This way you can implement some radical changes and you can follow their impact. Use A/B tests to determine the best layout configuration of elements. After you have tested the basics of the page, start multivariate testing. Multivariate testing usually takes a lot more time, due to the higher number of variations. Use MVT to tweak the page and optimize it to get the highest conversion rate. Experts usually run eight to ten A/B tests before they run a MVT. Improving existing page If you have an improved version of an existing webpage, always test your new version versus the old. Chances may be you new page is worse than the old one and you’ll be doing even worse. If your new page checks out to be an improvement, you should repeat the process above.
31
A/B or MVT
How long should I test? It’s hard to determine how long to run a test. As tester, you probably know the situation better than anyone and you can make a pretty good estimation, but here are some thumb rules to consider while testing. Time Run a test for at least ten days and include two weekends to level out any day-of-week seasonality. Fluctuations in conversions can occur depending on the day of the week. It’s also recommended to re-run a test a different time, to understand any long term seasonality effects. Number of visitors Even if you have a clear winner after twenty conversions, wait until you have at least fifty conversions to make any decisions. Chances may be you’ve been lucky and the rest of your visitors may not react the same. Statistical difference Run experiments until a statistical difference at the 95% confidence level is achieved. Statistical difference can be measured online or with a excel sheet which can be downloaded at tinyurl.com/statdif.
32
Heatmaps Heatmaps are graphical overlays of your website which point out what content is hot and what not. This is mainly done by tracking the mouse clicks of the visitor. Heatmaps provide you an insight for sensitive areas on your website which you can use to provide better placement to core content or promote product or information you are willing to highlight on your website to increase online sales, etc. For example, the heatmap of a webpage shows you an advertisement is easily overlooked and therefore less clicked. There can be a number of reasons why it’s not getting the attention you want it to have. By changing the advertisement or giving it a different location, you can improve the number of clicks and find out what was wrong with the original. Heatmaps give you the opportunity to look through the eyes of your visitors as they navigate on your website. Heatmaps help designers and marketers discover which areas of pages are more successful than others. Consumers tend to follow the same pattern when they view pages, so correctly analyzing where these areas are will increase the potential success of the page. Analyzing heat maps allows you to increase your conversion rates.
33
Heatmaps
Heatmap Software The ever growing offer of heatmap software makes it hard to choose the best program for the job. New software seems to make its way to the web daily, all with different price range and extra features. The software that popped out for me was Mouseflow. First of all, Mouseflow is very cheap. Next to the free version (100 visitors) it also provides a basic account (1000 visitors) for 10 euro a month. Compared to the other advanced software, this is very cheap. Mouseflow has many features. Not only does it offer a detailed mouse click heatmap, it also a mouse movement and scroll heatmap. This is backed up with a powerful web analytics package that give you lots of statistics and features to play around with. It records the pageviews, total visit length, interaction time, bounce rates, loading time, response time and the scroll reach. On top of that Mouseflow offers livestreaming, a very nice feature with which you can follow your users live. It also records whole visitor sessions, which you can playback in your browser or simply download to your hard drive to watch it whenever you want. These recordings include the mouse movements, clicks, scroll events and key strokes. Finally, the software is easy to implement. Just insert one line of Javascript and you’re ready to start testing.
34
There’s no such thing as an
AVERAGE USER
35
Personas
Personas There is no such thing as the average web user. Every user is unique and has his or her own interpretation of surfing the web. But we still want to make our website accessible for every user. This is where personas are very useful. A persona is a fictional profile of a random user, which exists from relevant contextual information, based on a product or service. Personas are used to represent the different user types within a targeted demographic, attitude and/or behaviour set that might use a site, brand or product in a similar way. They can loosely be based on customers, friends or family members. This way a persona becomes more realistic and easier to implement. It can also be based on a character from a movie or a play. In most cases, personas are synthesized from data collected from interviews with users. They are captured in short descriptions that include behaviour patterns, goals, skills, attitudes, and environment, with a few fictional personal details to make the persona a realistic character. Most personas include a portrait to make the character as 3D as possible. For each product, more than one persona is usually created, but one persona should always be the primary focus for the design. Personas help prevent “self referential design� when the designer or developer may unconsciously project their own mental models on the product design which may be very different from that of the target user population. Personas also provide a reality check by helping designers keep the focus of the design on cases that are most likely to be encountered for the target users and not on edge cases which usually won’t happen for the target population.
36
Observing people using a design will always trump an inspection
37
Heuristic Evaluation
Heuristic evaluation In 1990, Jakob Nielsen and Rolf Molich published the heuristic evaluation: ten basic principles (heuristics) that make sure a user interface is usable for its users. When multiple evaluators apply them to any UI, it should reveal gaps in the design of the UI that could cause problems for users. Someone who had never seen the target UI before, and who was not necessarily knowledgeable about the domain, should be able to determine whether any UI complied with these ten commandments of usable UI design. When a group of people do this separately, they come up with a list of problems. They then compare these problems and hand a list off to some engineers to go fix it. A heuristic evaluation provides a designer with fast, cheap and easy results. It can be done early in the development cycle and you only need a couple of testers, which can (partly) be novice users. Also, it doesn’t require any advanced planning. Although the method is still used, it receives a lot of criticism. The method seems to be an inspection, rather then an evaluation. It is not about the user experience. The reviewer judges the UI against some established rules of thumb instead of own personal experience/opinions on interface design. The evaluator does not perform any task or evaluates any flow as such, rather he just apply usability principles to find out usability issues. Heuristic evaluation may help a team know whether their UI complies with someone else’s guidelines. But observing people using a design will always trump an inspection or audit for getting solid evidence to determine a design direction.
38
Low Budget Testing As a designer you won’t always have a budget. Even larger projects can often lack a dedicated budget for user experience testing. Maybe the client doesn’t think it’s necessary, maybe the budget is really tight, or maybe there are other reasons. So you will have to get creative to test your work. It doesn’t really matter how much you know about user experience, after working on a project for a couple of weeks you’ve become blind to many of the problems. Sure, experience will help you avoid making many design mistakes in the first place, but something will slip through. Every time. That’s why doing even a very basic usability test will improve your site. Here’s an overview of do it yourself techniques you can use, which are absolutely free. A quick warning: these techniques might work in theory, but they won’t always work in practice. That’s why I’ve added my experiences and pro’s and con’s.
39
Low Budget Testing
Office Wall Location: Betterplace Office Duration: 30 minute setup For this test I’ve set up a wall at the coffee machine in our office. Here, I hung up a big sheet of A0 paper and put up 3 mock ups and a couple of markers. I then send an email around the office, asking them if they could look at and comment on my mock ups, while waiting for their coffee. Results The office wall test was a great way to get a lot of information and feed back without doing a lot of work. Setting up the experiment only cost me about half an hour and the next day I got a wall full of suggestions from people who know the website very well. This might be the only downside. The people in the office have seen the website over and over. Some elements that are very obvious to them might be confusing to people who visit the website for the first time.
Pros - Lots of great feedback in short period of time - People are willing to give their honest opinion - People benefit by giving useful feed back
Cons - People might know the site too well
40
Surveys Location: Free University of Berlin Duration: 5-10 minutes per survey The downside of online surveys is that you can’t interact with people in any way. Experience shows that most of the people will only give half answers and you will not have a chance to evaluate their survey. That’s why I set up a short real life survey, which contained 3 short assignments. The first and second assignment were about the Betterplace. org landing page and website and the third one was a short scenario in which they had to choose their favourite mock up. By fitting all 3 assignments on 1 peace of A4 paper, people wouldn’t be scared off by the length of the survey. Results I chose the Free university of Berlin, because I thought people there would respond positive to a short survey, but it turned out I was wrong. The students at the university weren’t eager to spent 5 to 10 minutes of their time on a survey, even if I told them it was for charity and it would help me a lot for my research. Others we’re busy, got mad or simply ignored me. Also, the interviews could only be done with the use of a laptop. This meant I had to look for people with a laptop, who weren’t busy studying, but even students who weren’t studying weren’t interested. I was aiming for 25 surveys, but I only got to do 7. If you’re planning on doing field surveys in the future, choose a location that you know well with people you know are open to help you. Pros - Quick results - Option to ask extra questions
41
Cons - Hard to find people willing to spent some of their time
Low Budget Testing
Street Research Location: Schlesische Strasse Berlin Duration: 1 minute We had a deadline coming up for the donations page and we had to make a decision about the lay-out, so time was of the essence. Therefore, I decided to go out on the street and ask people for their opinion. I printed a page showing a charity raising money for the Japan tsunami incident. Then I asked people to pretend a friend sent them a link to that page and asked them to donate money. The attendant would agree and “clicks” the donate button. After that I showed them three mock ups and asked which one was the clearest and the most logical to them. Results It’s not possible to ask people for a lot of information. I told people my survey would take a minute tops, but still only one in six people were willing to spend this time. Others were in a hurry or not in the mood. Some people even walked out during the survey. The other problem with this sort of research is results aren’t always accurate. During my research I tended to ask people why they chose that mock up and what they liked and hated about the others. Someone could chose mock up 3 because it looks nice, without considering the taxonomy of the information given on the page. By asking people for comments, I came up with an advice that didn’t fit the number of votes. Pros - Quick way to get a lot of opinions
Cons - Number of votes not always accurate
42
Tools & Techniques used by market leading companies 43
Other Companies
Techniques used by big companies Many companies use the designed experiment approach to making marketing decisions. It is an increasingly common practice as the tools and expertise grows in this area. There are many UX testing case studies which show that the practice of testing is increasingly becoming popular with small and medium businesses as well. Market leading companies known for UX testing include: Amazon.com BBC Google Microsoft Playdom (Disney Interactive) Twitter Zynga eBay Wikipedia While these companies are known for using behind the scenes tests to maximize profits, the practice occasionally makes it into the spotlight. The next pages include UX tests done by market leading companies who published the test setups, and sometimes results. This will give you an insight of how these companies are using tests to improve their product, service or system.
44
I can’t fault Google for this reliance on data. And I can’t exactly point to financial failure or a shrinking number of users to prove it has done anything wrong.
45
- Douglas Bowman
Google’s 41 Shades Experiment Google’s 41 shades experiment is one of the most known split test stories found on the internet. The story goes as follows: a team of designers at Google had to pick a shade of blue, so they picked a shade they all liked. But then a product manager tested a bunch of different colors and found out users were more likely to click another shade. And for Google, more clicks mean more money. So Marissa Mayer, Google’s vice president of search products and user experience, decided to test 41 different shades of blue. Incoming users were randomly assigned to pages with different coloured links, from blue-with-green to blue-with-blue. It turned out that people were more eager to click a blue-with-blue shaded link then a blue-with-green one. The result: the moment this story was posted on the internet, people were debating the necessity of testing 41 different shades of blue. Douglas Bowman, a very talented visual designer and manager at Google, even quit his job. He couldn’t live with Google’s design philosophy that strictly revolves around data. At the other hand: Google is one of the biggest companies in the world, with millions or users and billions of shareholder dollars at stake. It’s no easy task to please all of them. In this case it might be better to rely on data then the opinion of a group of designers. As trivial as color choices might seem, clicks are a key part of Google’s revenue stream, and anything that enhances clicks means more money.
46
Wikipedia Annual Fundraiser Banner Results $ 582.21 “Thanks for the brain message“ $ 2,901.08 “Admit it - without Wikipedia, you never could have finished that report. Click here to keep Wikipedia free for future students $ 2,992.91 “You depend on Wikipedia for information. Now it depends on you“ $ 24,274.21
“Please read: A personal appeal from Wikipedia founder Jimmy Wales“
x 1000 $
47
Wikipedia
Wikipedia Fundraiser Wikipedia is an online encyclopaedia which is created by people from all over the world. To keep Wikipedia running, they need money in the form of donations. That’s why Wikipedia organizes fundraisers to collect money. In 2010, the company came up with this great idea to have users and contributors make up banners and test which ones were most effective. Wikipedia decided to publish their results online.
The banner above, showing Wikipedia founder Jimmy Wales, won by a landslide which is not that big of a surprise. Seeing the founder appeal for a donation creates a very personal feel. User Quote:
Funny enough Jimmy’s ad got me to give them money. Two factors came into play, the personal touch from the founder and the fact that Wikipedia is a free service. Seeing the actual founder “appeal” directly to you felt a lot more personal, almost creating a sense of guilt for the avid Wikipedia user. The other two ads were from the Wikipedia organization instead of a human being. In the words of Seth Godin: “Humans like humans. They hate organizations.” This type of ads works very well for a free service like Wikipedia, but the same ad would not have performed as well for a for-profit corporations, since people do not relate to CEOs.
48
Twitter Sign Up Tests Over the years, there’s been a lot of research for the best text to put on a call to action buttons. For example: a button with ‘sign up’ can have a lower conversion rate than ‘sign up for free’. When Twitter started, they used five totally different texts.
Give it a Try: Emphasizes a low commitment barrier. Sign-Up: Descriptive, uses common term so to trigger the action. Also consistent with the term “sign-out” which Twitter also uses. Let Me In: Emphasizes being an outsider, possibly to try and trigger feelings of being left out so as to motivate to take the action of getting “in”. Get started now: “now” emphasizes immediacy, while “get started” suggests there is a larger process at hand. Join Today: Also emphasizes group activity, that by signing up you are “joining” the Twitter community. While Twitter was running this test, the other parts of their pages remained the same. This gives them cleaner test results. If you have fewer variables on your pages, you will have more reliable results.
49
Obama
Presidential MVT During the campaign for the 2008 presidential election in the United States of America, Barack Obama and his team used split testing to get people to sign up in their mailing list. The set up was simple: the page contained a media area, for either a picture or a video and a call to action button, in this case, enter your email address. The team used 3 pictures, 3 video’s and 4 different buttons. This means they had 3*3*4 = 24 different combinations. These tests kept running, until every combination was seen by 13.000 people. The team then analyzed the results and came up with the perfect combination of picture/video and button. This combination was used for the rest of the campaign. Dan Siroker, as part of the Obama team and CEO of Optimizely, claimed that the unoptimized page had a sign up rate of 8.26%. After they ran the tests and optimized their page, the sign up rate was 11.6%, and improvement of 40.6%. This improvement led to an additional $60 million in donations for the Obama campaign. Crazy as it sounds, the 2012 election could be decided by who makes the best use of website optimization.
50
Conclusion There are a lot of new and innovative tools and techniques on the market that can have a big influence on your website. New tools are developed every day en existing ones have a growing amounts of extra features. A/B testing A/B testing is used to gain insight into visitor behaviour and to increase conversion rate and has been growing for the last couple of years with no end in sight. The simple idea of testing version A versus version B boosted conversion rates for dozens of websites. Even market leading companies use it to increase their traffic. A/B testing software nowadays is available for everyone and is going to play an important role in web design and marketing in the future. Multivariate testing In a multivariate test, a webpage is treated as a combination of elements that affect the conversion rate. By testing different combinations of elements you can tweak your webpage into the perfect combination. In combination with A/B testing, this can be a great technique to improve conversion rates at Betterplace.org
51
Conclusion
Heatmaps Heat maps help designers and marketers discover which areas of pages are more successful than others. Properly analyzing heat maps allows you to increase your conversion rates. Heatmaps are a good way to get an insight of visitor behaviour and can be of great value to Betterplace. Personas By creating and using personas at an early stage in the process, you can get rid of flaws in your design. Personas are cheap and very useful and can be a great addition to Betterplace. Heuristic Evaluation I’ve included the Heuristic evaluation to this research, because the term is widely used by User Experience researchers and company executives. While testing this method, I realized it’s pretty much outdated. The evaluation has been around since the beginning of the 90’s and although it’s been updated multiple times since, it doesn’t apply today. Low Budget Testing Especially the office wall test showed to be very helpful during the design of new or improvement of existing pages for the Betterplace website. As a designer, you’ll never know for sure how visitors will react to changes. By involving the experience from the other office workers in the process, you can get rid of flaws in an early stage.
52
Advice
Advice and Recommendations Now I know which tools and techniques are valuable for Betterplace and which aren’t, I can add them in the design process. Before I started my research, the design process at Betterplace looked like this:
As you can see, there is no process concerning the visitors. If a page is rebuild, it is deployed and the only way to check if it’s better or worse than the old one is to wait for the statistics. With the tools and techniques from my research I can improve the design phase and add a testing phase in the process. This way we can measure if a new page is actually better than the old one.
54
Design Phase At this moment the design phase is a one man job with minimal feedback. The designer makes the decisions the way he thinks is best. Therefore I recommend to include other people into this process without keeping them from their jobs.
Office Wall Test The office wall test is a great way to share your design with the rest of the office and get quick feedback, without bothering your colleagues. There are a lot of people in the office who have a very strong opinion about the Betterplace website. Not only in product management and development, but in every department there are colleagues with lots of experience. The office wall test costs nothing and is really easy to set up. After one day, you will have a wall full of suggestions and feedback from the people who have known and worked with the website for years. Personas Another good way to find flaws early in the design process is by using personas. These can be created or found online. By imagining another person performing a task on your new design, you can find confusing parts and change them.
55
Advice
Test Phase To really measure the impact of a new page, I advice to implement a testing phase, which includes A/B testing and Multivariate testing.
A/B Testing Even if you think your design is perfect, there will still be some parts that made you doubt. Just test these parts with an A/B test. Changes may well be that you’re wrong and visitors won’t react the way you expected them to. With A/B testing you’ll have the ability to isolate and assess the performance of virtually every element of a web site. From page layout, to headline text, to complete color palette variations, A/B can be highly effective in identifying what works and what doesn’t from a user’s perspective. It can be easily scaled and deployed with no regard to internal technology infrastructure. The reaction of your visitors is something unpredictable. Even the most professional interaction designers have it wrong sometimes. By testing as much as possible you can be sure of your case.
56
Always test new versus old Whenever you improve a page, you should always test the new version versus the old version. Even if you have the statistics for the old page, they may be influenced by elements beyond your control. Your new page also might not be as good as you think and you’ll end up with less then you started. Make sure only new visitors will see the new page. This way, your returning visitors won’t be confused by changing parts of your website. Until your new page turns out to be an improvement, you can introduce it to your returning visitors. Multivariate Test After you have tested your basics with A/B tests, it’s time to tweak your page with Multivariate tests. This means headlines, pictures, texts, etcetera. By testing every combination, you will see which works and which doesn’t. Even the smallest change in your page can lead to an increase of your conversion rate. But be warned, with every extra combination you make, you need extra visitors. Betterplace has an average low six digit number of visitors per month, depending on the time of the year, so you should be able to make an estimation for testing. Try to include other departments in the process. Al lot of colleagues have a very strong opinion about the website, especially in marketing. A simple email with suggestions can help a lot. By analyzing the results, you can optimize pages and improve them.
57
Advice
Heatmaps Heatmaps visualize the stream of visitors on your website, showing hot and cold click zones. They provide an overall view of the activity and helps you to see every mouse movement and every click. If used in the right order, it will improve web usability, conversions and revenue. A heatmap can be very useful in designing your page, and in how to arrange the contents of each individual web page. It can provide you with an understanding of how people travel through web pages, and what parts of each page they tend to look at first. If you know the areas of your web page that visitors tend to focus on when they first arrive, you can design that page to suit. It is also useful to understand the areas that they tend not to look at. Heatmaps can be used in combination with A/B testing and Multivariate testing. If you know where your visitors tend to go first, you might want to experiment with that part of the page. If you see which parts of the page aren’t used at all, you might want to improve that part of the page and see the results.
58
Recources
Recources Books Bowles C., Box J. - (2011) Undercover User Experience Design - Berkeley, New Riders Cagan M. - (2008) Inspired: How To Create Products Customers Love - California SVPG Press Corbone G. - (2011) Simple and Usable, Web, Mobile and Interaction Design - Berkeley, New Riders Krug S. - (2010) Rocket Surgery Made Easy: The Do-It-Yourself Guide to Finding and Fixing Usability Problems - Berkeley, New Riders Wroblewski L. - (2008) Web Form Design: Filling the gaps - New York, Rosenfeld Media
60
Recources Articles Managing UI Complexity by Brandon Walkin Lean UX: Getting Out Of The Deliverables Business by Jeff Gothelf How to Shop for Unmoderated Usability Testing Tools by Bill Albert Forgotten principles of usability testing by David Travis Pioneering a User Experience (UX) Process by Amy Hillman 10 Usability Tips Based on Research Studies by Cameron Chapman User Experience Design by Peter Morville’s 10 Most Common Misconceptions About User Experience Design by Whitney Hess Multivariate Testing 101: A Scientific Method Of Optimizing Design by Paras Chopra Multivariate Testing in Action: Five Simple Steps to Increase Conversion Rates by Paras Chopra The Ultimate Guide To A/B Testing by Paras Chopra by Jacob Gube A/B Split Testing vs. Multivariate: Pros & Cons by Chris Goward Case Studies & Success Stories by Virtual Website Optimizer My big list of 24 Web Site Usability Testing Tools by Craig Tomlin Building a Data-Backed Persona by Andrea Wiggins Web Analytics Tools: Where to Begin? by Lynne Usability Expert Reviews: Beyond Heuristic Evaluation by David Travis
61
User experience design: Persona by Michel Kuik Picking the right tool for remote user testing by Matt Milosavljevic Putting a Bolder Face on Google by New York Times Visualize This: Winning Wikipedia fundraiser banners by Flowing Data Obama’s site is performing a test on you by Daryl Lang Twitter A/B testing sign-up button by Joshua Porter
Websites www.smashingmagazine.com www.johnnyholland.org www.uxbooth.com www.boxesandarrows.com www.wikipedia.com www.visualwebsiteoptimizer.com www.uxmag.com www.conversionvoodoo.com/blog/
62
A/B Tools Test Results
Note: The tools are changing everyday in terms of functionality and pricing structure.
63
Heatmap Test Results
64
Other tested software This is a list of tools and techniques I also tested, but didn’t make the research, because they turned out to be too expensive, time consuming or just not fit for Betterplace.
65
Video Tools Expert Reviews Online Surveys Eye Trackment Neuro Marketing Full Service Tools Card Sorting