Handbook for usability research A guide to product evaluation through end user feedback
1st edition June 2010
1
Table of contents 1. Introduction ....................................................................................................................... 3 2. Evaluating product design.................................................................................................. 4 3. Usability research .............................................................................................................. 5 4. Personas ............................................................................................................................ 7 5. Beginner’s mind ................................................................................................................. 9 6. Ethics and etiquette ......................................................................................................... 10 7. Interviewing .................................................................................................................... 11 8. Observation ..................................................................................................................... 14 9. Usability testing............................................................................................................... 16 10. Working with usability findings ...................................................................................... 18 11. Usability research or surveys? ........................................................................................ 20 12. Alternative methods of obtaining feedback .................................................................... 22 13. Integrating usability into procurement workflow ........................................................... 23 14. Engaging users in usability research ............................................................................... 24 15. Innovation business model ............................................................................................. 25 15. Conclusion ..................................................................................................................... 28 16. Terminology list ............................................................................................................. 29 17. References ..................................................................................................................... 31
1st edition June 2010
2
1. Introduction This publication supports UNICEF Supply’s commitment to continuous improvement so that we do better for children all the time. To know if a product design and specifications are suitable and effective requires an understanding of the geographic, economic, and social contexts in which they are used. Feedback from end users is essential to evaluate whether a product is fit for purpose. This information is useful for UNICEF staff responsible for procurement. But what are the skills, methods and tools needed to obtain such feedback? This handbook was developed as a staff resource following UNICEF staff training on skills and methods for usability research initiated in 2009. It provides practical tools and techniques to gain detailed and more reliable end user feedback for product evaluation and development. The usability research methods described in this handbook will help staff make decisions based on a greater understanding of the needs and priorities of the people who use UNICEF procured commodities. This handbook does not cover the technical, environmental, or regulatory aspects of product evaluation. Instead it focuses on evaluating products from the viewpoint of Country Offices, regional programmes, procurement services partners, health care workers, mothers, children, and others who use UNICEF supplies. Four usability methods are described within the document: personas, interviewing, observing and usability testing. The handbook also covers suggestions on how to conduct usability research when field research is not possible, makes recommendations for engaging country level staff in usability research and provides guidelines for choosing between surveys and usability research. The handbook is intended for internal UNICEF use. However it could be useful for broader audiences interested in usability research as well. The handbook was prepared with the valuable contributions from the following groups in UNICEF Supply Division, Copenhagen: Technical Function Group; Innovation Unit; Quality Assurance Centre; Medicines and Nutrition Centre; Water, Sanitation and Education Centre and Knowledge Management Unit. Enquiries and comments are welcome. Please write to: Atieno Ojoo, Technical Specialist, Pharmaceuticals, UNICEF Supply Division aojoo@unicef.org.
1st edition June 2010
3
2. Evaluating product design When you evaluate a product, examine the design and focus on whether the design is appropriate for your particular end users. This is a user-centred design approach to product evaluation. User-centred design optimises products around how people work, rather than forcing people to change the way they work to accommodate the designer’s approach to a product. People will more easily understand how a new product works if it contains the following design elements: Visibility Visible cues should inform the users about what to do with the product and what actions are available to them. Does the item need to be assembled? Is it clear how a particular medicine should be taken? Strong conceptual model The product should, as much as possible, utilise the conceptual models that are familiar to a particular group of end users rather than introduce new conceptual models. Good mapping When someone encounters a new product, two concepts should be clear: the possible actions available and the results of each available action. Good mapping ensures the relationship between these two concepts is apparent to the user. Feedback As users work with a product, they should receive feedback that indicates that the product is working correctly. For example, people should be able to see from the picture on the instructions whether or not they are putting up bed nets correctly.
The problem of egocentrism in product design Usability research was developed to combat the natural egocentrism that can be present in product development. Designers, developers and engineers develop their own subculture with their own jargon that can lead them into the trap of designing products for themselves. In a similar way, technical staff runs the risk of evaluating these products through their own lenses. Usability methods can help us see products from a user-centric perspective. These methods will be further explained below. When UNICEF procures products, staff have to evaluate against the product’s technical requirements, the customer’s performance requirements, available normative guidance and standards etc. Usability research ensures that the products have been considered from the user’s point of view.
1st edition June 2010
4
3. Usability research To complement traditional research, many companies are turning to usability research based on ethnographic methods. Traditional customer research relies on surveys, focus groups and interviews, however these methods are limited because people: • Do not always respond to surveys. • May not be consciously aware of how they use a product. • May not remember problems with the product. • Might be hesitant to criticise products they have received for free. • Have difficulty expressing themselves. Usability methods based on ethnography help us understand the user’s worldview. Usability research yields richer and more reliable information than some more traditional methods.
Sample size: Four to six users Ethnography relies on smaller samples and longer interaction time than surveys. Deep understanding, not broad coverage is the strength of qualitative research. According to usability pioneer, Jakob Nielsen, a sample of 4 to 6 users is adequate to gain valuable insights into a product from a user’s point of view.
Prioritising products for usability research If you are responsible for a large number of different products, how do you decide which products you should prioritise for usability research? These three factors can help you prioritise: 1) What are the potential consequences of the usability issues? If a product has the potential to harm a user, then it should be prioritised. For example, a container for disposing injection needles might be a good candidate for usability research to ensure safe needle disposal. 2) How complex is the product? The more complex the product, the greater the chance that usability issues will occur. and therefore, the more likely to require usability research. For example, a midwifery kit should receive priority over a blackboard. 3) What is the product’s history? If a product has a history of successful use then it is probably not necessary to prioritise usability research. However, if a product is new or has a history of usability issues, it is a good idea to prioritise usability research.
1st edition June 2010
5
Dimensions of usability A product’s usability should be assessed against the following requirements: Easy to learn: For example, can a health care worker with little or no training use this particular medical device? Efficient to use: For example, does the lid to the cold box still fit when the ice packs inside are frozen? Easy to remember: For example, can an aid worker remember how to put together resuscitation kit quickly in an emergency? Few errors: For example, does the labelling on a pharmaceutical product make errors more likely? Subjectively pleasing: For example, does the malaria medicine have a pleasant or bitter taste?
Human-centred design Another way of evaluating a product or service is by using the three lenses of human-centred design: Desirability: Do people in developing countries want the product? Feasibility: Is the product technically and organisationally feasible? For example, can you find a supplier who provides the specific product that the customer has requested? Viability: Is procuring this product financially possible for the UNICEF supply function? A company in California called IDEO Design specifically developed these lenses for NGOs who provide products and services to developing countries. To evaluate a product, IDEO recommends looking through the following human-centered lenses: The product or service that emerges at the point where the three lenses overlap should be the best solution for your procurement challenge. The five dimensions of usability and the three human-centred lenses offer a way to evaluate commodities with UNICEF constituents in focus.
1st edition June 2010
6
4. Personas “Personas are bright lights under which we do design….a powerful, malleable tool to help you look through the eyes of your users." Alan Cooper, author of The Inmates Are Running the Asylum
Working with personas Personas are fictional representations of end users designed to keep designers focused on the user’s priorities and needs. In procurement, personas can help you predict which products will work best for your end users. It is much easier to make decisions about products based on an individual then on an anonymous group. To develop and work with personas, follow these steps: 1) Define your user Interview your typical users and find out how they use the product. Take note of any problems they might have, where they are using the product and how often they are using it. Your data should include quotes that capture the user’s personality, demographics, lifestyle, goals, values, attitude and level of competence with the product. The more people you interview, the richer the detail of your persona. 2) Create a persona Read through your interview notes and synthesise the data into one fictional person who shares the common characteristics and behaviours of your typical end user. Give your persona a name and add a photograph. Then write a 2-3 page description which describes the personal story of this typical user. 3) List key attributes Read your persona description and analyse the key attributes of your persona. Look for characteristics that could affect the product you are evaluating. 4) List user requirements Make a list of user requirements for this product based on your persona’s key attributes. 5) Evaluate products Use your persona’s user requirements to guide you as you evaluate the products available to you from your suppliers. The below example illustrates how personas can be used. In this example, the water bottles, which Ali uses to collect water from the local well, will be evaluated.
1st edition June 2010
7
Persona description Ali is nine years old and lives in a small village in Gambia with his parents and two older sisters. He collects water from the local well each morning before he goes to school. Sometimes he stops along the way to play football with his friends. “I don’t mind getting water,” says Ali, “I feel proud that I can help my family. I like to get to the well early before the area gets wet and muddy. The mud brings too many mosquitoes.” Ali walks half a kilometre to the local well. He places the water bottle under the water pipe and pumps the handle up and down to draw the water. He carries the full bottle home on his head and delivers it to his mother so she can begin cooking breakfast.
From this persona description, Ali’s key attributes emerge: • • • • •
He is nine years old. He carries water from a nearby well to help his mother. He sets the water bottle down to fill it and to play football. He carries the water bottle on his head. He dislikes it when the area around the water pump gets wet and muddy.
If we look at Ali’s key attributes, we can predict his requirements for the water bottle. The water bottle that Ali uses has the following user requirements: • • • • •
It should be able to stand upright so Ali can place it under the faucet while he pumps the water. The size of bottle should be based on how much water a nine year old can carry. The bottle must be designed to be carried on the head. The bottle can be closed with a lid so it is not knocked over. The mouth of the water bottle should not be smaller than the opening of the pipe so the water does not spill.
Personas and the supply chain You can create personas for each user in UNICEF supply chain: the country office or regional program, the shipping agent, the health care centre and the final end user. This ensures that user requirements are considered from end to end of the supply chain. If you develop or procure a new product, you can use several different personas to compare and contrast different requirements, which can help you evaluate the product across the supply chain. For example, a collapsible water bottle may meet the needs of the shipping agent persona but may be completely unsuitable for the Ali persona. Personas help you evaluate products because they humanize your end user, help you define user requirements and provide a shorthand for communication within your team or organization. 1st edition June 2010
8
5. Beginner’s mind “In the beginner's mind there are many possibilities, in the expert's mind there are few.” Shunryu Suzuki Sōtō Zen Spiritual Leader When technical staff at UNICEF works with their constituents, they may face language, cultural and socio-economic differences. One way to overcome these differences is to adopt the Beginner’s Mind. Beginner’s Mind refers to having an attitude of openness, eagerness, and lack of preconceptions when interviewing or observing a subject. It is often difficult for experts and professionals to put aside their expertise when they conduct usability research. Therefore, learning how to work with a Beginner’s Mind is a good tool to overcome this difficulty. The goal of developing a Beginner’s Mind is to conduct interviews or observations without interpreting what you see or hear based on your own cultural and professional preconceptions. For example, to work with a Beginner’s Mind, you should ask questions as if you knew nothing about the product or how people were using it. It is important to ask questions that you think you might already know the answers to so that you can hear people explain. Their explanation may reveal their unique perceptions about a product, which may challenge your assumptions.
The Lens of Personal Experience Observing through the lens of personal experience can lead us to misinterpret situations outside our culture. For example, on a Navajo reservation in Arizona, a teacher makes a home visit to one of her students. However, each time she drives up to the student’s home, the family hurries inside the house and closes the door. The teacher assumes that her student and his family dislike her. The Navajo family thinks the teacher is making fun of them by pretending to come for a visit and then driving off. Both parties have interpreted the situation based on their own cultural prejudices. In Navajo culture, when visitors arrive at your home, it is polite to go inside the house to prepare for the guests. According to Navajo etiquette, it is important to close the front door as you prepare for the visitors so they have the opportunity to knock on the door and be received and greeted properly. Developing a Beginner’s Mind will help you avoid this type of cultural misunderstanding.
1st edition June 2010
9
6. Ethics and etiquette Usability research can be tricky. Setting up an interview or observation properly is the key to successful usability research. Observing a few simple guidelines before and during your usability research will help you establish the open rapport you need to gain a deeper understanding of your end user. Before your research:
Always get informed consent. Explain the purpose of your research to the user. Let the participant know how much of their time you will need. Get permission first if you intend to take photos and video or make an audio recording. Let the participant know you will be taking notes. Maintain the anonymity of the participant.
When you first meet your research participant, introduce yourself and start by letting the person know why you are conducting this research. Let the participant know what you will do with the information you gather. Make it clear to the participant that you are there to learn from him or her and that you are grateful for his or her participation. If the people you are observing are sitting, you should sit down too. Your goal should be to make the end user as comfortable as possible and help make them understand that their contribution and thoughts are important to you and UNICEF. Make the end user the expert When you conduct an observation, emphasize to participants that you are not evaluating them, you are evaluating the product. If they make an error while using the product, this gives you an opportunity to learn how the product can be improved. Think of any mistake that the participant makes as an error of design rather than a "user error”. During the interview or observation, you should be mindful of the uneven power dynamic between an organization like UNICEF and their constituents and work to minimize it. Do not wear branded clothing because this will only call attention to the hierarchical element of the relationship. When you introduce yourself, do not use job titles. During your research: • • • • • •
Always stop an observation or interview if the participant is upset or asks to stop. Maintain anonymity of the person you are observing or interviewing. Do not lie to participants. Stay neutral; do not laugh or express confusion at what participants say or do. Always thank people after the interview or observation session. Don’t correct people; try to understand their perceptions and why they may perceive things differently than you.
1st edition June 2010
10
7. Interviewing If you can get people to tell stories during an interview instead of just answering questions, you have a better chance of discovering what is important to them. Stories help you get past the limitations of surveys and other traditional customer research methods. Think of interviewing as inviting people to participate in the procurement process. Your users have an expertise that you can leverage. They Better understand the real environment that the product is used in. Interviewing someone usually takes about an hour. The interview should be a long, relaxed conversation that allows the participant to tell stories about their experience with the product. Prepare a list of questions for the interview but be flexible enough to stray from them if necessary. By allowing long pauses in the conversation, you give the user a chance to think and come up with more to say. The information gathered from stories may be more reliable than the rehearsed answers that people sometimes give in the context of uneven power dynamics. When you interview, be engaged and attentive. Focus on what the interviewee is saying. Listen and show a genuine interest in them and what they are saying. The goal is to create an atmosphere in which feels relaxed and is willing to open up. You want to make the conversation about the interviewee rather than about the product or UNICEF. Listen for “exclamation points” in the conversation. When people show their feelings, whether frustration or joy, it usually indicates that something is important to them and therefore important to your research.
Interviewing techniques Here are a few techniques for collecting strong stories in an interview: Start by building rapport Start with comfortable, specific questions which are easy for the participant to answer. Ask basic questions that will help you understand their overall life situation, their age, the number of people living in the household, their daily activities using the product, their work and any health problems they may have. Ask naïve questions Even if you are an expert on a product, it is important to ask questions that you may already know the answer to. Hearing users explain things in their own words will provide you with insights into their needs and priorities. Ask open-ended questions After you have established a rapport with the user through basic questions, you can ask openended questions that will generally provide you with more detailed responses. A sample open-ended question might be: “If you could change something about the product, what would that be?” 1st edition June 2010
11
Draw it Drawing is a good method for overcoming language barriers or other communication obstacles. The technical is particularly useful when speaking to children. By asking the participant to illustrate his or her experience with your product by drawing; you can better understand the process or the order of the participant’s activities. Use scenarios Some sensitive subjects like money or illness, are best approached indirectly. Try creating a hypothetical scenario to help the interviewee. For example, new medicines often involve risks. However, asking a patient what he or she is willing to risk may be too difficult for the patient to answer. Instead, try presenting two scenarios from which the participant can choose. You might ask a patient at a health care clinic the following: “If you had a choice between two medicines that could improve your health, which one would you prefer? The first medicine has no known side effects, but it will take you three months to see any improvement in your condition. The second medicine can cause nausea and fatigue, but it should improve your condition dramatically in just one week. “ Choosing between two scenarios often makes it easier for people to conceptualise their options and thereby make more informed decisions. The five whys When you conduct an interview, it is a good to frequently ask “why?” because it uncovers people’s deeper reasoning. Asking “why” to five consecutive answers leads people to express the reasons for their behaviour and attitudes. This technique may be suited for more complicated products and services. When you are working with simpler products, your research may benefit from just two or three “whys?”
Tips for Interviewing Children Children have great potential to provide valuable and especially honest feedback about products and services. Here are a few tips to help you get the best out of an interview with a child. • Take a little extra time to make the child comfortable. Some children are afraid of new people so it helps to take time to get to know them. • Consider the potential consequences to the child in their community if you are soliciting their opinion about sensitive matters. • Cameras and tape recorders can be new and frightening to a child so always ask for the child’s consent before using them. • Explain slowly and carefully why you are doing the interview and what you will do with the information you gather. • Use short sentences and simple words during the interview. • As much as possible, use peer translators when you interview a child who speaks a language different from yours. • Don’t ask questions that could reactivate his/her grief or trauma. 1st edition June 2010
12
Questions to ask end users Start with simple, comfortable questions to build rapport. Here are some questions: • • • • •
How many people live in your household? What do different people in your household do? How often do you visit the health clinic? How has this year been different than last year for you? Have you used this product before?
Once you have put the interviewee at ease, you can move to broader and deeper questions: • • • • • • •
How can we make this product easier for you to use? Is there something you would change about the product? What did you like about the product? Was anything difficult for you to find on this product? ( i.e. did the user have to hunt around for a button or latch?) Is there something that bothers you about the way this product is made? Were the product instructions easy to read? Did you notice (mention a particular feature of the product you are keen on having feedback in)?
While you watch the participant use a product, it is better not to interrupt. Only interrupt the participant when you need to clarify what is happening or when you are confused about something they say or do. To help gain clarification, you can ask the user: • •
What do you mean by that? What did you expect to happen?
Remember: The end user is the expert!
1st edition June 2010
13
8. Observation Observing end users as they interact with UNICEF-procured products is another way to gather information about their needs, desires and priorities. You can also learn about how your product works under the constraints of a particular environment. Observing often brings unexpected insight because environments of use can vary greatly and this can have a significant effect on how the product is used. In recent years, companies have employed usability researchers and sent anthropologists out into the field to discover more about their customers. For example, an engineer might shadow a cardiologist at work in a hospital to discover how to design user-friendly medical devices. A pharmacist might spend a day in a health clinic working alongside of health care workers as they administer vaccines. It is a good idea to go into the observation with some ideas about what you hope to find out, but also be prepared for the unexpected
Planning an observation Make an observation plan to guide you through the observation that includes: • • • • • •
The goal of the observation. Location and date of the observation. Names of participants. What you want to measure (for example, the time it takes to complete a task or the number of errors that a user makes). List of materials you will need such as a camera, paper and pen, video recorder, water for the participant, stopwatch, etc A list of questions that you want to ask after the observation.
Conducting an observation While you observe someone performing a task, take notes. Write down anything that catches your interest or surprises you. Watch for anything that changes the user’s behaviour or elicits an emotional response: whether frustration or excitement. Sometimes people get stuck and find another way to work a device or tool. This is referred to as a workaround. Work-arounds often indicate that a particular feature of your product is not intuitive enough. Observing people while they work is an unnatural situation and it is a good idea to acknowledge this to research participants. Your goal is to see how someone really interacts with your product and the best way to observe the participant’s natural behaviour is to be as anonymous as possible. Do not chat with participants as they work or interrupt them. This will distract them from the task they are performing. The only time you should interrupt is if you need to clarify something. Ask participants to “think aloud” as they walk you through a process or perform a task. Explain to participants that their comments will help you understand the how they work with the product. If 1st edition June 2010
14
participants can describe what they are thinking for you, it will help you understand their motivations, perceptions and reasoning. It is a good idea to take a few photographs of your observation to help you recall the details of your experience. Once the observation is over, take time to think about what you experienced and then write down the “big picture take-aways” from the observation. The immediate perceptions you have after the observation are meaningful yet easy to forget if you do not record them.
Documenting an observation When you document an observation, whether you are taking notes, recording audio or video tape or taking still photographs, try to capture: • • • • • • • •
Personal details (For a health worker: length of experience, location) Direct, unfiltered quotes The expressions and feelings of the person Ways participants interacted with others and things in the environment Things they care about most Emotional responses, positive or negative Processes Pain points
1st edition June 2010
15
Sample worksheet for documenting usability findings A sample template for documenting your observation as provided below can be used for both interviews and tests.
9. Usability testing Usability testing is different from observation as products are tested in a “lab” setting rather than in the environment of use. Therefore, testing is more structured and artificial than in an observation. Usability testing involves • • •
Writing carefully created tasks for a person to perform with a product while observers watch and take notes. Giving the person being tested scripted instructions before the test. After the test, giving the person a questionnaire to complete or is interviewed by the observer.
The purpose of the test is to observe how people interact with a particular product so that people’s preferences and any problems they may encounter with the product are identified. The observer usually asks the person performing the test to “think aloud” so that he or she can better understand the person’s behaviour. 1st edition June 2010
16
You can write tasks in a way that allows you to focus in on a particular feature of the product you are testing. For example, if you are testing a mobile phone and you want to test its “contacts” feature, then the tasks that you write will involve adding, deleting and finding contacts. The observer should not correct participants as they work because mistakes can reveal important flaws in the product. Correcting participants may also make them feel as if they, not the product, are being tested. During a usability test, you should focus on the participant’s thoughts and behaviour rather than whether what they are doing is “correct”. We also want to remind the participants that they are the expert on using the product, not us. Your job as an observer is to keep your user focused on the tasks and quietly observe him or her. While you are conducting a usability test: • • • • • • • •
Keep participants on task. Let the participant make mistakes. Answer questions with questions. Don’t tell the participant what to do. Don’t explain. Probe expectations. Investigate mistakes. Ask “Why?” often.
You should document the test results in a usability report and share the report with all stakeholders involved with the product.
1st edition June 2010
17
10. Working with usability findings Once you have collected data from interviewing, observing or testing end users, the next step is to organize it in a meaningful way. You can then analyse the data and apply the findings to your procurement work.
Organizing your data One way to organize and prioritise your usability findings from an interview, observation or test is to use the “post-it” technique. This technique allows many people to participate in the process of organizing key findings. You will need plenty of wall space and “post-it “notes for this technique. 1) Review the notes from each of your observations or interviews and highlight the significant elements, such as processes, reasons, pain points, tools and user quotes. Record each element briefly on a post it and stick it on the wall in any order. Do this for each user that you interviewed, observed or tested. The identity if each user is not important, the findings from each user will be mixed together. 2) Next, study the wall of post-its and look for two items that seem to go together naturally. Pick them out, put them up on another part of the wall, and give this new category a name. Continue grouping elements and creating new categories until all of the important elements and issues are categorised. The category names may need to be changed as more elements are added. At this point, you should be flexible, freely shift categories and elements around. 3) Review each category and try to prioritise each element within a category. Then organise each category according to priority. Structure your report accordingly with the highest prioritised elements first.
Implementing your usability findings As much as possible, put your usability findings into practice by applying them to the products you procure. You will not always be able to design products that meet all your end users’ needs; however, you can still use your findings to influence the product during the following stages of the procurement process: 1) When you analyze the initial request for a product, include persona descriptions and any usability findings as part of the analysis. 2) Before you send product specifications on to suppliers, consider your usability findings and refine the specification in accordance with the end users needs and priorities. 3) Finally, when you review the bids from suppliers, incorporate your usability findings into your analysis. When you are selecting the best supplier for a particular request, look at the product through the three human-centred lenses: Desirability, Feasibility and Viability. Your usability findings about the health care workers, children and other final end users may inform the Desirability lens while your usability findings for country offices, shipping agents and other stakeholders may inform the Feasibility Lens.
1st edition June 2010
18
4) Be sure to share your findings with your colleagues, whether you present it to them in a meeting or make it available to them online, so they can benefit from your findings. Sharing your stories of implementing change can inspire your colleagues and give them ideas for the products they are responsible.
Sample Usability Report Name: Location: Date: Product: Method: (e.g. Skype Interview, Observation in the Field and so on) Summary (What did you observe? Whom did you interview? What was the user’s experience level?) Focus (What did you hope to learn about the product? How can you improve its usability? How can you make it easier to ship?) Key Findings This is the body of the report where you should record: • • • • • •
What you learned from the interview or observation? What are the main themes that you stood out on this interview? What matters most to the participant? What things did the participant say or do that surprised you? Were there any memorable quotes you got from the interview? What new ideas for the future came up during this observation?
You can also subdivide your findings according to priority: critical findings, nice-to-have findings etc.
1st edition June 2010
19
11. Usability research or surveys? Generally, usability research uses small samples and yields good qualitative data while surveys have a broader coverage and are good for collecting quantitative data.
Qualitative data Qualitative data enables you to develop deep empathy for the people for whom you are procuring commodities. It produces richer, more detailed data than quantitative methods. This data can help you understand the constraints people work with when they use products and their unique perception of the product. However, qualitative will not determine “average” behaviours or attitudes or answer questions such as, “Are people in X region more likely to do this than in Y region?” This is because qualitative methods do not cover a sample large enough to be statistically significant.
Quantitative data Surveys yield quantitative information which is good for discovering trends, generating statistics, determining averages, assessing common practices and finding out whether your commodities are meeting the needs of your users. In later phases of the procurement process, surveys complement usability research and can help you understand the effect of you commodities in different countries or populations. For example, if you are developing a mother and child pack and would like to research the best options for a particular population of women, usability research will most likely be better at helping you discovering what your users need and want. However, if you want to find out if people are satisfied with an existing mother and child pack conducting a survey is probably the best course of action.
1st edition June 2010
20
Designing online surveys Online surveys are quicker and easier for participants to fill out and therefore are more likely to illicit a response. Several online vendors offer templates and space “in the clouds” that are easy to use and do not require the technical assistance of an IT professional. Survey Monkey (www.surveymonkey.com) is an online vendor that facilitates online surveys and offers a wide range of survey templates from which to choose. If you want to collect meaningful and relevant information from your survey, you have to create a survey that will generate relevant responses. The following steps will guide you in writing an effective survey: Identify goals and target personas Before you begin writing a survey, you must first identify your goal for creating the survey and the target persona from whom you want to collect information. What specific information do you want to learn from this survey? Use personas to help you focus on the end users you want to target. Introduce your survey Frame the context of your survey with a few short sentences to introduce the survey. Let the participant know why you are asking for their time and how it might benefit them. Create quantitative multiple-choice questions Show that you value your respondent’s time, by limiting the survey to 10 - 15 minutes. By using multiple-choice questions, you save the respondent from having to think about how they should phrase answers. Limit the answer scale of each question to five and always provide an “other” field so users can add their own text. When you want to measure an attitude, use the Likert scale, which offers the respondent the following options: Strongly Agree, Agree, Neither, Disagree and Strongly Disagree. Write well-formulated questions Use questions that the user can easily understand in one reading. Make sure your questions are objective, short, simple and targeted. •
Be objective: Survey questions should not reveal the author’s point of view and thereby influence the respondents’ answers. For example: “Are you satisfied with UNICEF's new improved vaccination program? “ Subjective question. Instead, you should remove the author’s point of view from the question and write: “Are you satisfied with UNICEF’s new vaccination program? “ Objective question
•
Be brief: If you write longer questions, you risk confusing your reader. Find the shortest way to ask a question without losing its intent.
•
Be simple: Use simple, clear words and phrases that your reader is sure to understand. Do not use technical jargon. Avoid words that extend the question to an extreme such as, always or never. Respondents may not want to answer such extreme questions.
1st edition June 2010
21
•
Be specific: Ask precise questions. Avoid questions that are too general, too complex or undefined. Stay away from using words like often, usually, generally, because these words leave room for interpretation.
If you are unsure about the survey you have created, test it on one of your colleagues to ensure that it is in accordance with these guidelines.
12. Alternative methods of obtaining feedback It is not always possible to get out into the field to conduct usability research; however, there are several good alternatives to field research for generating reliable user feedback. You can: • •
• • • •
Conduct interviews or observations via videophone or web cam using www.Skype.com. This option requires that your research participants in the field have access to a computer and web cam or videophone. Recruit people for usability testing from your location. Their profiles should match the user profile of real users in the field. Send survey kits into the field for self-documentation. Kits might include journals, disposable cameras, video recorders or any supplies that help users document their experience with your product. Enlist colleagues headed out on a mission to conduct usability research for you. Engage country offices in becoming partners in usability research. Schedule interviews and usability testing with country office representatives and other stakeholders when they come to your office for a meeting or conference. Schedule interviews and usability testing at organized events where you are participating such as conferences.
1st edition June 2010
22
13. Integrating usability into procurement workflow The procurement workflow at UNICEF begins when the Supply Division receives a request for a product. Next, the technical staff analyses the specifications received from the customer. This is the ideal point to conduct usability research, whether it involves making a few targeted phone calls, sending out a survey or conducting field interviews using videophones. Outline of UNICEF procurement workflow: UNICEF awards contract to best supplier
Supply Division receives request for commodity
Supply Division evaluates commodities against specifications
Supply Division analyzes and refines specifications
Suppliers send in bids to UNICEF
Specifications sent to prequalified suppliers
The time for usability research is early in the procurement workflow. Trying to incorporate usability research later in the procurement cycle makes it challenging to efficiently make changes to the product. Addressing a usability issue after you have sent product specifications to pre-qualified suppliers can mean that solving the usability issue is more expensive or not feasible. Usability research is best practiced when the technical staff first analyses the specifications before they are sent to the supplier.
1st edition June 2010
23
14. Engaging users in usability research It not always possible for UNICEF technical staff to travel out to the field, however, they can benefit from engaging country offices in usability research. Both technical staff and managers can use the following recommendations to improve usability research and user feedback.
Recommendations for technical staff Technical staff can engage users by: • • • • • • • • •
Sharing feedback with the end user, technical staff can help make end users partners in the procurement process. Mapping users to determine their needs and then sharing these user needs with others at UNICEF. Asking people who go on missions to include other products in their user research. Sending out survey kits that contain disposable cameras, journals, pens, paper and anything else that will enable users to provide product feedback to UNICEF. Identifying relevant staffing country offices who can act as UNICEF contacts. Sharing insights and take-aways after a mission. Becoming usability ambassadors by raising awareness of the benefit of usability and research. Expanding and improving the existing feedback loop. Making impromptu visits to sites that are relevant to your product when you are on a mission. When technical staffs go on mission with a large delegation, try to break off from the large group and see how your product works at a health care centre.
Recommendations for managers Managers can help engage country offices by: • • • • •
Developing usability research programs at regional offices and include funding in the budget for these programmes. Issuing a directive through, for example, the Quality Assurance Centre that would require usability research to be part of Key Performance Indicators.. Creating a pilot project financed by supply division to conduct usability research on a targeted product. Sharing in advance meeting times and contact information when representatives from country offices come to Copenhagen so that technical staff has an opportunity to meet with them. Offering to bring back feedback for colleague’s products so others can benefit from your mission.
1st edition June 2010
24
15. Innovation business model The many small decisions that UNICEF staff make each day about the products they procure can have a significant impact on UNICEF constituents. By proactively seeking out feedback from end users and understanding their challenges and constraints, you can make better decisions when you procure products. For example, if you have three different suppliers for the same HIV medication and you select the supplier who provides the clearest pictorial instructions, you have positively affected the health of potentially hundreds of UNICEF’s constituents. The innovation business model provides you a framework for the usability work that you do and the solutions and innovations, whether incremental or radical, that you discover. Diagram of Innovation Path
1st edition June 2010
25
The five stages of Innovation The Innovation Business Model includes the following five stages: 1) Acknowledging There are different ways in which UNICEF’s attention is drawn to a problem. In some cases, an NGO raises UNICEF’s awareness of a certain problem. In other cases, this is done by governments or even the end users themselves. Acknowledging a problem that calls for innovation is difficult, both due to the responsibility that goes with acknowledging it, and because most problems do not seem clear-cut. We tend to see problem areas, a complex and chaotic set of different problems with no clear pattern to it. In order to innovate to improve the lives of children, there must be either a perceived problem or a perceived problem area. User centred tools and methods such as observations and interviewing are methods we can use to at this stage to become aware of problems. Possible methods: Surveys Usability research 2) Defining When the problem or problem area is acknowledged, it needs to be defined for further development. A definition is thus primarily a decision about which problem or problem area requires innovation. At UNICEF, a definition can be made at either country, regional or division level, depending on the nature of the problem or problem area based on user centred design principles such as a focused usability research. Subsequently, the definition is forwarded to the relevant division. The relevant division should then make a rough selection and make a recommendation, based on a specially devised formula, which describes the nature of the problem and its scale, as well as possibly providing ideas on how to solve the problem. Possible methods: Usability research Personas Interviewing Observations 3) Deciding The recommendations are sent to the Innovation Advisory Panel secretary, who, together with the Innovation Project Manager, decides which recommendations are to be presented in which way to the Innovation Advisory Board. The Innovation Advisory Board then decides which problem UNICEF should move forward with, and how. The illustration of the generator shows us that there are two possible approaches. UNICEF can either cooperate with academia to find solutions to problems or definitions of problems within problem areas. Alternatively, it can cooperate with businesses to solve a problem. It is subsequently up to the relevant Management, Division or Centre at UNICEF (depending on the problem) to decide on how to move and where to anchor the responsibility for the further development process within the organization. 4) Developing
1st edition June 2010
26
Depending on the type of partnership, the developing process is followed by UNICEF as well as by the Innovation Advisory board. As the process is developing, it is outsourced to either business or academia, depending on the nature of the problem and the recommendations of the Innovation Advisory Board. It is key to the development process that it is centered on the end user and based on the innovation principles such as real user input from observations, interviewing, prototype testing or other ways of getting user input . All development partnerships should be based on a predefined set of success criteria and clear delivery milestones, as well as an ongoing dialogue and thorough follow-up during development. UNICEF is responsible for this process and, also, for involving members of the Innovation Advisory Board when needed. Iterative usability testing with prototypes helps you to develop products optimally. Possible methods: Prototype/ Usability tests 5) Solving After being presented with a solution, UNICEF and the Innovation Advisory Board decide what to do with the new knowledge, the developed prototypes, recommended services, etc. In the case of work being done in connection with a problem area, the newly defined problems can go through a fresh process in the innovation generator. The solution should be tested in the field through usability test or other ways of gathering the user’s input. Possible methods: Prototype/ Usability test
The Innovation platform http://artplatform.unicef.org/denmark/do/dansupplyfaces.nsf/in_mainpage?readform Discovering challenges and finding solutions using the methods and tools in this handbook can be incorporated into your regular workflow. The end user feedback you receive and the solutions you arrive at are important to your colleagues and to the innovation process. The new innovations you discover can improve future innovative products or projects. Therefore, the Innovation team has developed the Innovation Platform as a source of knowledge sharing of these challenges and ideas.
1st edition June 2010
27
15. Conclusion Usability research methods, derived from ethnography, can help UNICEF technical staff gain a deeper understanding of their end users’ needs and priorities. This knowledge can then feed into new innovation opportunities. Human-centred product evaluation methods help UNICEF learn how commodities are used in-context and a product’s strengths and weaknesses. By understanding end users’ attitudes, values and behaviour, UNICEF can serve its constituents better by providing commodities that suit their needs. The innovation business model gives technical staff a framework to work within when they conduct usability research. The Innovation Platform is a place to share ideas and innovations so that colleagues can benefit from each other’s work.
1st edition June 2010
28
16. Terminology list Beginner’s mind A mindset or attitude of openness, eagerness and a lack of preconceptions that allows for observation of scenarios or behaviour without interpretation. Cloud computing Using servers hosted on the Internet for computational purposes. Cloud computing vendors build Internet accessible data centres and sell computing time to customers. Cloud computing lets you use applications and files over the internet. Gmail is an example of cloud computing. Contextual inquiry A design methodology in which you conduct structured field interviewing. Contextual inquiry is based on the idea that understanding the context in which a product is used is essential for elegant design, and that the user is a partner in the design process. End-user Monitoring The periodic oversight of the effectiveness and timeliness of deliveries and the utilisation of supplies by end-users as intended (fit-for-purpose). Human-centered design A design philosophy and a process in which the needs, wants, and limitations of end users are given extensive attention at each stage of the design process. (Also referred to as User-Centric Design) Human factors engineering The application of knowledge about human characteristics and abilities (physical, emotional and intellectual) to the design of tools, devices, systems, environments and organizations. In-context Refers to being with people in their real settings, doing the things they normally do. Mental models A representation of the cognitive and emotional components of human thought processes that determine how individuals interpret and respond to the things they encounter. Personas Fictional characters created to represent the goals and behaviors of different types of users within a targeted demographic in order to help guide the design decisions about a product. Personas are created by synthesising data collected from interviews with users. The persona description includes the behavior patterns, goals, skills attitudes and environment as well as personal characteristics that help humanises the product’s user. Scenarios A character-rich story line describing the context of use and goals of a user that helps communicate the essence of a need in a specific context. Supply chain management 1st edition June 2010
29
The design, planning, execution, control, and monitoring of products and services with the objective of creating net value, building a competitive infrastructure, leveraging worldwide logistics, synchronising supply with demand, and measuring performance globally. (Source: APIX) Usability A field of study that measures and analyses the effectiveness, efficiency and satisfaction with which a specified set of users can achieve a specified set of tasks in a particular environment. (Source: International Standards Organization) User experience A term used to describe the overarching experience a person has because of their interactions with a particular product or service. The user experience includes how the product is perceived, learned, and used.
1st edition June 2010
30
17. References Anderson, David L, Frank F. Britt, Donavon J. Favre, Supply Chain Management Review, 2007 IDEO Design, Human-Centered Design Toolkit, www.ideo.com, 2009 Jandt, Fred E., Intercultural Communication, Sage Publications Inc., Thousand Oaks, California, 2004 Kreitzberg, C.B and Little A., the Power of Personas, MSDN, 2009 Nielsen J. and Loranger H., Fundamental Guidelines for Usability, Nielsen Norman Group, Fremont, California, 2006 Norman, Donald A., The Design of Everyday Things, MIT Press, London, 1988 Patton, Quinn, Utilization-Focused Evaluation, M. Sage Publications Inc. Thousand Oaks, California, 1997 Save the Children, Interviewing Children: A Guide for Journalists and Others, NBN International, Devon, England, July, 2003. Wiklund M.E. and Wilcox S.B., Designing Usability into Medical Products, CRC Press, Boca Raton, Florida, 2005
1st edition June 2010
31