Final Report UMass Dartmouth MyOrgs Site
Gabriella Barthe Professor Gulbrandsen Writing Computer User Documents December 17, 2018
Barthe
Table of Contents Executive Summary …………………………………………………………………………….3 Introduction and Project Plan …………………………………………………………………4 Goals ……………………………………………………………………………………..5 Report Structure ………………………………………………………………………...6 Methodology …………………………………………………………………………………….7 Testing Overview ……………………………………………………………………….7 Usability Testing Participants………………………………………………………….8 Scenarios & Tasks………………………………………………………………………10 Scenario 1………………………………………………………………………..10 Scenario 2………………………………………………………………………..10 Scenario 3 ……………………………………………………………………….10 Findings ………………………………………………………………………………………...12 Scenario 1 ………………………………………………………………………………13 Scenario 2 ………………………………………………………………………………14 Scenario 3 ………………………………………………………………………………15 Questionnaires Analysis ………………………………………………………………………16 Quantitative ……………………………………………………………………………16 Qualitative ……………………………………………………………………………...17 Recommendations ……………………………………………………………………………..18 Recommendation 1 ……………………………………………………………………18 Recommendation 2 ……………………………………………………………………18 Recommendation 3 ……………………………………………………………………18 Recommendation 4 ……………………………………………………………………18 Conclusion ……………………………………………………………………………………..19 Appendix A: Heuristics Evaluation …………………………………………………………20 Appendix B: Personas ………………………………………………………………………...32 Appendix C: Usability Test Plan ……………………………………………………………..34 Appendix D: Pre-Test Questionnaires ………………………………………………………40 User 1……………………………………………………………………………………41 User 2 …………………………………………………………………………………...42 User 3 …………………………………………………………………………………...44 User 4 …………………………………………………………………………………...45 User 5 …………………………………………………………………………………...46 Appendix E: Moderator Script ……………………………………………………………….47 Appendix F: Logs ……………………………………………………………………………...50
2
3
Usability Testing Final Report
User 1 …………………………………………………………………………………...50 User 2 …………………………………………………………………………………...53 User 3 …………………………………………………………………………………...55 User 4 …………………………………………………………………………………...56 User 5 …………………………………………………………………………………...56 Appendix G: Post-Test Questionnaire ………………………………………………………57 User 1 …………………………………………………………………………………...59 User 2 …………………………………………………………………………………...62 User 3 …………………………………………………………………………………...65 User 4 …………………………………………………………………………………...68 User 5 …………………………………………………………………………………...69 Appendix H: Release Form .…………………………………………………………………..70
Barthe
Executive Summary The University of Massachusetts Dartmouth MyOrgs site is a student centered webpage designed to engage students in clubs and activities on campus as well as give information on how those organizations are maintaining themselves. The following usability testing was performed independently from the University of Massachusetts Dartmouth and its site. Two personas of typical student leaders that would utilize multiple facets of the MyOrgs site were created and then used to conducted an evaluation of the site using a set of heuristic criteria based on the 5 E’s. Based on the information gathered a test plan was created. Usability testing was completed with five participants in December of 2018. During this testing the following was revealed: •
All of the participants were unwilling to use the site again of their own accord.
•
Participants commented that the site was bland and left little visual appeal.
•
Participants found some pages misleading or completely useless.
•
Some aspects of the site were unable to perform or complete their tasks.
•
Little to no directions or information on aspects of the site are given.
•
Multiple aspects of the site could complete similar functions but they are not interchangeable.
Based on these findings from testing, the following recommendations were made: •
Remove all unnecessary pages and create more visually appeal navigation.
•
Integrate more directions and straight forward content throughout the site.
•
Merge similar functions together and simplify the options while maintaining the ability to input custom responses.
•
Utilize endless scrolling features to save time while accessing the site.
4
5
Usability Testing Final Report
Introduction & Project Plan The following University of Massachusetts Dartmouth MyOrgs User Testing was completed by Gabriella Barthe, Professional Writing and Communications Masters Candidate at the University of Massachusetts Dartmouth as part of the Writing Computer User Documents Curriculum. Usability testing was completed on the University of Massachusetts Dartmouth MyOrgs website during the Fall 2018 semester under the guidance of Dr. Karen Gulbransen, Associate Professor of English. The project was completed in the following stages: • Creating personas of student leaders using the MyOrgs site. • Crafting a set of Heuristics and performing an evaluation of the site on those benchmarks. • Creating a usability test to perform on student participants. • Analyzing results of the usability testing to provide recommendations on the MyOrgs site. The project was completed in its entirety between the months of September – December 2018. This usability testing was conducted separate from the University of Massachusetts Dartmouth MyOrgs site and was not issued by the University.
Goals The University of Massachusetts Dartmouth MyOrgs site is aimed at providing students with a space to find organizations, maintain membership, and facilitate club activities, as well as a means for the University Student Activities Involvement and Leadership (SAIL) office and Student Government Association (SGA) to source information needed from the student body. Based on information gathered during preliminary interviews to craft student personas, and the heuristic evaluation, the following goals were crafted for the usability testing: • How intuitive is the site to use? How do users get started using the site? • What issues do students find while trying to access the site? What is deterring them from using it? • What aspects of the site do students find useful? What aspects of the site do students find are stopping them from completing their goals? • How inclined are users to access the site regularly?
Barthe
Report Structure This report includes the following sections: • Methodology – This section will review testing procedures, an explanation of personas used to find participants, a look at the participants, and an explanation of the scenarios and tasks used to evaluate the site. • Findings – This section will review the results of user testing broken down by each of the scenarios in terms of positive and negative results. • Recommendations – Recommendations will be made based on the results presented from all users. • Conclusion – This section will summarize recommendations and the need for further usability testing. Each of these sections is aimed at summarizing and analyzing the methods employed during this process as well as the overall results. Some items will be included in the appendixes, including: • Appendix A: Personas • Appendix B: Heuristics Evaluation • Appendix C: Usability Test Plan • Appendix D: Pre Test Questionnaire & User Responses • Appendix E: Moderator Script • Appendix F: Log of Moderator Notes • Appendix G: Post-Test Questionnaire & User Responses • Appendix H: Release Form
6
7
Usability Testing Final Report
Methodology This section aims at providing a view of the testing processes used during this usability testing on the University of Massachusetts Dartmouth MyOrgs site. This section includes a testing overview, explanation of participants, and a description of scenarios and tasks employed during moderated testing.
Testing Overview Usability testing was conducted on a total of five (5) participants at the University of Massachusetts Dartmouth. All completed tests lasted between 35 to 50 minutes and included: • A pre-test questionnaire delivered by the moderator • A moderator-led description of the testing process • Three scenarios each with a number of tasks for participants to partake in • A post-test questionnaire delivered by the moderator Equipment used for testing included: • Participant computer • Logging station • Audio recording software Each session was led by moderator script delivered by test administrator. The moderator sat with participants through all scenarios and encouraged feedback if needed. For a full description of the test and moderator script refer to the Usability Test Plan in Appendix C.
Barthe
Usability Testing Participants All participants are undergraduate students at the University of Massachusetts Dartmouth. Each of these students are considered student leaders in that they are on the executive board of at least one club or organization recognized by the University of Massachusetts Dartmouth. Each of these students was chosen by the moderator as they were known to them as serving a student leadership role in some capacity. All of these student leaders were compared to the personas found in Appendix B. U1 – Sawyer: • History Major • Senior • President of 20 Cent Fiction Productions • Treasurer of Enjambment • Previous secretary of NEHES • Previous secretary of Fishbowl • Active internet user U2 – Armani: • Graphic Design Major • Sophomore • Treasurer Elect of Fishbowl • Lead Graphic Design of The Torch • Resident Assistant First Year Housing • Active internet user U3 – Kate: • History and Physics Major • Junior • Vice President of the Student Government Association • Peer Health Educator • Active internet user U4 – Katie: • Sociology Major • Junior • Secretary of Enjambment • Works with SAIL • Active Internet user
8
9
Usability Testing Final Report
U5 – Marie: • English Major • Senior • Vice President Enjambment • Former Resident Assistant • Sorority member • Active Internet user Each participant’s opinions and responses to all survey questionnaires and moderator logs can be found in Appendixes D, F, and G.
Barthe
Scenarios & Tasks Participants will be given the following tasks to complete using the think aloud protocol. Each task is created with a specific scenario in mind and correlates to specific purposes of the site. Scenario 1 You are looking for a new club to join on campus. Knowing that MyOrgs is the place to go for more information on organizations at UMass, you decide to see what you might find. Task 1 Find a new organization on the UMass Dartmouth MyOrgs site that interests you and join it. Task 2 You want to gain a bit more knowledge about a club or organization before you join it. Search through the events catalog and RSVP for an event. This scenario is designed to gain the following input from the participants: • How do users respond to the overall look and feel of the site? • Are users able to easily navigate the site and find clubs? • What tasks do users think they can perform on the site? • How useful are tasks for general users on the site?
Scenario 2 You are the President of an organization. SGA requires that all organizations hold an election on campus every year for a new eboard. Task 1 Create an election through your club’s page for members to vote on. Task 2 You have noticed you made an error on the election ballot, go back and edit it. This scenario is designed to gain the following input from the participants: • Can the back end of the site be easily found?
10
11
Usability Testing Final Report
• •
Can the section be easily navigated? What do participants think of the direction available to them?
Scenario 3 You are the Secretary of an organization. You know the MyOrgs allows you to advertise events on the site so you decide to create an event for an upcoming meeting. Task 1 Create an event listing for your organization.
Barthe
Findings This section presents and analyzes participant responses and reactions to each of the three scenarios in the usability test. Specific recommendations are in the next section.
Positive Findings Some positive elements found during usability testing include: • Participants were able to find sections on the front end of the site with ease. • Visuals for club listings are pleasing and simplistic. • All participants were able to request to join an organization. • All participants found the consistency of the back end of the site useful. • Users enjoyed the amount of clubs available through the site. • Users found some category options interesting and useful.
Problematic Findings Severity Scale Negative findings are ranked on the following severity: • Cosmetic Issue – Unappealing visuals, unable to find information, glitch in design elements, missing images, etc. • Minor Negative Use – Issue in ability to complete tasks with minimal to medium level impact on user experience. • Major Negative Use – Issue in ability to complete tasks with medium to high level impact on user experience. • Unable to Complete Task – Issue in ability to complete tasks that either prevented users from completing tasks altogether or led to high enough frustration levels to prompt user to quit.
12
13
Usability Testing Final Report
Scenario 1 – Becoming Familiar with the Site & Participating in Orgs Scenario 1 on average lasted about 10 minutes and 42 seconds. Description
Participants Description
Participants Description
Participants
Scrolling through the organizations listing, users complained about the need to constantly click on the load 10 more organizations on a list of 100+ organizations.
Minor Negative Use U1 looked uninterested and kept arms crossed during the entire time scrolling through the clubs.
U5 “Why do I have to click this every single time? Why isn’t this U1, U2, U3, U4, U5 endless scrolling?” Finding specific categories or Major Negative Use organizations is difficult without knowing a specific group from the U4 “Why are these categories start. wrong? There are clubs that are performing arts that aren’t in this U2, U4, U5 group.” Events had no options to RSVP to Unable to Complete Task as there were no events approved by SAIL at the time. U1 in response to being unable to complete this task “I don’t like U1, U2, U3, U4, U5 anything UMass”
Barthe
14
Scenario 2 – Creating Elections On average scenario 2 lasted 12 minutes and 13 seconds. Description
Front page to organization pages Major Negative Use are not necessary and serves no purpose. U1 “This can be a blip in the corner”
Participants Description
U1, U2, U3, U4, U5 U5 “Why is this page even here?” Forms and Elections seem Minor Negative Use indistinguishable in use from one another. U1 “Forms and documents could be worded differently … I don’t know U1, U2, U3, U4, U5 what the difference is here.” Election ballots will redirect users Major Negative Use to multiple pages while trying to create them with zero directions. U1 “It gives you no instructions of what to do.”
Participants Description
Participants Description
Participants
U3 “This is just redirecting me and I don’t know what I’m supposed to do U1, U3, U4, U5 next.” Settings for ballots are easy to Major Negative Use mess up. Options are unclear and not defined. U1 “This would be nice if it was just one thing.” U1, U3, U4, U5 U4 “Why isn’t this a doodle poll?”
15
Usability Testing Final Report
Scenario 3 – Creating an Event On average this scenario took 6 minutes to complete. Negative Findings Description
Participants Description
Participants Description
Participants
Theme and event categories are Minor Use Issue limited. U5 “None of these really apply to my U1, U5 club so I just pick general meeting.” All events have to be approved Unable to Complete Action before they can be edited. U1 was pulling on their hair while U1, U2, U3, U4, U5 filling out the form. Adding date options was separated Minor Use Issue by different locations for each additional date & times would reset U2 “This is just odd. I don’t get why for each iteration. the dates are set up like this.” U1, U2, U3, U4, U5
Barthe
Questionnaires Analysis Before and after testing users were asked to fill out questionnaires related to their internet use, knowledge and familiarity with the MyOrgs site. These questions were aimed at getting their impressions and feelings about the site itself and the ability to complete the session’s tasks. These results can be viewed in further detail in the Appendix separated by each user.
Qualitative When presented with questions based on their opinion of the site users responded in this manner:
“Whenever I need to do something more than surface level it breaks down and makes me want to cry.” U1 “In comparison to other websites I have found the WUMD site deplorable in terms of not only layout, but it just hurts to look at. My eye has no idea where to go first or how to properly read it.” U2 “No, not really. This helped me look at it a bit more critically but it didn’t highlight anything I hadn’t noticed before.” U1 “Most things aren’t apparent. The website makes completing a task take ages to do.” U3 “I didn’t like it but I hate it more and more every day.” U3
When asked if students would willingly use the site again, all users said no. The only reason any of them presented for returning to the site was that they had to for one reason or another.
16
17
Usability Testing Final Report
Quantitative After testing was completed, when presented with a 5 point scale to rate each task 1 being very difficult, 5 being very easy the following results were found:
Task 1
Responses: 2 neither easy nor difficult (3), 1 very easy (5), 1 somewhat easy (4), 1 somewhat difficult (2)
Task 2
Average: 3.4 Responses: 1 very easy (5), 2 somewhat easy (4), 1 neither easy nor difficult (3), 1 somewhat difficult (2)
Task 3
Average: 3.6 Responses: 2 somewhat easy (4), 2 neither easy nor difficult (3), 1 somewhat difficult (2)
Task 4 Task 5
Task 6
Average: 3.2 Responses: 3 very difficult (1), 2 somewhat difficult (2) Average: 1.4 Responses: 1 very easy (5), 3 neither difficult nor easy (3), 1 somewhat difficult (2) Average: 3.2 Responses: 3 somewhat easy (4), 2 somewhat difficult (2)
Average: 3.2 On average, the rankings users gave to the tasks themselves were rather neutral coming in around the 3 range. Though Task 4 specifically (creating and editing an election ballot) overall users found very difficult to accomplish, indicating that something needs to be changed in the election functions.
Barthe
Recommendations The following section includes recommendations to improve the usability of the UMass Dartmouth MyOrgs site. The recommendations are based on the findings presented in the previous section.
Recommendation 1 – Utilize Endless Scrolling Features One of biggest issues users found from the start was the constant need to load more clubs, events, and other aspects of the site, in increments of 10. Adding an endless scroll feature will allow users to access information more quickly and with less frustration.
Recommendation 2 – Changing the landing page for individual clubs The page for individual club management is a black title page with a listing of the number of members in the organization. This information can be better accesses on the roster page and overall the cover page serves no purpose and slows users down. It is also really easy for users to accidentally leave this page as the navigation is not easily noticed in this form. If this landing page had icons of the general side navigation for users to click on rather than a blank title page it would better suit users’ needs.
Recommendation 3 – Add Directions to all Forms sections Each section that required users to fill in aspects of the site provided little to no direction for users to complete the forms. When issues arose, users often were unsure as to what the issues were at first or were confused as to why instructions weren’t more explicit to begin with.
Recommendation 4 – Merge Forms and Elections Sections Both the forms and elections sections were capable of doing most of the same functions but were listed as different concepts. Forms actually had more options than the elections section did in the long run and users were confused on which ones to use. While creating ballots users were also confused by the ability to create multiple ballots within one election and multiple pages within a ballot.
18
19
Usability Testing Final Report
Conclusion With the showcasing of several negative views on the University of Massachusetts Dartmouth MyOrgs site features, it becomes clear that more testing and work needs to be done on the platform. This usability testing was conducted with the aim of identifying what areas the site is succeeding at but mostly to figure out what it is that users wish to see fixed. Provided has been positive elements along with several personal opinions, and negative results ranked by severity, followed up by a few recommendations that could resolve many of these issues. In addition to retesting after making modifications, it is encouraged that a broader participant pool be tested on a well defined set of heuristics moving forward. The site could benefit from further input from student users to increase usage and effectiveness of the site.
Barthe
Appendix A: Heuristics Evaluation Introduction UMass Dartmouth’s MyOrgs Web site is geared toward student groups trying to keep clubs active and reach a new audience of potential members. More so than that, it is aimed at allowing professionals within the university a space to monitor club activities, membership, and general student involvement on campus. After performing interviews of student users, I created two personas in the realm of student leaders. Each of these two personas represented a different section of Student Leaders that used the site. The first - Sawyer Egan - represented the Presidents of organizations on campus. Those who were involved in more than one club or activity, were junior or senior standing, and are leaders among leaders. The second - Megan Perreira - represented the sophomores who likely held secretarial positions, are less involved, and are new to holding positions on campus. When looking into UMass Dartmouth’s MyOrgs site from a heuristic standpoint I did so with the understanding of Sawyer Egan as a persona. Using a scenario of a President trying to reach a larger community base for membership and comply with SAIL and SGA standards of holding an election through the site and having updated membership information. Tasks included: setting up an election on the myorgs site, marketing events to other students, checking and maintaining membership, and uploading files such as constitutions, photos, etc, to the site. The heuristics used for this research were the 5 E’s - effective, efficient, error tolerant, engaging, easy to learn, plus the concepts of intuitive use - quality of directions, presentation in a way that is clearly understandable, redundant navigation, and usability - the offering of simple ways to complete tasks, from Neilson’s 10 as well as aesthetics - visual appeal. This set of eight heuristics together creates a broader image of how the site works on behalf of its users.
20
21
Usability Testing Final Report
Methodology Persona The following evaluation was completed through the help of Sawyer Egan’s characteristics. Through the eyes of this persona, I have been able to access the MyOrgs website to assess the heuristics from a typical user’s perspective. Sawyer Egan represents the Student Leader that goes above and beyond. They are an upperclassman, highly involved in student politics, hold the position of President in their organization, and have been involved as a Student Leader for a number of years. Further information about Sawyer Egan includes: • Gender – Male • Age – 21 • College – Arts & Science Undergraduate • Technology Understanding – Uses a smartphone and computer daily • Internet Use – Social Media, Information, Gaming, Music, Streaming • Hobbies – Theater, Music, Hanging out with friends Scenarios The following three scenarios were used as a basis for the persona of Sawyer Egan to complete their tasks: Scenario 1 Sawyer is anticipating events for the beginning of the new school year and wants to reach as large of a population as possible. Knowing that new students are directed toward the MyOrgs site to get involved on campus, he has the idea to market the organization he is President of, along with a number of their events through the site. He wants to make his MyOrgs page as appealing as he can to attract and engage students. Scenario 2 UMass Dartmouth’s SAIL office just notified all organization Presidents via email that they are monitoring the status of every club. In order to stay in good standing, each organization must have at minimum 10 general members, and 4 e-board members actively confirmed as part of the organization on MyOrgs. Sawyer is uncertain of how many students are active on their site, but he did just hold a general interest meeting. Sawyer decides to invite all of the new interested students to join the club on MyOrgs and check on this standing. Scenario 3 According to SGA guidelines, all organizations must hold a general election for all e-board positions for the following school year before Spring Break. Sawyer’s club members are unable to meet with enough people to warrant a vote in that time
Barthe
frame. In order to meet SGA standards, Sawyer decides to use MyOrgs and create a ballot for members of the club to vote anonymously online within a set timeframe.
Tasks The main tasks used to evaluate the UMass Dartmouth MyOrgs Website for Student Leader use were: • Check and maintain student membership within an organization. • Upload files to the organization’s page. • Marketing the organization to potential new members. • Creating a usable election ballot through the site. Heuristics By combining concepts from Jakob Nielsen’s Ten Usability Heuristics and Whitney Quesenbery’s 5E model, the following set of 8 heuristics were chosen. • Effective – How completely and accurately a task can be undertaken o Options that are presented can achieve the task at hand • Error Tolerant – How easily can mistakes be fixed and prevented o Showcases issues clearly to the user o Creates clear path to fix issues o Emphasizes buttons and options to make tasks easier o Easy to understand • Efficient – The speed at which a task can be done o Number of clicks needed o Navigation o Easy to find actions • Engaging – Interactions of users with site content o Use of multimedia elements o Options for members to interact with • Easy to Learn – Ease at which users can complete new tasks o Consistency o Similar to other interfaces o Use of directions • Intuitive Use – How easily users can infer how to interact with site options o presentation in a way that is clearly understandable o redundant navigation • Aesthetic – How the site looks o Use of color and images o Use of design elements to showcase purpose of organizations
22
23
Usability Testing Final Report
Ranking System Each task was ranked throughout the process based on the following scale: • Positive Use – Tasks were easy to complete, options were clearly visible, no issues were immediately evident. • Positive Visuals – Pleasing aesthetics, options to change visuals present • Cosmetic Issue – Unappealing visuals, unable to find information, glitch in design elements, missing images, etc. • Minor Negative Use – Issue in ability to complete tasks with minimal to medium level impact on user experience. • Major Negative Use – Issue in ability to complete tasks with medium to high level impact on user experience. • Unable to Complete Task – Issue in ability to complete tasks that either prevented users from completing tasks altogether or led to high enough frustration levels to prompt user to quit.
Barthe
Findings Task Summaries Task summaries are written from the perspective of the persona Sawyer Egan as described in the methodologies section. Task 1: Checking Student Membership Checking student Membership was simple on the MyOrgs site if you knew where to go to find it. It took me seven clicks when I had the right path just to find the roster. Though, in aiming to find the roster without direction from another person it took me fifteen clicks to find the same list.
Once on the Roster page it is easy to manage positions, invite individuals and check those who are currently in the organization, are pending invites, and are asking to join the group. Though to take any action one has to carry out each individual person separately. So, if I wanted to add multiple users I would have to invite or approve each person one by one. The same goes for adding people to new positions. While this isn’t difficult it can be tedious and time consuming.
24
25
Usability Testing Final Report
While trying to reach this page though, the site opened 3 different tabs in my browser. The first started me from the main homepage which led me to the organizations I was a member of. To get to this page I had to know there was a drop-down menu by clicking on my profile picture - something the site doesn’t imply at all. From there I had to scroll down to the organization I wanted to focus on. Each of these listed the name of the org, a “category,” approved service hours, a place to add in a self-reflection both as a member and if you are an officer in an organization and then two buttons. The “organization site” link is almost easy to miss as it is just blue text. Immediately below that is a large blue button labeled “leave organization.” I almost clicked this button twice in my confusion.
After clicking organization site, a second tab was opened. This tab showed a grey page with basic contact information, space to find important files from the organization, and generally is a blank page. As a student leader you can click on the manage organization button in the top right corner which is easy to find but opens yet again another tab. This tab is a blank white page with the name of the organization dead center and the MyOrgs banner menu. If you click on the icon that is three lines a drop-down menu to edit this specific group comes up at which point you can find the roster. Initially when directed to this page I assumed some issue has occurred due to the drop-down menu being barely visible and there being nothing on the main screen to fully interact with.
Task 2: Uploading Files
Much like accessing the Rosters page, finding the space to upload files was fairly simple when I knew where I was going. The Documents page location is clearly marked in the drop-down menu and the tasks themselves are self-explanatory. When adding a document to the myorgs page if all aspects were not filled in I would be prompted with what exactly was missing. Though, there is no way to remove documents once they have been posted to the site.
Task 3: Setting up an election through the Site
Barthe
The path to set up this ballot was very similar to that of uploading the documents. Once in the managing organization section, I simply had to click “Forms” on the left drop-down. From here I clicked on the “Create Forms” button that is clearly visible in the top right hand corner. Originally, I had confused the Forms and documents pages until I opened this “Create Form” button and realized that this was a means of creating surveys or ballots of the site. Though, the elections tab is meant to focus of creating ballots. The issue here is that there are no apparent ways to set up options for parties to vote on. The election form merely shows a name, instructions, and who can see the post. Once saved the screen refreshes with no indication that any information is missing. When returning back to the election screen, nothing was saved.
Task 4: Marketing to Students Much like previous tasks the space to find how to create events was clearly marked, though using this tool proved difficult. After starting to create an event and fill out the form, much like the Election page, I was redirected back to the beginning of creating the event with no indication of anything being missing and no further instructions – despite the button at the bottom of the page listing “Next.” There was no means of adding images or visually appealing aspects to advertise through the site, despite there being events advertised with images on the home page. Because you cannot move forward until all elements are filled, it became frustrating when nothing was marked as missing or incorrect. Beyond that, progress is unable to be saved if I were to leave the page for further direction and return. Overall the dropdown menu items were limited and the experience was unengaging.
26
27
Usability Testing Final Report
Positive Findings The MyOrgs site conformed to these heuristics in a number of ways. 1. Effective • Desired tasks are clearly created • Buttons and pages are emphasized and clearly marked • Forms are short and to the point 2. Error Tolerant • Options are minimal to avoid confusion • Task buttons are clearly labeled 3. Efficient • Once on organization management page all actions can be found off the same drop-down menu • Pages are set up in almost the exact same way so finding a button or piece of information on the page is quick and easy 4. Easy to Learn • Once you are able to complete one task on the site, it is relatively quick to access what is needed for another 5. Intuitive Use • Actions perform in a manner that is similar to other websites 6. Aesthetic • Site has a unified look • Pages are clean, clear, and easy to read • No clashing images, colors, or font choices
Barthe
Negative Findings A number of negative findings in terms of these heuristics were also noted. For each, a description of the issue, a severity ranking, and recommendation are given. 1. Effective – How completely and accurately a task can be undertaken Severity Ranking Issue Explanation Recommendation Unable to Complete Task Unable to complete form to create an election ballot. Perform test of Election creation form for issues. Minor Negative Use Confusion of whether the election or form page should be used to create votes due to lack of options. Clearly set directions on the election and forms pages as to what these tasks are meant to be used for. Make options for elections more apparent. 2. Error Tolerant – How easily can mistakes be fixed and prevented Severity Ranking Issue Explanation Recommendation Major Negative Use Explanations for missing information not provided on most forms. Carry over prompting from the uploading documents page to other submission pages. Unable to Complete Task When “saving” or clicking “next” on the election, forms, and events pages, the original page reloads even when all fields have been filled. Perform test of submission pages and update the process. Major Negative Use Unable to delete past documents, forms, etc.- which leads to being unable to delete forms that have been messed up. Create option for deleting past forms, documents, and elections as well as a means of archiving past documents so as not to confuse members. Minor Negative Use Leave organization button is predominant click through link when trying to reach an organization to manage. Link to reach organization’s page is text while the leave organization option is a large blue button similar to that of the buttons on the rest of the site to complete needed tasks. Make option to reach organization’s page more apparent. Create a second more distinct button for leaving a group – perhaps the color red, or gray if wanting to stick with the minimalist color palette so as easy to distinguish between and create a more intuitive decision.
28
29
Usability Testing Final Report
3. Efficient – The speed at which a task can be done Severity Ranking Issue Explanation Recommendation Minor Negative Use When reaching the organization management landing page three separate tabs are opened. Change click through links to reach landing pages within the same tabs. Minor Negative Use To reach the organization management landing page it takes 7 clicks and some scrolling to access. Create an option in the first dropdown menu to reach organizations a user has access to edit. 4. Intuitive Use – How easily users can infer how to interact with site options Severity Ranking Issue Explanation Recommendation Major Negative Use Drop down menu to find organizations that users are a member of is not intuitive to find. There are three separate menus on the home page to sift through. Create one singular drop down menu with a more distinguishable mechanism to open it. 5. Aesthetic – How the site look Severity Ranking Issue Explanation Recommendation Cosmetic Issue/Minor Negative Use All pages throughout the management site looked almost identical. It became very easy to begin creating a form when really wanting to create an election. Provide some visual or distinguishing aspect between pages. Cosmetic Issue The entire site used very few graphics, had no means of interacting with the visual design of the page, and provides only two options to upload images – one of which (the events) cannot be accessed. Create more ways to create distinguishing designs on pages. Add more visual design elements throughout site.
Barthe
Recommendations After going through the four tasks, I found that the MyOrgs site was not entirely user intuitive. There is little to no direction to complete tasks on the platform itself. In order to find directions students have to search through UMass Dartmouth’s SAIL page off the main website for PDF walkthroughs of how to complete tasks. While this does provide some ease in usability, it was also difficult to find through the UMass main site which made things even more frustrating. More explicit directions on the site on how to complete specific tasks are needed. Along with that, more notable and visually apparent navigation is needed. Tasks that are used more often have links and buttons that are less noticeable to the eye - especially at a quick glance while the only points on the site that seem to be highlighted are “join a group” and “leave this organization.” While one of these is great for engagement, the other is frustrating as it makes it easy to accidentally leave an organization when you don’t mean to. The main page of the MyOrgs site is clean and visually appealing, though there is little room to create a visually appealing page for each individual organization. To get students and prospective members’ attention, one has to rely on their event being pushed to the main page, or an organization being found within the categories section of the organization list - which isn’t set up in any logical way. There should be more categories with fewer organizations in each for potential members to be able to find what they are looking for more easily as well as a system that doesn’t take a long time to load when opening a second page of results or endless scrolling. Clubs with names at the beginning of the alphabet are favored for this reason.
30
31
Usability Testing Final Report
Conclusions Several issues were found through this heuristic evaluation. Through use of the persona Sawyer Egan, I have been able to suggest a number of recommendations on how to address these issues.
References MyOrgs, University of Massachusetts Dartmouth. Retrieved from https://umassdartmouth.campuslabs.com/engage/ Nielsen, Jakob (1994). Retrieved from https://www.useit.com/papers/heuristics/heuristic_list.html Quesenbery, Whitney. Using the 5Es to Understand Users. WQusability.com. Retrieved from https:// www.wqusability.com/articles/getting-started.html
Barthe
Appendix B: Personas
32
33
Usability Testing Final Report
Barthe
Appendix C: Usability Test Plan
MyOrgs Test Plan Gabriella Barthe
34
35
Usability Testing Final Report
Purpose The purpose of this report is to outline the usability test plan for the University of Massachusetts Dartmouth MyOrgs site. This particular test is aimed at gathering information on user’s feelings, problems and frustrations while trying to complete a base set of tasks outlined by the test plan. By completing a base set of tasks we will be able to see how easily the site can be navigated and how difficult each of these tasks are to complete.
Problem Statement & Test Objectives This usability study will assess the user experience of the University of Massachusetts Dartmouth MyOrgs site through participant completion of a number of tasks on both a qualitative and quantitative level focusing on their ability to complete tasks, their overall feelings, problems and frustrations. After assessing the website with a Heuristic Evaluation the following criteria and concerns will be the focus of the Usability Test based on those results: •
Finding a new organization that interests you
•
Joining a club or organization
•
RSVPing for an event
•
Procedure for creating and editing an election
•
Procedure for creating and editing an organization event
•
Navigation: are users able to easily navigate the site?
•
User Satisfaction: What are user frustrations and pleasures?
Barthe
User Profile During the Heuristics Evaluation stage, I created 2 Personas that represent Student Leaders with access and need to use the University of Massachusetts Dartmouth MyOrgs site. Based on the characteristics of these Personas, the requirements for participation in the test plan have been adapted. These characteristics are not exact from the Personas but rather the opposite, these characteristics were the starting point for the Persona. The following characteristics make up the user profile we will be looking for in this usability test plan: •
Age 18-24
•
Current University of Massachusetts Dartmouth Student
•
Any college acceptable
•
Holds an Executive Board position in a University of Massachusetts Dartmouth organization
•
Has a general knowledge of the Internet
•
Has used the MyOrgs Website before
The most important criteria here is that user participants hold an Executive Board position in some capacity at the University of Massachusetts Dartmouth. Without this, they will not be able to access areas of the site these tasks are aimed at.
36
37
Usability Testing Final Report
Methodology Tasks & Scenarios Participants will be given the following tasks to complete using the think aloud protocol. Each task is created with a specific scenario in mind and correlates to specific purposes of the site. Scenario 1: You are a looking for a new club to join on campus. Knowing that MyOrgs is the place to go for more information on organizations at UMass, you decide to see what you might find. •
Sign in to site
•
Finding a new organization that interests you
•
Joining a club or organization
•
RSVPing for an event
You are the President of an organization. SGA requires that all organizations hold an election on campus every year for a new eboard. You also have to update your Bylaws on the MyOrgs site. •
Create and edit an election
•
Upload new Bylaws
You are the Secretary of an organization. You know that MyOrgs allows you to advertise events on the site so you decide to create an event for an upcoming meeting. •
Create and edit an organization event
Barthe
Preparation Participants will sit at a standard laptop computer with internet access. Preparation for testing will include •
Setting up computer and surrounding desk space to be clear of other materials.
•
Making sure testing area is calm and quiet.
•
Preparing physical questionnaires and testing materials.
•
Preparing video consent form.
•
Setting up camera equipment and screen capture.
•
Preparing area for moderator to take notes and read script aloud.
Testing The testing process will include five sessions with University of Massachusetts students. Participants will be guided through each session by a moderator with a script provided. The moderator will guide participants through the following aspects of testing. •
Pre-Test Questionnaire
•
Task-Scenarios
•
Task analysis
•
Post-Test Questionnaire
All questionnaires will be read aloud by the moderator and responses will be recorded on paper response sheet provided. All participant action will be recorded via camera and screen capture software.
Think Aloud Protocol Participants will be asked to use a think aloud protocol through out the task-scenario portion of the session. Participants will be asked to speak out loud the task at hand as they are performing it as well as any emotions, thoughts, or means of processing they are having while undergoing that task-scenario. With this participants will be asked to voice
38
39
Usability Testing Final Report
any frustrations, successes, and point out any functions they had to do to complete that goal – ie clicking on links, searching keywords, etc.
Session Length Below is a tentative schedule for the test session. Each session length is subject to change based on participant’s ability to complete the tasks. •
Introductions (5 min) o This will include a briefing of the test at hand, why we are here, who the moderator is, and filling out the consent forms.
•
Pre-test (5 min) o The Pre-Test will be administered before scenarios are given and answers will be recorded by the moderator.
•
Scenarios (40 minutes) o Moderator will guide participants through scenarios and ask them to complete a number of tasks. With each task, moderator will record notes on the events at hand.
•
Task Analysis (5 min) o Participants will be asked to rate their experiences with completing each task. Moderator will record answers.
•
Post-test (5 min) o Post-Test will be administered after scenarios are given and answers will be recorded by the moderator.
Barthe
Appendix D: Pre-Test Questionnaire Pre-Test Questionnaire 1. Thinking about your past experiences online, have you ever found a website particularly pleasing, appealing, or easy to use? Explain. 2. Thinking about your past experiences online, have you ever found a website particularly frustrating, unappealing, or difficult to use? Explain. 3. When looking for a club or organization to join what information is most important to find? 4. Think about your experiences with the University of Massachusetts Dartmouth MyOrgs website. How often do you access this site? What are your main reasons for using the site? 5. What are your initial feelings regarding your prior experiences with the University of Massachusetts Dartmouth MyOrgs site?
40
41
Usability Testing Final Report
U1 Pre-Test Questionnaire 1. Thinking about your past experiences online, have you ever found a website particularly pleasing, appealing, or easy to use? Explain. Yes, unsure of a specific one. Intelligently designed, streamlines, nothing useless, makes sense design wise and from a usability standpoint.
2. Thinking about your past experiences online, have you ever found a website particularly frustrating, unappealing, or difficult to use? Explain. Yes. Websites pre-graphic design purple background with light blue text horrible. Things that require login for no reason, profiles needed to buy things, slow, poor graphics.
3. When looking for a club or organization to join what information is most important to find? What they do, the purpose of the club.
4. Think about your experiences with the University of Massachusetts Dartmouth MyOrgs website. How often do you access this site? What are your main reasons for using the site? Access MyOrgs a few times a month. Currently run a large organization on campus, in past smaller orgs. Needs it to run elections, update information etc.
5. What are your initial feelings regarding your prior experiences with the University of Massachusetts Dartmouth MyOrgs site? When any website is labeled UMass Dartmouth I immediately think it will be shit. It has been updated since I started using it. Whenever I need to do something more than surface level it breaks down and makes me want to cry.
Barthe
U2 Pre-Test Questionnaire 1. Thinking about your past experiences online, have you ever found a website particularly pleasing, appealing, or easy to use? Explain. There are many websites that are appealing and easy to use. This ranges from amazon to Google. They are just nice to look at graphics wise as well as easy to understand, maneuver and use.
2. Thinking about your past experiences online, have you ever found a website particularly frustrating, unappealing, or difficult to use? Explain. In comparison to other websites I have found the WUMD site deplorable in terms of not only layout, but it just hurts to look at. My eye has no idea where to go first or how to properly read it.
3. When looking for a club or organization to join what information is most important to find? Time commitment, Cost (If any), meeting times, any event that club or org is hosting, a cut and clear thing that says what my role would be.
4. Think about your experiences with the University of Massachusetts Dartmouth MyOrgs website. How often do you access this site? What are your main reasons for using the site? I try to avoid using the MyOrgs website as much as possible. It feels like it’s always down or malfunctioning, and if not either of those its just simply a pain to find an organization and find the details about it. I only ever use this site when a club or org that I am already a part of asks. Currently it almost feels like using the website for anything other than officiating membership is a fruitless effort.
5. What are your initial feelings regarding your prior experiences with the University of Massachusetts Dartmouth MyOrgs site?
42
43
Usability Testing Final Report
My initial feelings is that it does not have anywhere near the polish that many other UMD sites have. It’s a pain to look through and find things, it’s a hassle to even get the right info since even the clubs and orgs dislike it so much.
Barthe
U3 Pre-Test Questionnaire 1. Thinking about your past experiences online, have you ever found a website particularly pleasing, appealing, or easy to use? Explain. I spend a lot of time online, and I find most social media sites quite easy and enjoyable; along with most online shops.
2. Thinking about your past experiences online, have you ever found a website particularly frustrating, unappealing, or difficult to use? Explain. The UMass sites are slow and out of date. I find they glitch out, lead down dead ends, and take so long to get their final point.
3. When looking for a club or organization to join what information is most important to find? Their contact, their roster, their constitution, their advisor.
4. Think about your experiences with the University of Massachusetts Dartmouth MyOrgs website. How often do you access this site? What are your main reasons for using the site? I use the site every day; sometimes for up to 3 hours a use. I approve and maintain every club on campus.
5. What are your initial feelings regarding your prior experiences with the University of Massachusetts Dartmouth MyOrgs site? I hate my orgs. It wastes my time.
44
45
Usability Testing Final Report
U4 Pre-Test Questionnaire 1. Thinking about your past experiences online, have you ever found a website particularly pleasing, appealing, or easy to use? Explain. Sites like Twitter and Facebook are easy to use but in terms of being appealing I’m not sure. Most sites are pretty straight forward nowadays. Squarespace sites tend to be pretty.
2. Thinking about your past experiences online, have you ever found a website particularly frustrating, unappealing, or difficult to use? Explain. I usually think of COIN when I think of this. It doesn’t work most of the time, it takes forever to load, and I hate my life every time I use it.
3. When looking for a club or organization to join what information is most important to find? What do they do, who is involved with it, and are they actually running.
4. Think about your experiences with the University of Massachusetts Dartmouth MyOrgs website. How often do you access this site? What are your main reasons for using the site? I rarely go on the MyOrgs site. I think it is pointless and even though SAIL wants us to use it we get nothing out of it.
5. What are your initial feelings regarding your prior experiences with the University of Massachusetts Dartmouth MyOrgs site? I don’t think it is worth the time. Students don’t use the site and people don’t like being forced to do things.
Barthe
U5 Pre-Test Questionnaire 1. Thinking about your past experiences online, have you ever found a website particularly pleasing, appealing, or easy to use? Explain. Most sites are standard and easy to use. They all tend to follow the same guidelines. If it is clean, not too busy and actually says what it is on it I like it.
2. Thinking about your past experiences online, have you ever found a website particularly frustrating, unappealing, or difficult to use? Explain. Do you remember those websites from the early days of the internet when no one really knew what a good website was but they’re for some reason still paying for a domain name? Every so often I look up something random and a search result brings me to a website with a 5mm pixel of a puppy on repeat as a background, the cursor is a pumpkin that changes color and its playing a horrible song that I can’t stop.
3. When looking for a club or organization to join what information is most important to find? I usually don’t look for other clubs to join I just make them myself.
4. Think about your experiences with the University of Massachusetts Dartmouth MyOrgs website. How often do you access this site? What are your main reasons for using the site? I like checking data, I find it interesting. Being able to see numbers and update things is fun to me. I think I go on the site once or twice a month just when I’m interested in seeing how things are going.
5. What are your initial feelings regarding your prior experiences with the University of Massachusetts Dartmouth MyOrgs site? Its okay. The site looks fine but it takes a while to find anything.
46
47
Usability Testing Final Report
Appendix E: Moderator Script
Test Moderator Script Introduction Hello, my name is Gabriella Barthe. I am a graduate student in the Professional Writing and Communications program. Today I will be guiding you through a Usability Test of the University of Massachusetts Dartmouth MyOrgs Website. The aim of this test is not to judge your ability to use the site, but rather to gather information on the site itself. If at any time during the process you need my assistance, do not be afraid to ask. For the remainder of this process I will be speaking from a script. This is to ensure continuity throughout all users. If at any time you need a clarification, ask me to pause and I can do my best to re-explain for you.
Pre-Test Questionnaire Before we begin, I would like for you to complete a Pre-Test Questionnaire. Please answer these questions as openly and honestly as you can.
Cameras & Videotaping Permission Throughout this whole process cameras will be recording your experience, as well as recording the screen itself. The microphone will pick up your voice as you speak aloud. I ask that you vocally describe what you are doing, what you are seeing, and what you are thinking while executing the tasks at hand. The recordings are for the purposes of reviewing specific situations, and to aid in our research. With your permission to record, we will move forward with the process. [hands consent release] When you have completed this form please let me know.
Barthe
Introductions For reference on what we will be doing here today, the rest of this process will include completing tasks on the University of Massachusetts Dartmouth MyOrgs website to see what the experience is like for its users. As I stated before, this test is not in regard to you but the website itself. The more insight you can give us on your feelings and experiences while completing these tasks, the better. Please think out loud as you complete these tasks. By this I mean narrate your actions “I see these images on the screen…,” “I am clicking on …,” “I am interested in this …,” “Where is the button?” For each task I will give you a scenario and task to complete out loud. Do your best to complete each task and navigate the site on your own. If you are confused with what I am asking you to do please let me know.
Tasks For each of the following tasks I will read to you a scenario and then the task at hand. When I complete the description of the task you can begin navigating the site. Please remember to think out loud and let me know when you have completed the task. Are you ready to begin? In front of you is the University of Massachusetts Dartmouth My Orgs site. If you are not already logged-in please do so using your UMass Dartmouth School email address and password. Once you are logged in your first situation is as follows: You are a new student to the campus and have been directed to the MyOrgs site for use with clubs and organizations. Explore the site and familiarize yourself with the content. Let me know when you have completed this action. You have realized an opening in your availability during the week. You have decided to take on a new activity. Unsure of what clubs have been active on campus, you decide to check the MyOrgs site for more information. Please browse the MyOrgs site for a club
48
49
Usability Testing Final Report
or organization you would be interested in joining and request membership. Let me know when you have completed this action. You have decided to attend an event for an organization you are not a member of on campus to see what other groups are like. Find an event on the MyOrgs site you would like to go to and RSVP for the event. Let me know when you have completed this action. The organization you are on the Executive Board for has been told by SGA that you must hold elections for next year’s executive board on MyOrgs. Access your organization’s MyOrgs management site and familiarize yourself with the functions available to you. Once you feel familiarized with this section of the site create an election ballot for your members. Let me know when you have completed this action. You have come to realize you messed up the dates on your election ballot. Go back and edit your first ballot to lengthen the election length by one day. Let me know when you have completed this action. Your organization has decided to hold an event on campus to raise money for Relay for Life. To garner interest in this event, you have decided to advertise to students on MyOrgs. Create an event on the MyOrgs site for this fundraiser. Let me know when you have completed this action. There was a spelling mistake in your event. You just realized and need to go back in to change the issue. Go back and edit your Event. Let me know when you have completed this action.
Post-Test Questionnaire You have now completed the tasks at hand for this process. Thank you so much for your cooperation. Before you leave, we have a few questions we would like you to answer. Please be as honest as possible with your responses and let me know if you have any further questions. When you have completed this questionnaire, you are free to go.
Barthe
Appendix F: Logs U1 Notes Total Time: 49 minutes 05 seconds. Login didn’t require anything. Task 1: • • • • • • • • • • • • •
Uninterested in looking through the tabs on the site. Nothing listed under events at all. Finding interesting clubs – surprised at the number of clubs. Wish there was a show all function Wish for autoscroll function Noticed poetry orgs Where’s goose fighting club? “Nothing really interests me I do enough” Found a register button Tufts is in news section. All news is Spring 2018 but last are from September Forms ?? Uninterested expression arms crossed laid back stance.
Task 2: • • • • • • •
Didn’t want to join a club Mistakenly clicked events tab Art History Club is dead Scrolled all the way down to find the join tab when it was at the top Feels as though clickables will be at the bottom “I don’t like anything UMass” Aggravated
Task 3: • • • •
No events listed Checked past events knew that was there. Noticed “free food” perks Couldn’t register because it passed.
Task 4: •
Surprise at number of members
50
51
Usability Testing Final Report
• • • • • •
Finding the manage click makes sense Landing page for 20 cent fiction blank and useless “This can be like a blip in the corner” “Forms and documents could be worded differently … I don’t know what the difference is here.” “I’ve never seen someone use the news tab” Found new section of service hours. “What’s service hours? I don’t even know what this is for?”
Task 4 b: • • • • • • •
• • •
Include instructions I don’t know what’s that? Save goes to create ballots Tried to submit multiple screens redirected back If you delete a field even before hitting submit it will highlight red This should be a doodle poll “It gives you no instructions of what to do” Can’t click on the space to type in an answer. Tried tabbing and arrow keys etc. o Hit okay to try to switch boxes thinking it will move to next box o Gave an error message. Can move fields around Can make multiple ballots Page properties? o Can label a page o Form properties sends back to access restrictions confusion arises o Back to ballots o Election different than ballots “This would be nice if it was all just one thing” o “This is easier to fuck up” o Accidentally hit save sends back to elections page.
Task 5: • • • •
Editing easily found Accidentally extended it by one month Doesn’t actually know what day he set it on Originally actually did set the wrong day.
Task 6: • • •
Never had to do this before Theme options Add another date – wish it was either right under or above start date start time but makes sense its separate because you can add different locations
Barthe
• • • • • • • • • • • • •
Happy ooo followed by who the fuck needs a map No place for room number even for on campus stuff It keeps same start time not end time for second date Changes end time if you change the date No real predictive features No enter your own or other features for themes Event categories don’t make sense for what the company wants to do “student org event I guess” Would like to be more specific o “Maybe if every theme had its own set of options” Perks: “What does credit mean?” Pulling on hair while filling out form No real way to put anything original Has to submit for approval
52
53
Usability Testing Final Report
U2 Notes Task 1: •
User seems generally happy and excited to partake in the test
•
Ranks the main page as okay
•
Dislikes the “show more” button on the main list of orgs
•
Dislikes the categories and notes that they are not at all accurate
•
“I’ve never looked at this news section before”
Task 2: •
I’m already in so many clubs though
•
“oh baby”
•
I like that there are photos on the club pages but I think they could be bigger
•
Button to join was easy to find
Task 3: •
No events listed except ones that had already past over a year ago.
•
Could not complete task
Task 4: •
Landing page is confusing o Went back a page because thought he did something wrong
•
Confused on where to go from landing page checked all three drop down navigation bars before he found the right one.
•
Looped back to personal organization listings o “Why are past orgs listed first?”
•
Noted that forms, documents, and elections as the three options were confusing based on the similarities of what can be done with them
•
Confused by lack of directions
•
Created a form instead of a ballot.
Task 5: •
Did not realize they created the wrong thing.
Barthe
Task 6: •
Familiar set up to the rest of the site
•
Everything seems pretty straight forward
•
Enjoyed section more than other parts of the site
54
55
Usability Testing Final Report
U3 Notes Throughout entire session user was upset about being on the site, loud, and very energetic. Task 1 & 2: •
This part of the site is fine, slightly tedious but it does what it needs to do
•
Found aspects of the site with ease.
•
Slightly annoyed by the fact that no new events had been approved recently but knew where to go.
Task 3 & 4: •
“so I know where to find this normally and I think its fucking stupid”
•
Shows me part of the site that is not accessible by most participants in which she can manage other organizations and points out how there is no “next” button to look at all of the options on orgs. Instead she has to return repeatedly.
•
Back on track – annoyed – “I hate this site so much”
Task 5 & 6: •
Spent the rest of the session talking about how no one uses the site to find events, etc. They usually use sites like facebook, doodle poll, or just word of mouth for all of it.
Recognizes the use on SAIL and SGA’s side but the site itself doesn’t provide a good representation of that for the users. Regular students don’t use the site and its on the orgs to make them. Visibly angry from the very start. “This is just redirecting me and I don’t know what I’m supposed to do next.”
Barthe
U4 Notes Overall indifferent to the site when started looking at it. •
Liked the home page before signing in – had good visuals.
•
After signing in the page was uneven o There was a notification that popped up on the left and created a huge blank space on the right.
•
Tried checking categories for clubs and ultimately picked a random one because didn’t want to scroll.
•
Got bored of the site very quickly.
•
Gave up on finding the ability to manage an organization. Had to point it out for her to move forward.
•
Visually frustrated with election ballots
•
Quit trying to make a ballot when the page redirected.
•
Stayed frustrated for the rest of the session. o Didn’t have the energy to answer questions fully after being aggravated with the site functions. Would reply in short answers and didn’t like the site.
56
57
Usability Testing Final Report
Appendix G: Post-Test Questionnaire Post-Test Questionnaire 1. Please rate your experiences with each of the tasks you were asked to complete. Task 1:
Very Difficult
Somewhat Difficult
Neither Difficult nor Easy
Somewhat easy
Very Easy
Somewhat Difficult
Neither Difficult nor Easy
Somewhat easy
Very Easy
Somewhat Difficult
Neither Difficult nor Easy
Somewhat easy
Very Easy
Somewhat Difficult
Neither Difficult nor Easy
Somewhat easy
Very Easy
Somewhat Difficult
Neither Difficult nor Easy
Somewhat easy
Very Easy
Task 2:
Very Difficult
Task 3:
Very Difficult
Task 4:
Very Difficult
Task 5:
Very Difficult
Task 6:
Barthe
Very Difficult
Somewhat Difficult
Neither Difficult nor Easy
Somewhat easy
Very Easy
Please comment on what was easy and difficult to do. Be sure to highlight any particular frustrations or pleasurable experiences.
2. Would you be willing to use this system again? 3. Would you be able to navigate this system in the future on your own? 4. Do you feel the system is good as it is or has space to grow? 5. Do you feel confident enough in this system to instruct another person on its use? 6. Overall: How easy or difficult do you find the site as a whole? 7. Has your experience with the site now differed from your experiences in the past? 8. Has your initial impression of the site changed at all from this experience? If so, please comment on what may have changed your impression.
58
59
Usability Testing Final Report
U1 Post-Test Questionnaire 1. Please rate your experiences with each of the tasks you were asked to complete. Task 1: Familiarizing self with site
Very Difficult
Somewhat Difficult
Neither Difficult nor Easy
Somewhat easy
Very Easy
Neither Difficult nor Easy
Somewhat easy
Very Easy
Neither Difficult nor Easy
Somewhat easy
Very Easy
Neither Difficult nor Easy
Somewhat easy
Very Easy
Neither Difficult nor Easy
Somewhat easy
Very Easy
Task 2: Requesting to join org
Very Difficult
Somewhat Difficult
Task 3: RSVP for an event
Very Difficult
Somewhat Difficult
Task 4: Election Ballot
Very Difficult
Somewhat Difficult
Task 5: Editing Ballot
Very Difficult
Somewhat Difficult
Task 6: Creating an Event
Barthe
Very Difficult
Somewhat Difficult
Neither Difficult nor Easy
Somewhat easy
Very Easy
Please comment on what was easy and difficult to do. Be sure to highlight any particular frustrations or pleasurable experiences.
Nothing was pleasurable about it. The initial 20 cent fiction page was frustrating. No need for a giant logo and how many members they have. Aggrivating to click another button.
2. Would you be willing to use this system again? Yes, I have to. I don’t have a choice. If SGA didn’t recuire it I would probably use Facebook. No one uses myorgs to search for events except for maybe new students so maybe I should start to reach that demographic. 3. Would you be able to navigate this system in the future on your own? Yes. 4. Do you feel the system is good as it is or has space to grow? Has space to grow. It is usable the way it is but can always be improved. The picture on the main page still has a humans vs zombies table which is before my freshman year. (They don’t exist anymore) 5. Do you feel confident enough in this system to instruct another person on its use? Yes. 6. Overall: How easy or difficult do you find the site as a whole? First time you’re on it, it’s confusing. After you’ve used it once or twice it’s pretty simple. 7. Has your experience with the site now differed from your experiences in the past?
60
61
Usability Testing Final Report
Yes. I’ve just gotten better at it, nothing has changed with the site. I’ve just used the website a lot, so I know where things are. I don’t know how to explain that. MyOrgs did not improve anything since I started using it, I just learned how to use their suboptimal website. 8. Has your initial impression of the site changed at all from this experience? If so, please comment on what may have changed your impression. No, not really. This helped me look at it a bit more critically but it didn’t highlight anything I hadn’t noticed before.
Barthe
U2 Post-Test Questionnaire 1. Please rate your experiences with each of the tasks you were asked to complete. Task 1: Familiarizing self with site
Very Difficult
Somewhat Difficult
Neither Difficult nor Easy
Somewhat easy
Very Easy
Neither Difficult nor Easy
Somewhat easy
Very Easy
Neither Difficult nor Easy
Somewhat easy
Very Easy
Neither Difficult nor Easy
Somewhat easy
Very Easy
Neither Difficult nor Easy
Somewhat easy
Very Easy
Task 2: Requesting to join org
Very Difficult
Somewhat Difficult
Task 3: RSVP for an event
Very Difficult
Somewhat Difficult
Task 4: Election Ballot
Very Difficult
Somewhat Difficult
Task 5: Editing Ballot
Very Difficult
Somewhat Difficult
Task 6: Creating an Event
62
63
Usability Testing Final Report
Very Difficult
Somewhat Difficult
Neither Difficult nor Easy
Somewhat easy
Very Easy
Please comment on what was easy and difficult to do. Be sure to highlight any particular frustrations or pleasurable experiences.
2. Would you be willing to use this system again? Yeah, its okay.
3. Would you be able to navigate this system in the future on your own? Probably. I might forget a thing or two but now that I know what I’ve messed up I can figure it out.
4. Do you feel the system is good as it is or has space to grow? It definitely could be more straight forward. Also the visuals are pretty boring.
5. Do you feel confident enough in this system to instruct another person on its use? Not at all.
6. Overall: How easy or difficult do you find the site as a whole? I think the site for a general student is fine, but if you want to actually create anything on it just don’t. Go use facebook.
7. Has your experience with the site now differed from your experiences in the past? I barely used the site in the past so yeah.
Barthe
8. Has your initial impression of the site changed at all from this experience? If so, please comment on what may have changed your impression. No. I’m still going to avoid using the site and I think it isn’t set up well.
64
65
Usability Testing Final Report
U3 Post-Test Questionnaire 1. Please rate your experiences with each of the tasks you were asked to complete. Task 1: Familiarizing self with site
Very Difficult
Somewhat Difficult
Neither Difficult nor Easy
Somewhat easy
Very Easy
Neither Difficult nor Easy
Somewhat easy
Very Easy
Neither Difficult nor Easy
Somewhat easy
Very Easy
Neither Difficult nor Easy
Somewhat easy
Very Easy
Neither Difficult nor Easy
Somewhat easy
Very Easy
Task 2: Requesting to join org
Very Difficult
Somewhat Difficult
Task 3: RSVP for an event
Very Difficult
Somewhat Difficult
Task 4: Election Ballot
Very Difficult
Somewhat Difficult
Task 5: Editing Ballot
Very Difficult
Somewhat Difficult
Task 6: Creating an Event
Barthe
Very Difficult
Somewhat Difficult
Neither Difficult nor Easy
Somewhat easy
Very Easy
Please comment on what was easy and difficult to do. Be sure to highlight any particular frustrations or pleasurable experiences.
Most things aren’t apparent. The website makes completing a task take ages to do.
2. Would you be willing to use this system again? I don’t want to, but I have to.
3. Would you be able to navigate this system in the future on your own? Yes I would.
4. Do you feel the system is good as it is or has space to grow? It needs improvement. 5. Do you feel confident enough in this system to instruct another person on its use? Yes I do. 6. Overall: How easy or difficult do you find the site as a whole? Yes, I know more parts of it and they’re all terrible.
7. Has your experience with the site now differed from your experiences in the past? I didn’t like it but I hate it more and more every day.
66
67
Usability Testing Final Report
8. Has your initial impression of the site changed at all from this experience? If so, please comment on what may have changed your impression. Yes, I hate it more than when I started. Nothing has changed and it still doesn’t make sense.
Barthe
U4 Post-Test Questionnaire 1. Please rate your experiences with each of the tasks you were asked to complete. Task 1: Familiarizing self with site
Very Difficult
Somewhat Difficult
Neither Difficult nor Easy
Somewhat easy
Very Easy
Neither Difficult nor Easy
Somewhat easy
Very Easy
Neither Difficult nor Easy
Somewhat easy
Very Easy
Neither Difficult nor Easy
Somewhat easy
Very Easy
Neither Difficult nor Easy
Somewhat easy
Very Easy
Task 2: Requesting to join org
Very Difficult
Somewhat Difficult
Task 3: RSVP for an event
Very Difficult
Somewhat Difficult
Task 4: Election Ballot
Very Difficult
Somewhat Difficult
Task 5: Editing Ballot
Very Difficult
Somewhat Difficult
Task 6: Creating an Event
68
69
Usability Testing Final Report
Very Difficult
Somewhat Difficult
Neither Difficult nor Easy
Somewhat easy
Very Easy
Please comment on what was easy and difficult to do. Be sure to highlight any particular frustrations or pleasurable experiences. The website is set up in a way that makes it hard to find something if I’m I don’t know what I’m looking for. I wasn’t able to find links right away. 2. Would you be willing to use this system again? Yes. 3. Would you be able to navigate this system in the future on your own? I probably would be able to, yes. 4. Do you feel the system is good as it is or has space to grow? I think everything has room to grow, really. I think the graphics could be more eye catching. 5. Do you feel confident enough in this system to instruct another person on its use? No. 6. Overall: How easy or difficult do you find the site as a whole? It is a bit confusing but eventually things start to look the same. It isn’t set up for people to just scroll through though. 7. Has your experience with the site now differed from your experiences in the past? Not really. 8. Has your initial impression of the site changed at all from this experience? If so, please comment on what may have changed your impression. Nope.
Barthe
Appendix H: Release Form Recording Permission Form I hereby give my permission for my participation to be recorded as part of my involvement as a user in the University of Massachusetts Dartmouth MyOrgs usability Test. I understand and consent to the use and release of all recorded information pertaining to the usability test. I further understand that the recording and any highlights extracted from those recordings may be used for review by the client. Quotes may be used within papers as part of research. I understand that any recording methods may be used for the purposes described in this release form without further permission. I understand that if for any reason I do not want to continue I can leave at any time during this session.
Print Name _____________________________________________________
Signature _______________________________________________________
Date ________________________
70