Comprehensive Usability Evaluation by Meghann

COMPREHENSIVE USABILITY EVALUAITON Usability evaluation investigating the Psychology 301 website hosted using the software application known as BeachBoard. Multiple techniques were utilized such as interviews, surveys, heuristic evaluations, card sorting, symbol matching as well as usability lab tests

TABLE OF CONTENTS Introduction_______________________________________________________________2 Interviews & Surveys (User Needs Analysis)______________________________________________ 2 Heuristic Evaluation_____________________________________________________________________ 2 Symbol Matching________________________________________________________________________2 Card Sorting____________________________________________________________________________2 Usability Lab Test________________________________________________________________________3

Background______________________________________________________________________3 Method____________________________________________________________________4 Recruitment and Selection of Participants_____________________________________________________4 Informed Consent________________________________________________________________________4 Equipment______________________________________________________________________________4 Setting_________________________________________________________________________________5 Researchers Task________________________________________________________________________5 Participants Task________________________________________________________________________5 Post Task Survey________________________________________________________________________6

Usability Test______________________________________________________________6 Task 1. Find Assignment Schedule ________________________________________________6 Task 2. Find reading assignment___________________________________________________________6 Task 3. Navigate through the site to take quiz 24_______________________________________________6 Task 4: Locate Class policies______________________________________________________________7

Symbol Usability Evaluation__________________________________________________7 BeachBoard Symbology___________________________________________________________________7 Usability Test Construction of the BeachBoard Symbols_________________________________________7

Card Sorting Task__________________________________________________________8 Heuristic Evaluation ________________________________________________________8 BeachBoard Violations___________________________________________________________________8 Psychology 301 Related __________________________________________________________________9 Heuristics in Detail_____________________________________________________________________10

Overall Results____________________________________________________________22 Heurtistic Evaluation Results______________________________________________________________22 BeachBoard Related Violations____________________________________________________________23 Psychology 301 Related __________________________________________________________________24 Usability Test Performance Data Results_____________________________________________________24 Symbol Usability Evaluation Results_________________________________________________________25 Card Sorting Task Results_________________________________________________________________25

Positive Attributes of Psy 301________________________________________________26 Conclusion and Recommendations ___________________________________________27

Introduction The following usability evaluation investigates the Psychology 301 website hosted using the software application known as BeachBoard at California State University, Long Beach (CSULB). The goal is to identify significant design problems and offer suggestions to improve the website, while keeping in mind the objectives of the site and the primary users. To meet this goal, Herron, Minakata, Nguyen, & Sulaitis, Inc., conducted interviews and usability testing on BeachBoard from May 10th to May 20th, 2008. The investigation carried out used the following methodology:

Interviews & Surveys (User Needs Analysis) Interviews and surveys were conducted with the primary and secondary users of the site. Primary users, individuals who use the site most often, were defined as current students of the Psychology 301 course. Two psychology courses, Psychology 200 (Research Methods) and Psychology 310 (Introductory Statistics), were surveyed to gather information on these potential primary users. Questionnaires were administered to each of these courses, which allowed investigators to develop a better understanding of the goals and needs of the primary users. Secondary users, who interact with and have extensive experience with the site, were defined as the instructors of the course. Semi-structured interviews were conducted with Psychology 301 course instructors and graduate assistants to gain a better understanding of the course. Heuristic Evaluation A comprehensive evaluation of the Psychology 301 course website was performed using heuristic evaluation techniques based on Nielsonâ&#x20AC;&#x2122;s (2005) usability guidelines for Homepages and Web Usability. Four investigators performed independent evaluations of the website and then compiled, as well as rank ordered the heuristic violations by severity. Recommendations to correct each violation found are provided. Symbol Matching Two symbol tests were developed using the existing iconography found on the Psychology 301 website to access the level of recognition in primary users. The tests consisted of a matching section, where the participants matched each icon with a written description (found on the website), and a fill-in section where the participants were asked to write in their best guess of the iconâ&#x20AC;&#x2122;s meaning. Card Sorting A card sorting task that required participants to sort 60 cards labeled with Psychology 301 course content was used to gather information on how primary users would organize the information architecture of the course. Participants were asked to organize the content into piles and then come up with a label that best describes the their groupings. These findings provide

direct insight as to whether the current website is organized effectively to facilitate primary users. Usability Lab Test Four participants from CSULB performed selected tasks on the website. The tasks chosen were determined from the information gathered from interviews and surveys and directly corresponded with severe violations found in the heuristic evaluation. The qualifying criteria to participate in the evaluation were that participants must be psychology majors or planning to become psychology majors and have not previously taken the Psychology 301 course. These findings provide visible insight into the thought process of primary users as they navigate the Psy 301 website to complete a given goal.

Background Psychology 301 is an online course, entitled â&#x20AC;&#x153;Psychology as a Discipline and Profession,â&#x20AC;? which is offered at California State University, Long Beach (CSULB) to psychology majors. The main goal of the course is to educate psychology majors about the general field of psychology (a full list of goals can be found at the end of this section). It is a required course for all Psychology majors and must be taken no later than their first semester in the major. Students who do not complete the course within their first semester of being admitted to the major can loose their status as a Psychology major. There are approximately 200-400 students enrolled in the course each semester. Due to such high volume, the majority of the course is conducted online. There are 3 professors who are actively involved in the course throughout the semester as well as 2 graduate assistants. Course instructors are primarily responsible for the development and maintenance of the Psychology 310 website. This includes updating the site with quizzes, assignments and announcements as well as responding to student problems and questions regarding the course. The graduate assistants help students with assignments and procedural questions. Since Psychology 301 is primarily an online course with little direct student-teacher interaction, the websiteâ&#x20AC;&#x2122;s design directly effects whether or not students are able to find, access, and understand the information provided. With so many students enrolled in the course, it is vital that the information presented online empowers the users, allowing them to complete assigned tasks and achieve the goals set forth by the course instructors. These goals include: Goals of Psychology 301: 1.) To acquaint students with information about the profession such as the major fields, characteristics of psychologists and information about professional organizations. 2.) To help students find a focus within the field of psychology.

3.) To provide information which will help students design a plan of study that will maximize their chances for admission to graduate school or make them competitive in the job market with a BA. 4.) To simulate the process of applying to grad school or seeking a job so that the actual process will be familiar and therefore more successful. 5.) To cover study skills, this may enhance student classroom performance, i.e., the use of Library Tools and Internet searches. 6.) To familiarize students with key historical and ethical issues which psychologists may confront in research or professional settings. 7.) To encourage students to function independently, become organized, and pay attention to detail. 8.) To allow students to function in a professional manner--adhering to deadlines, following through on commitments, and learning to accept the consequences of their actions.

Method Recruitment and Selection of Participants Four participants were recruited by evaluators through either Psychology classes (Psychology 200 and Psychology 310) or direct approach. Participants included students from California State University, Long Beach (CSULB). The participants used had characteristics similar to the ones found in a demographic survey conducted prior to the start of usability testing. The only difference being that the four participants used had no prior experience with the Psychology 310 website on BeachBoard. The participants were all undergraduate students currently enrolled in a psychology course. Informed Consent Consent forms were given to the participants before the start of the experiment. The informed consent provided participants with information regarding procedures to be used, including the use of cameras for recording the participant during the experiment, as well as duration, potential risks, confidentiality, and the right to voluntarily end participation at any time. After signing the Informed Consent prior to the start of the experiment, the researcher gave each participant a guided tour of the Center for the Study of Advanced Aeronautics (CSAAT) labs, introduced the researchers involved, and informed the user of the type of data being collected. Equipment The computer the participants performed tasks on was a Lenovo laptop running a Windows XP operating system with an IntelÂŽ Core â&#x201E;˘2 Duo CPU at 2.20GHz using a 14.1 inch 4

display. The Psychology 301 website on BeachBoard was presented to the participant through the Mozilla Firefox browser (as recommended in the instructions on Psy 301). A Logitech web camera was affixed to the top of the PC display to capture a head shot of the participant as they navigated the site. In a room separated by a one-way mirror, the investigators viewed the participants’ display using remote software, RealVNC. The data was captured via the software application, Camtasia, which captures and records real time screen shots of the participants’ computer. The web camera images as well as voice conversations were collected using voice and image conference software. A Logitech headset (with attached microphone) was used to facilitate communication between the investigators and participant. Setting The usability test was conducted in the Center for the Study of Advanced Aeronautics Technologies (CSAAT) lab at California State University, Long Beach. Participants were individually observed through a one way mirror and were also digitally recorded. Each session was approximately an hour and fifteen minutes long. Instructions were given to participants via a two-way headset. Researchers Task At least two researchers were present to perform each experiment. One researcher recorded the experiment on Camtasia, measured the amount of mouse clicks each user executed, measured the amount of time each task took with a stop watch, and kept notes of the user’s actions. The second researcher interacted with the participants directly by performing introductions, giving instructions, giving tasks over the headset, conducting the card sorting and symbol surveys, giving demographic surveys, and debriefing the participant at the end of the experiment. Participant Task Participants were asked to perform four tasks using the Psychology 301 website on BeachBoard and to verbalize their thoughts and motivations behind each mouse selection before making it. The four tasks include: (see Usability Test, p. 5) Task 1. Find the Assignment Schedule for Psy 301. Task 2. Find the reading assignment associated with assignment 24. Task 3. Navigate through the site to take quiz 24. Task 4: Locate the class policies regarding turning in late assignments.

Post Task Survey After completing the usability portion of the experiment, the participants were given a survey designed to evaluate their experience. Next, the participants were given a brief fill in the blank exam testing their understanding of the symbols/ icons used throughout the Psy 301 website followed by a matching exam using the same symbols. Once finished with this portion of the experiment the participant was asked to complete a card sorting task in which they were given 60 index cards each with a short label containing course content. The participants were then asked to organize the cards into piles according to similarity. After 4-10 piles were constructed the participants were asked to create new titles for each pile. At the end of the experiment the participants were asked to complete a short demographic survey and then debriefed.

Usability Test The usability test was designed and conducted based on the user needs analysis and heuristic evaluation. Four participants from CSULB performed selected tasks on the website. The tasks chosen were determined from the information gathered from interviews and surveys and directly corresponded with severe violations found in the heuristic evaluation. The qualifying criteria to participate in the evaluation were that participants must be psychology majors or planning to become psychology majors and have not previously taken the Psychology 301 course. These findings provide visible insight into the thought process of primary users as they navigate the Psy 301 website to complete a given goal. The four tasks include: Task 1. Find Assignment Schedule 

Using BeachBoard, please locate the “Assignment Schedule” that is viewable through BeachBoard (not a download). Find the schedule that corresponds to your last name. From this schedule please locate the assignment that is due on May 8 th, 2008. Please read the title of the assignment out loud.

Task 2. Find reading assignment 

From the “Assignment Schedule” page, please locate the reading assignments on BeachBoard that are necessary to complete Assignment 24. Once you have found them, you will see that, there are a number of readings for this assignment. Please open any one of these readings and read the first sentence out loud.

Task 3. Navigate through the site to take quiz 24 

BeachBoard allows students to take quizzes and surveys. In order to take a quiz you must first know the procedures. Please find and read the instructions for taking a quiz on BeachBoard. After reading the instructions, please find and retake take quiz 24 titled (About Academia). Please answer questions 1, 3, 5, and10 (Do not worry about giving 6

the correct answers). Once you have entered your answers. Please submit the quiz. Make sure that you receive conformation that it has been received. Please say “done” once you see that your quiz is entered and recorded. Task 4: Locate Class policies Please locate the page in BeachBoard that contains information regarding Psy 301’s policy on turning in late assignments. When you’ve located this information, please read the first sentence of the page out loud.

Symbol Usability Evaluation BeachBoard Symbology Throughout the BeachBoard website, an abundance of symbols are used to aid the users in making their decisions. Symbols are usually provided because they are supposed to convey meaning beyond the information that is transmitted via the text on the BeachBoard website, as opposed to the implementation of symbols for the sake of aesthetics.

Usability Test Construction of the BeachBoard Symbols All of the possible symbols that the user could have encountered on BeachBoard were collected and compiled. After the compilation of these symbols, they were organized into two symbol usability tests. These tests were used to assess the validity of the symbols that are used throughout the BeachBoard website. The first of these tests is a comprehension test, which asks the participant to give a description of what the symbols mean to them. The participants were given instructions to take their “best guess” if they did not have an answer to the symbol. (See Appendix A) The second symbol test was a recognition test, which required the participant to match the symbol with the corresponding meaning/answer. Essentially, this test is analogous to a matching test in where students are given test items and they are to match these items with the correct answers that are randomized on the right column (which contained all the answers to the test items; plus distracter answers) to ensure that the test is not too revealing. (See Appendix B) The results of these tests yielded two forms of data (i.e., qualitative and quantitative) that were used to assess BeachBoard symbols. The qualitative data, which was retrieved with the comprehension test, provided direct interpretations of these symbols from the target population (i.e., the student user) rather than an interpretation of the symbols from the designers of the website. The quantitative data, retrieved from the recognition test, provided data on the students’ ability to correctly (i.e., in an accuracy percentage) recognize the meaning of the symbols found on BeachBoard.

Card Sorting Task A Card Sorting Task was used to provide investigators with important Primary user feedback. The card sorting task was developed using the current “course content” found on the Psychology 310 website. This assisted investigators in finding out how primary users would structure course information which may provide the course with a more intuitive navigation and taxonomy. The cards were created using 3 x 5 index cards with 1.5 x 4 Avery binder labels. Sixtytwo cards were created and each were labeled with course content that could be found on the site. For example, a card may have the label “Access to Quizzes” or “External Links” which they had to organize. The task used an “open card sorting” technique. This method provided participants with cards containing course content with no pre-existing groupings. Participants were presented with the stack of labeled index cards, blank index cards and a pen. The participants’ task was to sort the labeled cards into separate stacks by how they would naturally find them and not try to design a navigation menu or use the current Psychology 301 menu. Once the cards were sorted, participants were also instructed to label each stack in a way that described the content.

Heuristic Evaluation The following section contains a list of violations of the 10 usability heuristics developed by Jakob Nielsen as they apply to the Psychology 301 course website, hosted by the BeachBoard software at California State University, Long Beach (https://www.BeachBoard.csulb.edu). A team of 4 investigators independently evaluated the current version of the Psychology 301 by performing a number of probable tasks that students and faculty would perform. These tasks were developed from information regarding “course goals” taken directly from the Psychology 301 website. The website was evaluated in terms of overall flow, ease of use, and navigation. * The following is an ordered list of heuristic violations found and ranked by degree of severity.

BeachBoard Violations Heuristic violations listed in this category are directly related to the design and software development of BeachBoard. Thus, the following list attempts to assist the web and software developers of BeachBoard. Top 10 Heuristic Violations (Most Important) BeachBoard 1.) The system does not inform user of saving on quiz/test page when the “save” button at the bottom is selected. However, it does inform user when individual questions are “saved”)

2.) Bread crumbs are not used consistently (they use inconsistent names with the locations they relate to) 3.) No control over the presentation of announcements, instructions, and/ or assignments. (Unable to checkmark complete or “read” sections) 4.) No visual indication that a link has been previously selected 5.) High Level Navigation buttons don not indicate the users current position 6.) instructions are given from CSULB not BeachBoard 7.) No search bar exists for BeachBorad 8.) Undo/Redo function is not supported but Back/Forward is supported 9.) Breadcrumb bar disappears when scrolling takes place 10.) Bread crumb actions are not consistent with description. (Bread crumb links take the user to different pages than assumed or in some cases where they have not been previously)

Psychology 301 Related Heuristic violations listed in this category are related to the development and design of the Psychology 301 website on BeachBoard. Thus, the following list is aimed to help the faculty that developed and currently update the Psychology 301 website. Top 10 Heuristic Violations (Most Important) Psychology 301 Related 1.) Links are not available when referenced 2.) Current order of High Level Navigation (HLN) buttons do not follow user mental models 3.) Inconsistent format and depth of information in the site. (Some sections are shallow and other are deep) 4.) “Assignment schedule” contains too many colors and does not effectively inform users the meaning of the colors. 5.) Confusing use of the color Red to symbolize a topic heading and clickable link) 6.) Heading titles are too long

7.) Help function available only for BeachBoard not PSY 301 8.) HLN button titled “Instruction” does not effectively describe the information within it 9.) Instruction on using BeachBoard for Psy301 is not easy to find (FAQ exist in “Instructions” button) 10.) Schedule button is redundant

Heuristics in Detail This section contains a detailed description of Jakob Nielsen’s 10 usability heuristics and how they are in direct violation of these standards on the Psychology 301 website. Violations of each heuristic are divided into two sections: BeachBoard (BB) related and Psychology 301 related to help facilitate corrections. Each violation has been given a title, a description of the violation, the location of where it was found, and a recommendation to correct the violation.

Nielson Heuristic 1. Systems Status and Update “The system should always keep users informed about what is going on, through appropriate feedback within reasonable time.” Beach Board Related 1. High Level Navigation buttons should indicate the users’ current position Location of Problem: All Pages Description of Problem: The navigation buttons on the left side of each page do not help or hint to the user their current position within BB. Although each page has a title that mirrors the button heading, this is not a clear and quickly recognizable tactic. Also, these headings can easily be lost as the user scrolls down the page, since they do not stand alone at the top of the page. This is a well known standard and may decrease the user’s understanding of the site as well as hinder the flow. Recommendation to fix Problem: As the user selects a button, have the button change color to indicate to the user what page they’re on.

2. No visual indication that a link has been previously selected

Location of Problem: Anyway a link exists COURSES > PSY 301 PSY-DISCIPLINE & PROFESSION 3861 > SCHEDULE

Description of Problem: BB does not keep the user informed that a link has already been visited by changing to a different color (which is standard in many web applications). This may disorient the user as he navigates through the site causing him to accidentally re-visiting already viewed links, especially when a page (i.e., Instruction and Assignment pages) contains a large number of links. Recommendation to fix Problem: After a link has been viewed by the user the color of the link should change to a different color. This new color should remain the same whenever the user re-visits the site. 3. Breadcrumb bar disappears when scrolling takes place Location of Problem: All Pages Description of Problem: Breadcrumbs (that indicate the user’s most current position in a site) should remain persistent, and always visible at the top of the page. In BB, the breadcrumbs are associated with the page frame and thus as the user scrolls down the breadcrumbs disappear and become useless as navigational aids. Recommendation to fix Problem: Have the breadcrumbs always remain on the top of the page and actively reflect the users position. Psychology 301 Related 1. “Assignment schedule” contains too many colors and does not effectively inform users. Location of Problem: Syllabus section Description of Problem: (Color and information overload)

The third link brings the student to the page with prerequisites and assignments. Although the creator of the website did a great job categorizing the students by last name, the next page is quite confusing to the student. This violates the “Visibility of System Status” heuristic as well as the “Aesthetic and minimalist design” heuristic because there is too much information jumbled into one section, which might confuse the student. Recommendation to fix Problem: Instead of using various colors to indicate where an assignment is located, the designer can create categories under which certain assignments will be located to make it less confusing to the student.

Nielson Heuristic 2. Match Between System and the Real World “The system should speak the users' language, with words, phrases and concepts familiar to the user, rather than system-oriented terms. Follow real-world conventions, making information appear in a natural and logical order.” Psychology 301 Related 1. Current order of HLN buttons does not follow user mental models. Location of Problem: All Pages Description of Problem: The navigation buttons (in red) do not reflect any noticeable order for their placement. Currently 10 different navigation options exist which may confuse the user and tax their memory as to which button contained which information. Recommendation to fix Problem: 1. As mentioned above (in Aesthetic & Minimalist Design”) remove the “Schedule” button. (its info moved to the “Assignments” button) 2. Separate the existing buttons into two groups of separate color 3. The groups shall be related to each other as follows a. Vital Class info i. Announcements ii. Syllabus iii. Assignments iv. Text, Readings v. Quizzes/ Surveys

vi. Instructions (change to “FAQs”) vii. External websites 2. HLN button titled “Instruction” does not effectively describe the information within it. Location of Problem: All pages (High Level Navigation) Description of Problem: The High Level Navigation button titled “Instructions” does not effectively describe the information it contains which may cause users to accidentally select it, causing a navigational error. “Instructions” may refer to a number of non-related information. For example, although the button refers to “instructions” on how to use the BB site, users may interpret is as “instructions” for specific class assignments… which is not the case. Recommendation to fix Problem: Change the button title from “Instructions” to “Course Instructions.” 3. Terms used do not actively reflect the users’ mental model. Location of Problem: COURSES > PSY 301 PSY-DISCIPLINE & PROFESSION 3861... > ANNOUNCEMENTS

Description of Problem: The use of the HLN button titled “Instruction” is not detailed enough. The user may confuse the meaning of this label as Psychology 301 instruction when in fact the instructions are FAQ about Beach Board. Recommendation to fix Problem: Change the current title “Instructions” to a more detailed title such as “FAQ”. However the HLN title should be chosen based on the primary users vernacular. (determined through survey, prototyping, or interview) Nielson Heuristic 3. User Control and Freedom “Users often choose system functions by mistake and will need a clearly marked "emergency exit" to leave the unwanted state without having to go through an extended dialogue. Support undo and redo.” BeachBoard Related

1. No control over the presentation of announcements, instructions, and/ or assignments. (Unable to checkmark complete or “read” sections) Location of Problem: Announcement Page COURSES > PSY 301 PSY-DISCIPLINE & PROFESSION 3861... > ANNOUNCEMENTS

Description of Problem: Users are not able to visually indicate what announcement they have already read. Each time the user logs-in they are bombarded with an often long list of announcements that they have to pause and read to make sure nothing new has slipped in. This overloads the user, making BB a daunting task. Recommendation to fix Problem: Allow the user to either “check box” or change the background color for any announcement, instruction, or assignment. This will allow the user to quickly see what they have already read and not be FORCED into re-reading old notes. Also, once the item has been selected, the list order could be re-shuffled and the checked item could be sent to the bottom of its section. Or the section can be collapsed into its main heading. This will reduce the visual clutter of each page. 2. Undo/Redo not supported but Back/Forward is supported. Location of Problem: N/A Description of Problem: Undo/Redo not supported but Back/Forward is Recommendation to fix Problem Back/Forward available. Save function available Breadcrumbs/home available Menu still available

Nielson Heuristic 4. Consistency and Standards

“Users should not have to wonder whether different words, situations, or actions mean the same thing. Follow platform conventions.” Beach Board Related 1. The system does not inform user of saving on quiz/ test page when the “save” button at the bottom is selected. (It does inform user when individual questions are “saved”) Location of Problem: COURSES > PSY 301 PSY-DISCIPLINE & PROFESSION 3861 > QUIZZES/SURVEYS > ACCESS TO QUIZZES > TAKE ASSESSMENT 24 ABOUT ACADEMIA

Description of Problem: As a user I saved some of my responses for the quiz and I was not informed that the responses were saved (i.e., the screen just “blinked” but no saving status was shown). (When saved at the bottom of the page… individually it works.) Recommendation to fix Problem Add a feature that will keep the user informed about the system status (e.g., a short message that tells the user what questions have been saved) such as the pop-up that double confirms your intent to submit responses. 2. Breadcrumb heading and textbooks headings are inconsistently labeled Location of Problem: Anywhere in the site where there is a “Courses” breadcrumb Description of Problem: The breadcrumb titles do not use terms that accurately reflect where their link takes the user. For example, the breadcrumb titled “Courses” takes the user to different page than the tab courses. Recommendation to fix Problem The breadcrumb titles should directly reflect where the link takes the user. 3. Doesn’t have a global symbol key (i.e., The Grade page has a symbol key at the bottom) Location of Problem: Grades COURSES > PSY 301 PSY-DISCIPLINE & PROFESSION 3861... > TOOLS

Everywhere

Description of Problem: None of the icons that the site uses are explained in a site key (e.g., the MyGrades function) Recommendation to fix Problem: The site should have a system-wide key, or legend, that will inform the user about the meaning behind the icons.

4. Bread crumbs are not used consistently (they use inconsistent names with the locations they relate to) Location of Problem: Main page Description of Problem: When you transfer/click on a tab the users are not informed where they are being taken for instance if you click on tab Gradebook, you are taken to a page that says my grades. If you try to look at the cookies it says that the last item clicked is tools. However I never clicked anything that said tools. The user is not informed that they are being taken to the Gradebook they just end up there actually on the that page. Recommendation to fix Problem After user clicks on a tab on the right hand side of the screen (Gradebook) a screen should pop up and inform the user that they are going/ being redirected/ to the Gradebook Before users are taken to the next page. There should also be some identifiers on the screen that let them know where they are located in the system There should be a header on the top of the page to help the users know where they are. Currently they are listed in some of them but not in Gradebook, faculty/staff, and announcements. 5. Bread crumb actions are not consistent with description. (Bread crumb links take the user to different pages than assumed) Description of Problem: When I click on the breadcrumbs to go back to a prior-, or higher-, level, I am not taken to the identical webpage. I was sent to the courses page, which looks completely different than the homepage on My BeachBoard; this could confuse the user. Recommendation to fix Problem:

Make the titles on the My BeachBoard textboxes to be links to the corresponding pages. This will ensure that the user will know these pages exist beyond the My BeachBoard tab. Psychology 301 Related 1. Confusing use of the color Red to symbolize a topic heading and clickable link Location of Problem: COURSES > PSY 301 PSY-DISCIPLINE & PROFESSION 3861 > SCHEDULE

Title of Problem: Confusing use of the color Red to symbolize a topic heading and clickable link Description of Problem: BB instructs its users that when a topic is in red (on the instructions page) it is a clickable link with additional information within it. However, this is not standard throughout BB. In many cases the color red is used only to represent a topic heading and not a link, while blue links as scattered throughout other sections. Sections with links already contain a visual icon (a folder) to the left of it indicating that more information is available within. The color red may suggest an ERROR and disrupt the users flow. Recommendation to fix Problem: Since BB already contains the use of visual pictures (folders) to represent that extra information is available, BB doesnâ&#x20AC;&#x2122;t need to change the color of internal links to red. Instead use blue (the universal standard) to symbolize that a link exists.

2. Inconsistent format and depth of information in the site. (Some sections are shallow and others are deep) Location of Problem: Menu tabs Description of Problem: Pages not always set up the same. Some have folders describing what is inside; some have text then listing all the assignments Recommendation to fix Problem Menu tab description could be a pop up when mousing over tabs Menu could side out or drop down with options/expanded menu

3. Ambiguous icons used in the review section after the user submits a quiz. Location of Problem: COURSES > PSY 301 PSY-DISCIPLINE & PROFESSION 3861 > QUIZZES/SURVEYS > ACCESS TO QUIZZES > REVIEW ASSESSMENT: 24 - ABOUT ACADEMIA

Description of Problem: The review section, after a quiz, shows ambiguous icons that one must hover over to get an indication of what they may mean. Recommendation to fix Problem Show a key that will explain what the system icons represent.

Nielson Heuristic 5. Error Prevention “Even better than good error messages is a careful design which prevents a problem from occurring in the first place. Either eliminate error-prone conditions or check for them and present users with a confirmation option before they commit to the action.” Psychology 301 Related 1. BB doesn’t inform/ warn the user that they must complete the entire quiz (can’t stop mid way) Location of Problem: Quizzes > 18a GRE

Description of Problem: The user can open the quiz w/out a warning that they must be able to take the quiz all the way through Recommendation to fix Problem: Before the user is allowed to continue, they should get an message that pops up and informs and asks them if they are sure they want to proceed because they will not be able to go back (which is found in the GRE quiz that can not be retaken)

Nielson Heuristic 6. Recognition Rather Than Recall “Minimize the user's memory load by making objects, actions, and options visible. The user should not have to remember information from one part of the dialogue to another. Instructions for use of the system should be visible or easily retrievable whenever appropriate.” 18

Psychology 301 Related 1. Options are not available when referenced Location of Problem: Syllabus section Description of Problem: The second link under the syllabus tab, references the Psychology Department website, however, it does not provide a link to get to that website. This violates the “Recognition not recall” heuristic because it does not provide the student with the option of transferring to the Psychology website to view critical information. Recommendation to Fix Problem: Instead of indicating to the student that this option is available, a link should be provided to the student that will redirect them to the Psychology website so that they can view the office hours.

Nielson Heuristic 7. Flexibility and Efficiency of Use “Accelerators -- unseen by the novice user -- may often speed up the interaction for the expert user such that the system can cater to both inexperienced and experienced users. Allow users to tailor frequent actions.” NONE Found

Nielson Heuristic 8. Aesthetic and Minimalist Design “Dialogues should not contain information which is irrelevant or rarely needed. Every extra unit of information in a dialogue competes with the relevant units of information and diminishes their relative visibility.” Psychology 301 Related 1. Schedule button is redundant Location of Problem: Schedule / Assignments pages COURSES > PSY 301 PSY-DISCIPLINE & PROFESSION 3861 > ASSIGNMENTS

Description of Problem: 19

BB currently holds two High Level Navigational buttons that contain the same information. This unnecessary clutter adds more visual information that the user must interpret to proceed. The Schedule and Assignment buttons contain the same information. Recommendation to fix Problem: Remove the High Level Navigation button titled “Schedule” and move the small amount of information under this button to the button titled “Assignments.” This will allow the user to locate necessary information, since now, all assignment relate information will be located in the same place. Plus, this will minimize the site “busy” appearance. 2. Heading titles are too long Location of Problem: Assignments page COURSES > PSY 301 PSY-DISCIPLINE & PROFESSION 3861 > SCHEDULE

Description of Problem: Under the “Assignments” button the sub-heading titles are too long. For example, “Instructions for Preparation for all required assignments.” Long titles disrupt the “scanability” of each section, forcing the user to waste valuable time reading each section. Recommendation to fix Problem: Keep the sub-heading title short and descriptive. For the example listed above, change “Instructions for Preparation for all required assignments” to “Assignment Instructions.”

Nielson Heuristic 9. Help users recognize, diagnose, and recover from errors “Error messages should be expressed in plain language (no codes), precisely indicate the problem, and constructively suggest a solution. None Found

Nielson Heuristic 10. Help and Documentation “Even though it is better if the system can be used without documentation, it may be necessary to provide help and documentation. Any such information should be easy to search, focused on the user's task, list concrete steps to be carried out, and not be too large.” Beach Board Related 20

1. Give instructions from BeachBoard not CSULB Location of Problem: Everywhere Description of Problem: The BlackBoard service does not provide help documentation for their own site. Recommendation to fix Problem: Create an actual help section that is hosted on the BlackBoard server with instructions on how to use the website, rather than having the school create a help section. 2. No search bar exists for BB Location: Throughout the entire application Description of Problem: No search textbox exists for users to directly use key terms to search the Psychology 301 site. Recommendation to fix Problem: Search textboxes are standards across many internet applications that allow users a more direct way to search for items in a site. Psychology 301 should employ this technique to aid in searching, especially since Psychology 301 contains a wealth of information that not easy to search through using Look See and Decide searching patterns. Psychology 301 Related 1. Help function available only for beach board not PSYCHOLOGY 301. Location of Problem: Psy301 main page > Instructions > General Instructions – Click here for important information (This tag is not helpful)

Description of Problem: Help function available only for BeachBoard not PSYCHOLOGY 301. Help documents are not easy to find, they are under “instructions” where you’d think that you’d find actually assignment instructions. Instructions are more of a “how to” and addressing common problems. FAQs only address explanations to course rules. Recommendation to fix Problem 21

Add Help function for the course Add walkthrough examples or pictures or dummy examples such as a fake quiz Help Chat box Step Guides Course text guide 2. Instruction on using BB for Psychology301 is not easy to find (FAQ exist in “Instructions” button) Location of Problem: Throughout the application Description of Problem: Instructions for Psychology 301 users to use BB are difficult to find. A section that contains FAQ is used to satisfy this category. However, the “help” icon at the top of the page takes the user to a BB technical Help page with instructions for using BB. Recommendation to fix Problem: Psychology 301 should include instructions that allow users to effectively understand how to use the Psychology 301 site through BB. For example, the “instructions” should contain a link to the BB technical “help” page. The technical page should contain “howto-videos” similar to the videos found on the faculty technical help page.

Overall Results Heuristic Evaluation Results Heuristic Evaluation is a method used to identify usability violations of software applications. Four researchers conducted independent heuristic evaluations, which revealed a number of important violations that greatly reduce the effectiveness and usability of the online course Psy 301. To maximize the effectiveness of web developers and course faculty in adopting these findings, the violations have been divided into two groups; BeachBoard Related Violations and Psy 301 Related Violations.

“BeachBoard Related Violations” refer to heuristic violations that may be corrected most easily by the programmers responsible for the BeachBoard software application in that these violations typically require programming experience to correct. The analysis revealed the following violations ranked in degree of severity:

Beach Board Related Violations 1. The system does not inform the user of saving on quiz / test pages when the “save” button at the bottom is selected. (It does inform user when individual questions are “saved”) (Heuristic Violation: Consistency and Standards) 2. Bread crumbs are not used consistently (they use inconsistent names with the locations they relate to) (Heuristic Violation: Consistency and Standards) 3. No control over the presentation of announcements, instructions, and/ or assignments. (Unable to checkmark complete or “read” sections) (Heuristic Violation: User Control and Freedom) 4. No visual indication that a link has been previously selected (Heuristic Violation: Visibility of system status 5. High Level Navigation buttons should indicate the users current position (Heuristic Violation: Visibility of System Status) 6. Give instructions from BeachBoard not CSULB (Heuristic Violations: Help and Documentation) 7. No search bar exists for BB (Heuristic Violations: Help and Documentation) 8. Undo/Redo not supported but Back/Forward is (Heuristic Violation: User Control and Freedom) 9. Breadcrumb bar disappears when scrolling takes place (Heuristic Violation: Visibility of System Status) 10. Breadcrumb actions are not consistent with description. (Bread crumb links take the user to different pages than assumed) (Heuristic Violation: Consistency and Standards) Psychology 301 Related Violations refer to heuristic violations that occur in regards to the Psychology 301 site itself. These violations may best be corrected by the faculty responsible for

the course. The following analysis revealed the following violations ranked by degree of severity. Psychology 301 Related

1. Links are not available when referenced (Heuristic Violation: Recognition Rather than Recall) 2. Current order of HLN buttons does not follow user mental models. (Heuristic Violation: Match Between System and the Real World) 3. Inconsistent format and depth of information in the site. (Some sections are shallow and other are deep) (Heuristic Violation: Consistency and Standards) 4. “Assignment schedule” contains too many colors and does not effectively inform users. (Heuristic Violation: Visibility of System Status) 5. Confusing use of the color Red to symbolize a topic heading and clickable link) (Heuristic Violation: Consistency and Standards) 6. Heading titles are too long (Heuristic Violation: Aesthetic and Minimalist Design) 7. Help function is available only for beach board not PSY 301. (Heuristic Violation: Help and Documentation) 8. HLN button titled “Instruction” does not effectively describe the information within it. (Heuristic Violation: Match between system and the real world) 9. Instruction on using BB for Psy301 is not easy to find (FAQ exist in “Instructions” button) (Heuristic Violation: Help and Documentation) 10. Schedule button is redundant (Heuristic violation: Aesthetic and Minimalist Design)

Usability Test Performance Data Results A post hoc analysis was conducted on the performance data (i.e., no a priori designation of task difficulty was made on the usability testing tasks), which consisted of the amount of time (in seconds) the users took to complete each task and the number of clicks that the users took to finish the each task. The performance data indicated that participants did take longer for the 24

difficult task (i.e., M = 297.25 seconds for Task 3), when compared to the easy task (i.e., M = 173.75 seconds for Task 2) (See Table 1 for individual time scores). Also, participants made more clicks for the difficult task (i.e., M = 22 clicks for Task 3), when compared to the easier task (e.g., M = 6.88 clicks for Task 2.) (See Table 2 for individual values for click data.) These performance data suggest that our task did differ in terms of difficulty and diversity among the tasks that were chosen to be included in the usability test. The task differed in difficulty because of the way in which certain tasks required markedly more clicks and time to accomplish the task. Further, the tasks differed in terms of the amount of navigation and interpretation of the website that the users had to do.

Symbol Usability Evaluation Results The participants completed the two symbols tests (i.e., a comprehension test and a recognition test) in where they were, first, asked to ascribe meaning to each of the symbols that they can encounter on BeachBoard. In this comprehension test, only two symbols were correctly identified (i.e., “Reading” and “Grades”) by the four users. The other finding was that the two out of the four users correctly identified the “Completed” and “Announcement” icons (See Table 3 for individual user scores). The results from this symbols comprehension test indicated that the symbols were not effective at conveying meaning to the users. Secondly, the users were given a “matching test” and this test consisted of the same symbols used in the comprehension test. One of the only differences between this test and the comprehension test was that more possible answers than were possible were also included as distractor items. This recognition test yielded more accurate responses from the users compared to the more difficult symbol comprehension test. For example, four out of four users correctly matched/recognized six symbols on the test (i.e., “Personal Information”, “Correct Answer”, “Edit Document”, “Incorrect Answer”, “Digital Dropbox”, and “External Links.”) Furthermore, four other symbols were correctly recognized by two out of four users (i.e., “Book Open”, “Folder”, “Grades”, and “Announcements”) (See Table 4 for individual user scores.) Overall, these results suggest that the symbols are not effective at transmitting meaning to the users. The lack of convergence on the two tests also indicates that the symbols are not effective. One possibility about this finding is that the users did not see these symbols in their context (except for the “Quiz” symbols, which were provided context on the test.)

Card Sorting Task Results Each participant’s card sorting task was recorded and analyzed for similar categories. Of the 4 participants, 34 categories were created. From these 34 categories, common prefixes were removed to reveal underlying topics and then matched for similarity. After matching categories and removing repetitive topics, 14 categories remained which consisted of all of the previous categories (no categories were left out). The raw data was then entered into Lamantia’s card sorting template (2003) for analysis. 25

As part of the task, users were instructed to create labels for their sorted categories. Some categories that were straightforward matched user created categories such as “Quizzes/Surveys”, “Syllabus”, and “External Websites”. These titles were accurate representations of the information it contained. Other results show that several of options on BeachBoard’s high level navigation menu do not match user-created categories. This may cause a problem for users by leading them astray into different areas of the site, thus extending search times. An example of this misleading problem in the website is the “Assignment” button. By clicking on this button, users expect that this will lead them directly to the individual assignments for the course, however, this button provides information about assignments such as “Information about when assignments become available” or “Instructions for Preparation for all required assignments”. In the card sorting task, the same information in the current “Assignments” tab was re-labeled as “Instruction for Assignments.” For a list of all 14 categories and BeachBoard equivalent see Appendix E. The course content in each user’s card sorting task also showed that that the information layout in the PSY 301 course did not match user expectations. Much of the information found in the current “Instructions” menu button were grouped together as “how to do” questions and labeled “FAQ/Help”. By adding a “Facts” and “Help” button to the current menu, the “Instructions” menu button could be freed up to contain actual assignment instructions, which would match user expectations.

Positive Attributes of Psy 301

The psychology 301 online course is very beneficial to students. However there are problems that are brought about from BeachBoard as well as the layout of the Psychology 301 course that make the website difficult to use. If we could eliminate these problems we could alleviate some of the frustration that is encountered by both the students and faculty. An ideal website would be one that is very intuitive (where you don’t have to search for information as well as understand the course and its requirements). If this were accomplished we would see drastic improvement in students’ success. The recommendations that we have provided are intended to enhance the efficiency of the online course. The course is very valuable as it allows a large number of students to be taught at the same time. There are also other positive strengths and attributes of the site which are listed below: o The website’s current color scheme in the “assignment schedule” demonstrates a clear attempt at considering the users’ needs.

o The current format of information in the “Gradebook” allows easy comprehension and scanability. Students are able to look at the information contained in the Gradebook and quickly understand what it means. o The Sorting tabs on “Announcements” page are very helpful. The tabs allow students to view announcements by various orders (i.e. students can view the announcements that have been posted in the last 7 days). The tabs provide freedom and control for the students. o The current use of iconography in the “Review Assessment” section is helpful for students. They easily understand what the symbols mean when in this context (the two symbols are listed below). Students understood that this symbol meant that they answer this question wrong Students understood that this symbol meant that they good this question correct

Conclusion and Recommendations The following recommendations have been developed based on the findings from the Heuristic Evaluation, Usability, Symbol, and Card Sorting Tests. Findings are based on violations of Nielson’s Ten Heuristics for User Interface Design. These findings provide detailed descriptions, examples, locations and recommendations for each violation.

Finding 1: Links are not available when referenced Description • • •

Information on Psy 301 refers users to different locations without providing a link. Disrupts the flow by forcing the user to search for additional information. Users can not complete an assignment in a linear fashion.

Example • •

Syllabus > On campus meetings Assignment Schedule

Why This Is a Problem 1. Violates “Recognition Rather than Recall” Heuristic 2. Interview: – –

Preferred if the site “flowed” rather then directing users to many different places to complete task. Students have difficulty navigating/searching site for information.

3. Usability Video Clips also demonstrated that during Task 1(finding the Assignment Schedule) student looking/searching for an assignment. The participant went to the assignment schedule looking for the readings that were due and found that only the quizzes were listed. Recommendation (see Figure 1) 

Provide an appropriate link (internal or external) to redirect the user to the referenced information without forcing them to initiate a new search.

Figure 1

Finding 2: Current order of HLN buttons does not follow the user’s mental model Description • •

HLN buttons do not reflect any noticeable order. 10 different HLN buttons exist which may confuse the user and tax their memory as to which button contained which information.

Example •

10 HLN options on left-hand side of page

Why This Is a Problem 1. Violates “Match Between System and the Real World” Heuristic 2. Interview: - Student’s assumptions do not match actual site structure. 3. Card Sorting: “Instructions” = “FAQs” * In the card sorting task users arranged information that is currently under instruction as a new section labeled Facts and Questions (FAQ’s). 4. Usability Video Clips revealed that during Task 1, the participant had difficulty locating the assignment schedule. They were confused which High Level Navigation (HLN) button to select to find the assignment schedule. They were confused because the assignment schedule could be under the “assignment” (HLN) or the “schedule” (HLN). Recommendation (see Figure 2)   

Remove the “Schedule” button and place its information under the “Assignments” button. Separate the existing buttons into groups of separate color. Remove the “Faculty/Staff” button and place its information under the “Syllabus” button.

Figure 2

Finding 3: “Assignment Schedule” contains too many colors which confuse users Description • •

Color and information presentation overload users. Too much information is trying to be conveyed using different color schemes. – Colors should be used as redundant coding not primary coding.

Example •

“Assignment Schedule” page

Why This Is a Problem 1. Violates “Aesthetic and Minimalist Design” Heuristic Recommendation (see Figure 3)  Use alternating color scheme (only two colors) for each assignment.  Use a standard format and size for all information displayed.  Use links to direct users to text and quizzes.  Explain what information is conveyed using colors.

Figure 3

Finding 4: Confusing use of the color RED & inconsistent page format Description • • •

The color red is used to symbolize a clickable link, an important sub-heading, and important information. The meaning of “red” changes in different areas of the website. Page formats are not standardized throughout the website. Some sections are deep forcing novice user to disorient while some are broad causing increased search times.

Example • •

“Instructions” HLN button instructs users that red symbolizes a clickable link No standard format exist for links

Why This Is a Problem 1. Violates “Consistency and Standards” Heuristic 2. Interview: –

“Failure to read instructions cause students to perform poorly in the course” .

3. Usability Test revealed a participant being confused because a link that needed to be selected was written in red. This failed to utilize the standard format of links which are blue underlined text. Recommendation (See Figure 4)  Follow existing standards and use blue and an underline to suggest links.  Previous selected links should change in color.  Use red only to highlight important information.  Keep page layout standard.

Figure 4

The following figure illustrates multiple meanings of red text.

Finding 5: Heading titles are too long, contain too much text Description • •

Long titles disrupt the “scanability” of each section. Disrupts the flow by forcing the user to read each detail.

Example •

“Assignments” HLN button > “Instructions for Preparation for all required assignments”

Why This Is a Problem 1. Violates “Aesthetic and Minimalist Design” Heuristic 2. Interview: – “Preferred if the site “flowed” rather then directing users to many different places to complete task”. – “Failure to read instructions causes students to perform poorly in the course”. 3. Usability Test demonstrated that subjects found blocks of text intimidating. A participant stated that “too much” was used throughout the site especially in the instruction page. This suggests that the use of bullet points would reduce the reading load and increase scanability. Recommendation (see Figure 5)   

Keep headings and sub-headings short and descriptive. Support visual hierarchy or information by making headings visually distinct (i.e. larger and bold). Use bullet points and grouping to promote scanning.

Figure 5

Finding 6: Users get easily disoriented on the website Description • • •

Since many sections of the site appear visually similar, users get confused as to where exactly in the site they are. Breadcrumb titles do not always accurately reflect the user’s current location. Titles used are inconsistent with actual location. Main page title and breadcrumbs are not static. Users can scroll past it.

Example • •

“Assignments” page title is hidden when the user scrolls down Breadcrumb navigation is inconsistent with user’s desired direction. The “Course” breadcrumb takes users away from the PSY 301 course home page.

Why This Is a Problem 1. Violates “Visibility of System Status” Heuristic 2. Usability test showed that the participants were confused by the inconsistent events when using the save button. During the test, the participant was taking an exam and saving the data using the save button located on the right-hand-side of the screen. After opting to save the data, a confirmation “pop up” appears when the save button is selected. However the save button option located at the end of the exam (bottom of screen) does not present a confirmation “pop up” screen. Recommendation (see Figure 6)   

Page titles and breadcrumbs (that accurately describe where the user currently is) should remain persistent at the top of each page. Breadcrumbs should use the same concise term throughout the site which reflects the actual user’s location. HLN should visually depict what page the user is currently on.

Figure 6

Finding 7: Users have no control over how information is displayed Description • • •

Users are unable to distinguish old information from new. Users are unable to sort information. Beachboard does not provide a course specific user search tool (i.e. search box).

Example • •

“Texts, Readings” page forces the user to search for the most recent assignment. Users are forced to navigate the site to find information instead of assisted searching.

Why This Is a Problem 

Violates “User Control and Freedom” and “Match Between System and the Real World” Heuristic

Recommendation (see Figure 7)   

Allow the user to sort information (i.e. New vs. old). This is currently done on the “Announcements” page. Enable users with a way to “checkmark” a completed section. Provide a search box allowing user to easily and quickly find content by inputting search terms.

Figure 7

References Lamantia, J. (2003). Analyzing card sorting results with a spreadsheet template. Retrieved May 1, 2008, from http://www.boxesandarrows.com/view/analyzing_card_sort_results_with_a_spreadshee t template Nielson, J. (2005). Heuristics for user interface design. Retrieved April 30, 2008 from http://www.useit.com/papers/heuristic/heuristic_list.html

Table 1 Performance Data for the Amount of Time (in seconds) it Took Each User to Finish each Individual Task.

User 1

User 2

User 3

User 4

Task Number

Difficulty of Task

Time (seconds)

Task 1

Medium

Task 2

Easy

Task 3

Hard

151

Task 4

Medium

Task Number

Difficulty of Task

Time (seconds)

Task 1

Medium

120

Task 2

Easy

120

Task 3

Hard

156

Task 4

Medium

Task Number

Difficulty of Task

Time (seconds)

Task 1

Medium

232

Task 2

Easy

151

Task 3

Hard

240

Task 4

Medium

251

Task Number

Difficulty of Task

Time (seconds)

Task 1

Medium

305

Task 2

Easy

340

Task 3

Hard

642

Task 4

Medium

453

Table 2. Performance Data for the Number of Clicks it Took Each User to Finish each Individual Task.

User 1

User 2

User 3

User 4

Task Number

Difficulty of Task

Number of Clicks

Task 1

Medium

Task 2

Easy

Task 3

Hard

Task 4

Medium

Task Number

Difficulty of Task

Number of Clicks

Task 1

Medium

Task 2

Easy

Task 3

Hard

Task 4

Medium

Task Number

Difficulty of Task

Number of Clicks

Task 1

Medium

Task 2

Easy

Task 3

Hard

Task 4

Medium

Task Number

Difficulty of Task

Number of Clicks

Task 1

Medium

Task 2

Easy

10.5

Task 3

Hard

Task 4

Medium

Table 3. Symbol Recognition Data (Percentages calculated out of four users)

Symbol

Percent Correct 100 %

100 %

50 %

100 %

75 %

50 %

100 %

75 %

100 %

50 %

25 %

75 %

100 %

Appendix A For the following SYMBOLS: 1. Describe what you think the symbols in the Left column MEAN to YOU. 2. You encountered the following symbols while navigating through BeachBoard. 3. If you are not sure of an items meaning, please write your best guess.

Symbol

Meaning

In the Quiz section:

Throughout the website:

Appendix B For the following SYMBOLS: 1. Compare the symbols on the Left column with one another. 2. Chose the BEST meaning for the corresponding symbol. 3. Choose only ONE answer for each symbol.

Symbol

Your Answer for the Symbol

Meaning

(Write the Letter) A. Incorrect

B. Clipboard

C. Document

D. Page

E. Calendar

F. Personal Information

G. Announcement

H. Partial Credit

I. Assessment

J. Test

K. Correct

L. Folder

M. Paste

N. Grades

O. Edit Document

P. External Link

Q. Tasks

R. Open Book

S. Portfolio

T. Digital Dropbox

Appendix C

User Created Categories

BeachBoard Menu

Announcements

Syllabus

Instructions for Assignments

Instructions

Schedule & Assignment Information

Schedule

Assignments

N/A

Faculty/Staff

Reading Assignments

Texts, Readings

Quizzes/Surveys

Grades & Completed Work

Gradebook

External Websites

Additional Information

N/A

FAQ/Help

N/A

Incomplete Assignments

N/A