Check & Connect
Recommendations Report
Taylor Long | Kelcie McKenney | Max Fulton | ZoĂŠ Leschak
Table of Contents User Experience Testing 03 04 05 06 06 07
Key Findings Directed Storytelling Personas Heuristic Evaluation Feature Desirability Analysis Wireframe Testing
Prototype 08 09 10 11 12 13 14 15 16 17 18 19
Coordinator Profile Page Mentor Profile Page Student Profile Page 1 Student Profile Page 2 Student Profile Page 3 Add Data Pop-Up View 1 Add Data Pop-Up View 2 Add Data Pop-Up View 3 Case Notes Single Case Notes Configure Report Pop-Up Report
Going Forward 20
Suggestions regarding the next steps in the redesigning process
Appendix i ii iii iv v vi
Mentor Persona Coordinator Persona Student Persona Feature Desirability Analysis Data Link to Brisa Interview Wireframe Testing Script
01 | TABLE OF CONTENTS
Introduction
“Check & Connect is a comprehensive intervention designed to enhance student engagement at school and with learning for marginalized, disengaged students in grades K-12, through relationship building, problem solving and capacity building, and persistence. A goal of Check & Connect is to foster school completion with academic and social competence.” –Check & Connect website We were approached to begin testing and start preliminary designing on an electronic version of the current paper form mentors fill out during checks and interventions with students. With a strong focus on testing for user experience, our process was defined by hearing from real mentors and coordinators and developing off of their comments to improve and enhance our design into one that wasn’t only usable but also enjoyable. The following report will cover each step of our testing process along with our final prototype that resulted from this direct user testing. More information and further details are located in the appendix, and resources for further testing are also provided.
INTRODUCTION |02
User Experience Testing Key Findings
During our testing process we hosted Directed Storytelling, created Personas, individually performed a Heuristic Evaluation, did an in-depth Feature Desirability Analysis, and did guided testing with mentors and coordinators on our developed prototype. Found through this process, the following are the four most important takeaways.
Flexibility - Throughout all testing stages, a need for flexibility in the system as determined to be highly prevalent due to difference between sites all the way to differences between mentors. Celebrate Successes - Through speaking with mentors and coordinators, Check & Connect’s emphasis on helping students lead to the realization that celebrating successes should be the main goal of this system. Direct Feedback - While the current paper form is successful at collecting data, it does little to show mentors the results from this collection. This system should immediately give mentors direct feedback so they don’t have to work hard to understand the results of the data they’ve entered. Reward Mentor - Through directed storytelling we found that a lot of members understood why they collected the data but had a difficult time deciphering it. Another key goal of this system should be to reward the mentor for collecting the data by making analysis easier and providing resources that help them in the “connect” piece of Check & Connect.
03 | USER EXPERIENCE
Directed Storytelling
Directed Storytelling allows designers to get first hand experiences and stories which are prompted by questions that the interviewer (designer) asks participants. Using this method, designers can learn so much about a user’s needs just by learning about the user’s daily life. We personally interviewed three Check and Connect employees and the amount of information we learned from them was astounding. This preliminary research method allowed to grasp what C&C employees were looking to improve in the system.
What We Learned ++ Graphs / charts created out of mentor data by coordinator ++ Mentors want to talk with other mentors for support ++ Mentors want more suggestions for interventions (one-on-one activities) ++ Mentors and Coordinators share the same priority of helping students before other duties. ++ Strong focus on monitoring attendance What Was Confirmed ++ The “connect” piece for mentors is the most important ++ Data collection takes up too much time ++ Differences are site by site, need for flexibility ++ Giving students positive feedback and celebrating their successes gives students incentive for improvement in the future
USER EXPERIENCE | 04
Personas
Developing personas is a key stage in the design process. After interviews with actual mentors and coordinators, personas are developed based on these interviews to create a face for the user we are designing for. These personas are referred back to during the design process to make sure all goals are being met. Sarah Clarkson, Mentor Age: 28 Case Load: 40 Students Goals: ++ Wants more time to connect with her students ++ Puts a strong focus on celebrating successes to build confidence ++ Strives to show students what they are capable of achieving
“The most valuable thing I get back from my job is seeing my mentees improve.”
Joan Ericson, Coordinator
Age: 43 Coordinates 20 Check & Connect Mentors Goals: ++ Wants to focus on helping mentors improve their relationship with students ++ Would like to provide mentors with more resources ++ Wants an easier way to keep track of mentors from all schools in her site
“If a few mentors don’t keep up with their data collection, it affects everything.”
Michael Long, Student
Age: 15 Class Load: 7 classes / day, 5 days / week Goals: ++ Really wants to graduate on time ++ Wishes he could catch up on his late assignments ++ Wants to be on the school basketball team, but his grades won’t allow it
“Sometimes I feel like my mentor is the only one who really listens to me.”
05 | USER EXPERIENCE
Heuristic Evaluation
Heuristic evaluation is a method that allows literally anyone to test for usability problems based on “rules of thumb”. In other words, finding problems in a design can be tested by anyone as long as they know basic guidelines on how the design is utilized. We tested the Check & Connect paper form ourselves to find anything that could be improved when moving to the website-based form. Findings
++ ++ ++ ++ ++ ++
Form was great for filling in data, but only takes the first step towards comprehending it No visibility of system status - filled in data but hard to tell what it means Hard to make comparisons No space for notes No flexibility The visibility of system status is the most important aspect because despite the form doing what it is initially intended for (entering data), it doesn’t take the next step to effectively evaluate that data.
Feature Desirability Analysis
Desirability testing is vital in finding what the user feels is important to include in the design. Feature Desirability Analysis is a form of desirability testing that asks the user what attributes are important, as well as what isn’t so important. Analysis of features is generally split into 5 attribute categories in which users can decide from for each feature: Required, desired, Exciting, Neutral, and Anti-features This helps us determine which features Check and Connect employees feel should be required in the design. Findings showed that ALL these features were desired (category Exciter/Delighter or Desired) ++ Generate reports for review by coordinator / by mentee ++ Flag High Risk Students ++ Month (Grid/Spreadsheet) View These three features were most desired ++ Progress Indicators by Data Type: Instead of just showing cumulative progress over a period of time, this would let you filter data for a certain aspect that a mentee might be struggling the most in. This would help focus on the highest priority when it comes to a mentees. ++ Intervention Suggestions: After talking to our client directly during directed storytelling, and asking them via survey if they would like a way to easily access suggested interventions with mentees, it was unanimous that it would be helpful when mentors need a little guidance as to how to interact with their mentees. This feature wouldn’t be intrusive at all, but simply a way for mentors to look up a suggested solution just as you would look up in a Check & Connect manual. ++ Add Notes: From both the directed storytelling and desirability testing, it was clear that there needed to be space for notes in the system.
USER EXPERIENCE | 06
Wireframe Testing
After the first prototype is made, testing begins on mentors and coordinators to determine what is working, what isn’t working, and what needs to be improved in the preliminary design. Scenarios are developed for users to work through during testing to determine how easy the prototype is to use.
Results The following are the results we found that would be applicable towards and design and include general statements from the mentors and coordinators that were tested. ++ ++ ++ ++ ++ ++
Mentors wanted to see a notes section because they felt that would be easier than referring back to a separate notebook. “This is actually really helpful because right now we keep notes in a notebook, but it would be nice to have notes directly with our forms,” -Mentor Mentors thought it would be nice to see how often they met with each mentee within the last month. Coordinators wanted to see how often each of their mentors at their site was entering data. “It would be really nice to see how often they were entering data, too,” –Coordinator. Both mentors and coordinators want to know the number of high risk students and in which category they are high risk in. If the number of high risk students was listed, “That would make it easier. And if there was a way to break it down because of the different factors that cause high risk, it would be easier to meet with each student to problem solve with them.” Mentors and coordinators alike emphasized they would want the capabilities to make a report easily and accessibly. When asked what she would like to see, on mentor said, “Maybe a list of possible reports to choose from based on mentor’s check data.” In regards to seeing the monthly view, mentors noted they liked being able to see the monthly view because they were used to seeing on the paper version.
“I kind of wish we had this now, because it would make it a little bit easier for us,” –Mentor
07 | USER EXPERIENCE
Prototype This prototype section holds screenshots of each page within the prototype and a short description regarding what each of those pages / dashboards holds. To access the interactive portion / wireframe of this prototype follow the link provided below. http://invis.io/MKPBPC5S
Coordinator Profile Page Coordinator Dashboard
This entire page view represents the Coordinator Dashboard. If Joan Erickson logged in, this would be her starting screen.
Check & Connect
My Dashboard Current View: COORDINATOR DASHBOARD
< Back to Mentor
Joan Erickson
Print Report
Role: Coordinator, Mentor Caseload: 6 mentors with 78 students
Students at High Risk
The coordinatorâ&#x20AC;&#x2122;s default view lists all the mentors assigned to that coordinator. The list is default sorted by last name with each entry also showing the number of students assigned to each mentor and the school each mentor works at. Scrolling down further would reveal more mentors. This feature should help coordinators manage their responsibilities more easily. DIRECTED STORYTELLING
Last 8 Months change
change
Phone: 123-456-7890 E-mail: sclarkson@mcneely.ms.edu
JAN
List of Mentors
Joan Erickson Coordinator
Resources
Mentors assigned to Joan Sarah Clarkson 40 Students McNeely Middle School
John Perry 32 Students McNeely Middle School
Lucia Nelson 35 Students McNeely Middle School
David Hicks 12 Students McNeely Middle School
Sortable List
By default, the list of mentors sorts by last name, but also has the option of sorting by Days since last meeting and # High Risk Students. This view shows # High Risk Students selected (in blue) for sorting, moving Sarah Clarkson to the top of the list with 2 high risk students.
SEP
OCT
NOV
DEC
JAN
FEB
Sort by: Last Name | Days since last meeting | # High Risk Students
Add Data
Print Report
High Risk Students: 2 | 28 days since last meeting
Add Data
Print Report
High Risk Students: 1 | 22 days since last meeting
Add Data
Print Report
5 days since last meeting
Add Data
Print Report
4 days since last meeting
Customizable Graph
In this coordinator view, the graph could default to show the number of high risk students per month over the last six months. The coordinator may change the variable shown as well as the time frame. This offers a quick, simple synthesis of whatever the coordinator finds most important. DIRECTED STORYTELLING HEURISTIC EVALUATION WIREFRAME TESTING
PROTOTYPE | 08
Mentor Profile Page Mentor Dashboard
This entire page view represents the Mentor Dashboard. If Sarah Clarkson logged in, this is her starting screen.
Check & Connect
DIRECTED STORYTELLING
Visit Student Profile
Clicking on a student’s name, profile picture, or anywhere on the student’s line that isn’t a button or link will bring the mentor to the Student Profile page.
Sarah Clarkson Mentor
Sarah Clarkson
Print Report
Interventions change
Role: Mentor Caseload: 40 students Days since last meeting: 28
Last 8 Months change
Phone: 123-456-7890 E-mail: sclarkson@mcneely.ms.edu JAN
Students assigned to Sarah Michael Long 9th Grade McNeely Middle School
Stephanie Hanly 9th Grade McNeely Middle School
Jean Martin 3rd Grade McNeely Middle School
Eli Vargas 4th Grade McNeely Middle School
Add Data
This button brings the mentor directly to the Add Data Pop-up where meeting data can be entered in a simplified layout that represents one day.
Print Report
This button brings the mentor directly to the Configure Report Pop-up to generate a customizable, printable report about the student for sharing with the student or perhaps another mentor or coordinator. DIRECTED STORYTELLING
09 | PROTOTYPE
Resources
Current View: MENTOR DASHBOARD
< Back to Coordinator
List of Students
The mentor’s default view lists all the students assigned to that mentor. The list is, by default, sorted by last name with each entry also showing the student’s grade and school. Scrolling down further reveals more mentors.
My Dashboard
SEP
OCT
NOV
DEC
JAN
FEB
Sort by: Last Name | Days since last check-in | High Risk (2)
Add Data
Print Report
High Risk: Unexcused/Unverified Absence | 5 days since last check
Add Data
Print Report
High Risk: Tardee, Skipping Classes | 7 days since last check
Add Data
Print Report
2 days since last check
Add Data
Print Report
3 days since last check
Sortable List
By default, the list of students sorts by last name, but also has the option of sorting by Days since last check-in and High Risk. This view is sorted by High Risk (in blue). High Risk is given more context by listing the areas that make the student high risk.
Customizable Graph In this mentor view, the graph could default to show the number of interventions per month over the last six months. The mentor may change the variable shown as well as the time frame. This offers a quick, simple synthesis of whatever the mentor finds most important for keeping up with all student progress.
HEURISTIC EVALUATION, WIREFRAME TESTING
Student Profile Page 1 | Profile Student Profile
This entire page view represents the Student Profile that the student’s mentor or coordinator could access.
Check & Connect
My Dashboard
Sarah Clarkson Mentor
Resources
Current View: Student Profile
< Back to Mentor
Michael Long
Add Data
hide profile
15 years old 9th Grade McNeely Middle School 6 months in program
DIRECTED STORYTELLING
< Previous | Next >
Print Report
Unexcused Absences change
View Notes
Last 8 Months change
General Notes
Profile Basics
Basic information like age, grade, school and time in program is listed next to a profile photo. A General Notes section offers 1-2 sentences introducing the student. The Parents’ Contact Info link below pops up names, phone numbers, and any other helpful info for communicating with parents or guardians. WIREFRAME TESTING DIRECTED STORYTELLING
At a Glance
This section highlights areas that have improved and areas that have gotten worse as of the most recent visit. Recent Interventions also highlights the most recent actions taken by the mentor. Areas of High Risk are easy to spot in red. All areas show the latest number recorded. At the top right, you can switch to viewing more cumulative trends based on the last month. DIRECTED STORYTELLING FEATURE ANALYSIS
Michael enjoys playing basketball and eating pizza with his friends. He transfered last semester and is having a difficult time adjusting to his new routine.
Parents’ Contact Info >
At a Glance
JAN
SEP
OCT
NOV
DEC
JAN
FEB
Academics: 2013-2014
Based on: Last Entry | Last Month
Areas of Improvement: Number of Missing Assignments (0), Tardees (1).
Areas of Concern: Unexcused/Unverified Absences (7 - High Risk), Number of D’s or F’s (3).
Change Year
0
Qtr. GPA 2.00
Cum. GPA 2.00
Credits Earned 16
Credits Possible 18
0
1.83
1.92
22
24
Grades
A’s
B’s
C’s
D’s
F’s
1st Qtr.
0
1
4
1
2nd Qtr.
0
2
1
3
3rd Qtr. 4th Qtr.
Recent Interventions: Share “check” data (1), Problem Solve About Risk (1)
Met State Acad. Standards
Math:
Y
Reading:
Y
Writing: N
STUDENT DATA
< February >
Enter Table Edit Mode
Day
M
Date
3
Year
View:
T
W
Th
F
M
T
W
Th
F
M
T
W
Th
F
M
T
W
Th
F
4
5
6
7
10
11
12
13
14
18
19
20
21
22
25
26
27
28
1
Meeting
Time Spent (minutes)
30
General Notes
Student Data
This shows the top of the student data section which the mentor can scroll down through to view more.
View Notes
This button opens a pop-up with all the case notes for a student, listed chronologically. DIRECTED STORYTELLING, WIREFRAME TESTING
25
5
30
Academics
This section shows grades from the last year including not only D’s and F’s, but also A’s, B’s and C’s. This information can be entered by the mentor but ideally could be automatically populated by the school’s student information system.
Add Data
This button brings the mentor directly to the Add Data Pop-up where meeting data can be entered in a simplified layout that represents one day.
Month
Trend
Customizable Graph In this student profile, the graph could default to show the number of absences per month over the last six months. The mentor may change the variable shown as well as the time frame. This offers a quick, simple synthesis of whatever the mentor finds most important for keeping up with the students progress.
HEURISTIC EVALUATION, WIREFRAME TESTING
Print Report
This button brings the mentor directly to the Configure Report Pop-up to generate a customizable, printable report about the student for sharing with the student or perhaps another mentor or coordinator. DIRECTED STORYTELLING
PROTOTYPE | 10
Student Profile Page 2 | Month View Student Data by Month
This view shows what the screen looks like after scrolling down or clicking the hide profile link next to a students name. The default view shows all the data by month in the familiar grid view of the paper-based C&C spreadsheet. Days of the week and the day of the month are both shown at the top and months can be clicked through with the arrows. The current day is always highlighted in blue for easy reference. HEURISTIC EVALUATION
Data Areas
The areas of data collection are modeled exactly after the current C&C spreadsheet and organized by the same categories. Both the Check section and the Connect section had be hidden (accordion style) to simplify the view. HEURISTIC EVALUATION
Recommendations by Area
The small icon next to each area opens a pop-up window with recommended interventions based on high risk data in this area. When an area becomes high risk, this icon turns red to remind the mentor that it contains relevant information. Recommended interventions were quite popular in our feature analysis. DIRECTED STORYTELLING HEURISTIC EVALUATION FEATURE ANALYSIS
11 | PROTOTYPE
Check & Connect
My Dashboard
Sarah Clarkson Mentor
Resources
Current View: Student Profile
< Back to Mentor
Michael Long
< Previous | Next >
Add Data
hide profile
< February >
Enter Table Edit Mode
Print Report
Year
View:
Day
M
T
W
Th
F
M
T
W
Th
F
M
T
W
Th
F
M
T
W
Th
F
Date
3
4
5
6
7
10
11
12
13
14
18
19
20
21
22
25
26
27
28
1
Meeting
Time Spent (minutes)
30
25
5
30
General Notes
*
*
*
Month
Trend
CHECK hide section Academic Data Number of D’s or F’s
1
1
2
3
↑
Number of Missing Assignments
4
2
1
0
↓
Tardee
1
2
2
1*
↓
Skipping Classes
0
0
0
0
Unexcused/Unverified Absence
1*
1
6*
7*
Excused Absence
0
0
0
0
Behavioral referral/infraction
0
0
0
0
Detention
0
0
0
0
Suspension (in/out-of-school)
0
0
0
0
Behavior Data
↑
CONNECT hide section
Top Level Items
At the top of a given day’s entry, a check mark indicates that a meeting took place, time spent tells how long the meeting lasted, and the general notes icon turns blue if general notes were taken at the meeting. Clicking on this icon opens a pop-up of the notes. Associating notes with meetings was one of the most unanimously desired features in our feature analysis. DIRECTED STORYTELLING HEURISTIC EVALUATION WIREFRAME TESTING FEATURE ANALYSIS
High Risk Highlighting
Once data is entered that represents high risk for that area, the cell automatically turns red. This gives the mentor immediate feedback and relieves them of the burden of remembering. A default high risk value could be set by the system, but the school could edit this based on their own assessment of what “High Risk” means for each area. DIRECTED STORYTELLING HEURISTIC EVALUATION WIREFRAME TESTING FEATURE ANALYSIS
Enter Table Edit Mode
Clicking this link makes the whole table editable. By default the table is not editable to protect the data and allow for clicking on cells to reveal notes. HEURISTIC EVALUATION
Trend Arrows
These arrows will show up or down and be color coded by red (bad) and green (good) based on the most recent entry, compared to the one before it. This offers the mentor an immediate notice of changes to inform the conversation while they’re entering data. DIRECTED STORYTELLING HEURISTIC EVALUATION WIREFRAME TESTING FEATURE ANALYSIS
Indication of Notes
Each cell with a blue asterisk shows that a note has been recorded specifically about that entry. Clicking on the cell (outside of an edit session) will reveal just this note. DIRECTED STORYTELLING HEURISTIC EVALUATION FEATURE ANALYSIS
Student Profile Page 3 | Year View Student Data by Year Switching to the year view reveals aggregations of each months data, with totals where appropriate. The data collection areas are all the same, but edits cannot be made in this mode as each cell represents the sum of many cells in month view. The view centers on a full school year (not calendar year). Using the arrows next to the school year allows the mentor to check on last years overall data as well.
HEURISTIC EVALUATION
General Notes
The general notes icons turn blue (and have blue asterisks) when any notes have been stored within a month (most months will have this). Clicking on the icon reveals all the case notes recorded during that month. HEURISTIC EVALUATION WIREFRAME TESTING FEATURE ANALYSIS
Check & Connect
My Dashboard
Resources
Sarah Clarkson Mentor
Current View: Student Profile
< Back to Mentor
Michael Long
< Previous | Next >
Add Data
hide profile
< 2013-2014 School Year >
Print Report
View:
Month
Aug
Sep
Oct
Nov
Dec
Jan
Feb
Mar
Apr
May
Year
‘13
‘13
‘13
‘13
‘13
‘14
’14
‘14
‘14
‘14
Meetings
1
4
4
5
6
4
4
Time Spent (minutes)
30
75
90
120
60
85
90
General Notes
*
*
*
*
*
*
*
Year
Month
Trend
Total
↑
550
38
CHECK hide section Academic Data Number of D’s or F’s
0
1
2
5
3
1
3
↑
15
Number of Missing Assignments
1
5
3
10
8
1
0
↓
28
Tardee
2
1
5
13
7
7
6
↓
37
Skipping Classes
0
0
8
10
3
0
0
Unexcused/Unverified Absence
0
1
1
8
0
0
15
Excused Absence
0
0
1
0
0
1
0
↓
2
Behavioral referral/infraction
1
0
3
5
0
2
0
↓
11
Detention
0
0
2
7
0
0
0
9
Suspension (in/out-of-school)
0
0
0
1
0
0
0
1
Behavior Data 21 23
CONNECT hide section Communication
High Risk Highlighting
In the year view, monthly high risk values are set based on month totals. As in the month view, these totals’ defaults are set by C&C and then editable by a particular school. DIRECTED STORYTELLING HEURISTIC EVALUATION WIREFRAME TESTING FEATURE ANALYSIS
Trend Arrows
As in the month view, these arrows will show up or down and be color coded by red (bad) and green (good) based on the most recent month’s aggreation, compared to the one before it. This offers the mentor a way of assessing whether or not this month marked an improvement in a particular area. DIRECTED STORYTELLING HEURISTIC EVALUATION WIREFRAME TESTING FEATURE ANALYSIS
PROTOTYPE | 12
Add Data Pop-Up View 1 Time Feature
Giving mentors the capability to capture time spent with mentees during meetings, check-ins, or interventions is information that can be captured to quantify the effort of mentors and presented when applying for grants. This feature was directly requested by a mentor during testing. FEATURE ANALYSIS
Check & Connect
Adding general notes into the system connects case notes and data in a way that isn’t present on the paper system. During directed storytelling, mentors noted that the separation of the notes and data was difficult to keep track of and confusing with the amount of paperwork. WIREFRAME TESTING, DIRECTED STORYTELLING
Sarah Clarkson Mentor
Resources
Current View: Student Profile
< Back to Mentor
Michael Long Add Data: Michael Long hide profile
< Previous | Next >
X
Add Data
Change
Print Report
Thursday, February 28th, 2014 Change
< February >
Enter Table Edit Mode
Time Spent Day (minutes) M T Date
3
4
W
Th
F
M30
5
6
7
10
General Notes from Meeting Meeting Time Spent (minutes)
30
General Notes
Check Hide
Year
View:
T
W
Th
F
M
T
W
Th
F
M
T
W
Th
F
11
12
13
14
18
19
20
21
22
25
26
27
28
1
Very productive meeting today. After the rough patch in November, I think Michael is 25 really starting to get back on5 track.
Month
Trend
30
CHECK hide section
Academic DataAcademic Number of D’s or F’s
Number Number of Missing Assignments
General Notes
My Dashboard
1
2
3
↑
2
1
0
↓
1
2
2
1
↓
0
0
0
0
3
6
7
1
of D’s and4 F’s
Behavior Data Number of Missing Assignments Tardee Skipping Classes
Behavior
Unexcused/Unverified Absence Excused Absence
5
Number of Tardies0
3
0
1
Late 0 to class because of our 0 meeting.
↑
0
Behavioral referral/infraction
0
0
0
0
Detention
0
0
0
0
Suspension (in/out-of-school)
0
0
0
0
CONNECT hide section Communication
Specific Notes
Being able to connect a note to a specific data entry is an aspect of our prototype that was positively received during wireframe testing. This is an easy way to remind mentors of the context of their data entry. WIREFRAME TESTING
13 | PROTOTYPE
Add Data Pop-Up View 2 System Feedback
If a number is keyed in that meets the “high risk” requirement then the data entry automatically shows the system status by turning red. This automatic feedback is a visual reminder to mentors that the student meets high risk during the data entry. HEURISTIC EVALUATION
Check & Connect
My Dashboard
Sarah Clarkson Mentor
Resources
Current View: Student Profile
< Back to Mentor
Michael Long Add Data: Michael Long hide profile
< Previous | Next >
X
AddDoc Data Download as Word
Change
Print Report
Thursday, February 28th, 2014 Change
Enter Table Edit Mode
< February >
Year
View:
Check Hide T Day M
W
Th
F
M
T
W
Th
F
M
T
W
Th
F
M
T
W
Th
F
4
5
6
7
10
11
12
13
14
18
19
20
21
22
25
26
27
28
1
Date 3 Academic
Meeting
Month
Trend
25
5
30
1
1
2
3
↑
4
2
1
0
↓
2
2
1
↓
0
0
0
Number of D’s and30F’s Time Spent (minutes)
3
General Notes
Number of Missing Assignments
CHECK hide section
0
Academic Data Number of D’s or F’s
Hide Link
Clicking the “Hide” link will minimize the Check section of the data entry to open up more space in the popup instead of just having scrolling capabilities. While this link is not needed to make the system usable, it will make the data entry easier and more enjoyable for users.
Behavior
Number of Missing Assignments
Behavior Data Tardee Skipping Classes
Number of Tardies
1
Skipping Class
0
7
1 0
Unexcused/Unverified Absences
Late to class because of our meeting.
We discussed these absences at length. 3 6 helping his According to Michael, he was 0 0 parents with chores on Monday and missed most of his classes. 0 0
Unexcused/Unverified Absence
5
Excused Absence
0
7
Behavioral referral/infraction
0
Detention
0
0
0
0
Suspension (in/out-of-school)
0
0
0
0
↑
0 0
CONNECT hide section Communication
Change Links
Both “Change” links give the user the ability to change either the date of the data entry or the student that the information is being configured to. This improves flexibility of the system and gives mentors more control over data entry. HEURISTIC EVALUATION, WIREFRAME TESTING
PROTOTYPE | 14
Add Data Pop-Up View 3 Checking Data
When data only needs to be checked, a check mark will appear instead of an entry box to include a number value. This is automatically configured to fight confusion of filling out data and questioning whether a check or a number value is needed. HEURISTIC EVALUATION
Check & Connect
My Dashboard
Sarah Clarkson Mentor
Resources
Current View: Student Profile
< Back to Mentor
Michael Long Add Data: Michael Long hide profile
< Previous | Next >
X
AddDoc Data Download as Word
Change
Print Report
Thursday, February 28th, 2014 Change
Enter Table Edit Mode
Connect Hide Day M T W
3 4 5 BasicDate Intervention
Meeting
Time Spent (minutes)
30
General Notes
< February >
F
M
T
W
Th
F
M
T
W
Th
F
M
T
W
Th
F
6
7
10
11
12
13
14
18
19
20
21
22
25
26
27
28
1
Share “check” data
CHECK hide section Academic Data Provide regular feedback Number of D’s or F’s
1
Number of Missing Assignments
4
Behavior Data Tardee Skipping Classes
Excused Absence
Discuss staying in school
Problem solve about risk
1
Intensive Intervention 0
Unexcused/Unverified Absence
5
Facilitate goal setting 0
Year
View:
Th
Michael seemed interested in looking 25 reviewing his improvements,5 so we looked at his data. Focusedon missing assignments and tardees, both of which were down (improving) this week.
Month
Trend
30
1
2
3
↑
2
1
0
↓
2
2
1
↓
0
0
0
3
6
7
0
0
0
Behavioral referral/infraction
0
0
0
0
Detention
0
0
0
0
Suspension (in/out-of-school)
0
0
0
0
↑
CONNECT hide section Communication
Text Notes
Any note added to a specific data entry will show up in a way that makes it easy for the mentor to follow their notes and keep track of what has been completed. These notes will also show up later in the case notes section, but viewing the notes now during data entry is beneficial to the user. WIREFRAME TESTING, DIRECTED STORYTELLING
15 | PROTOTYPE
Case Notes Viewing Feature
Case notes can be organized by either all notes, a specific date range, or by a specific day. This ability to filter notes gives mentors the flexibility to view and follow their notes without having to refer back to a separate notebook. HEURISTIC EVALUATION, DIRECTED STORYTELLING, WIREFRAME TESTING
General Notes
General notes will be connected back to a day of data entry. These notes can cover the meeting with a student or details about an intervention. The ability to refer back to these notes will help mentors keep data entries in perspective. During wireframe testing, mentors said they appreciated being able to see their notes so easily. DIRECTED STORYTELLING, WIREFRAME TESTING
Check & Connect
My Dashboard
Resources
Sarah Clarkson Mentor
Current View: Student Profile
< Back to Mentor
< Previous | Next >
Michael Long Case Notes: Michael Long
AddDoc Data Download as Word
hide profile
Viewing: All Dates | Date Range | Day
< February >
Enter Table Edit Mode
Thursday, 28th, Day February M T W Th2014 F
X
Print Report
Sort: Most Recent at Top | Oldest atView: Top
Year
MViewTdataW
Th
F
General:Date Very productive rough12patch think20 Michael really25 starting 3 4 meeting 5 6today. 7 After 10 the11 13in November, 14 18 I19 21 is 22 26 to 27 get back on track.
28
1
Meeting
M
T
W
Th
F
M
T
Number of Tardies (1): Late to class because of our meeting.
W
Th
F
30 25 5 Time Spent (minutes) Unexcused/Unverified Absences (7 High Risk) ↑: We discussed these absences at length. According to Mi-
Trend
30
chael, was helping his parents with chores on Monday andmissed most of his classes. GeneralheNotes
CHECK hide section Thursday, February 21st, 2014 Academic Data
Month
View data
General: Spend some time discussing the absences. Quick five minute meeting. Number of D’s or F’s 1 1 Unexcused/Unverified Absences (6 - High Risk): Number of Missing Assignments 4 2
2
3
↑
1
0
↓
2
1
↓
0
0
Thursday, February 13th, 2014 View data Behavior Data General: Productive meeting today. January has been a pretty good month for grades, but he’s still struggling with attendance. He acknowledged his absences and we talked 1 2 about.
Tardee Skipping Classes
0
Wednesday, January 15th, 2014 5
0
View data 3 6 General: Spent most of the day talking about science grades. Missed some assignments that got him behind. 0 0 0 Magnitias nos dolor aut erum quid et omnihit aturianisiti desequidusam dolorem excepudant omnitestiis repBehavioral referral/infraction 0 reptatque volectate nem quisquiberum 0 accuptiur? Tatur adignis illorumqui 0 reseque labor resed qui consequi ut ex exerum volorro temporibus, volor mosti re consed moloreptatur sum que evel illitem quatibus sunt aut Detention 0 0 0 Unexcused/Unverified Absence
7
Excused Absence
0
Suspension (in/out-of-school)
0
0
0
↑
0 0 0
CONNECT hide section Communication
Specific Notes
As noted in the data entry pages, notes tied to a specific data entry give context to why the student had the experience they did. Tying a definition to a data point helps mentors to recall instead of having to recollect the background of the data point. WIREFRAME TESTING
High Risk Indicator
If a student was at high risk in a specific area when a note was made, red text will pop up in order to quickly and easily remind mentors why they made the note for the student. This direct showing of system status is immediate feedback to mentors. HEURISTIC EVALUATION, DIRECTED STORYTELLING
Download as Word Doc
This link allows mentors to easily transfer their notes into a Word Document to be printed at ease. If a coordinator asks about notes or if the mentor wants them on the go, this feature solves the mentors problem without wasting time or complicating the process. DIRECTED STORYTELLING
PROTOTYPE | 16
Single Case Note Overall
When a specific data point it clicked on a graph, the note connected with that data will pop up for the mentor to easily connect to the context. This feature was positively received during wireframe testing and mentors noted they really appreciated seeing their notes easily without having to dig through their separate case notes notebooks.
Check & Connect
My Dashboard
Sarah Clarkson Mentor
Resources
Current View: Student Profile
< Back to Mentor
Michael Long
< Previous | Next >
Add Data
hide profile
< February >
Enter Table Edit Mode
Print Report
Year
View:
Day
M
T
W
Th
F
M
T
W
Th
F
M
T
W
Th
F
M
T
W
Th
F
Date
3
4
5
6
7
10
11
12
13
14
18
19
20
21
22
25
26
27
28
1
Meeting
Time Spent (minutes)
30
Michael Long 25
General Notes
Thursday, February 28th, 2014
CHECK hide section
X
Trend
5
30
2
3
↑
1
0
↓
↓
Number of Tardies (1): Late to class because of our meeting.
Academic Data Number of D’s or F’s
1
1
Number of Missing Assignments
4
2
Tardee
1
2
2
1
Skipping Classes
0
0
0
0
Unexcused/Unverified Absence
5
3
6
7
Excused Absence
0
0
0
0
Behavioral referral/infraction
0
0
0
0
Detention
0
0
0
0
Suspension (in/out-of-school)
0
0
0
0
WIREFRAME TESTING
Month
View data
Behavior Data
↑
CONNECT hide section Communication
View Data
Having a link to view data allows mentors to immediately see the entire data entry from the day the note took place.
17 | PROTOTYPE
Configure Report Pop-Up Choose Items
Mentors can choose which data points will be included in the report, but only quantitative data can be compared. The need for a generated report is one of the most important aspects of the prototype because we found through all levels of testing that mentors need to see the results of their data input in order to feel like the data insertion was worthwhile HEURISTIC EVALUATION DIRECTED STORYTELLING WIREFRAME TESTING FEATURE ANALYSIS
Choose Time Frame
Giving mentors the capability to change the time frame gives them flexibility, something we found the paper version of Check & Connect lacks. This gives mentors the opportunity to view data in any window of time that they prefer. HEURISTIC ANALYSIS
Check & Connect
My Dashboard
Sarah Clarkson Mentor
Resources
Current View:
< Back to Mentor
< Previous | Next >
Michael Long Configure Report Enter Table Edit Mode
Choose Items to Day M T W Summarize Date
3
4
5
< February >
Add a Graph
Th
F
6
7
M
Meeting Unexcused/Unverified Absences
Time Spent (minutes)
Academic Data
30
Number of D’s or F’s Number of Missing Assignments
Behavior Data
Tardees
W
Th
F
M
T
W
Th
F
M
T
W
Th
F
13
14
18
19
20
21
22
25
26
27
28
1
1
4
Number of D’s or F’s
25
5
30
*
*
Th, 2/13 1
Choose Time Frame: 2
Th, 2/21
Month
Trend
* Tu, 2/4
Year
View:
Skipping Classes General Notes
CHECK hide section
T
February 10 11 12
X Print Report
Add Data
Th, 2/28
2
3
↑
1
0
↓
↓
Month Year
Number of Missing Assignments
Tardee
1
2
2
1*
Skipping Classes
0
0
0
0
1*
1
6*
7*
Excused Absence
0
0
0
0
Behavioral referral/infraction
0
0
0
0
Detention
0
0
0
0
Suspension (in/out-of-school)
0
0
0
0
↑
CONNECT hide section
Add a Graph
The add a graph feature is the simplest way to make data recognition visual. At most three data categories can be selected to best show comparison over time without confusion the viewer. WIREFRAME TESTING
PROTOTYPE | 18
Report Personal Information
Student Report
The inclusion of a student’s photo along with general notes and information about that student helps mentors to recall information rather than recollect it about the student featured on the report.
Date: 2/28/14 Mentor: Sarah Clarkson
Michael Long
15 years old 9th Grade McNeely Middle School 6 months in program General Notes
Michael enjoys playing basketball and eating pizza with his friends. He trasnfered last semester and is having a difficult time adjusting to his new routine.
WIREFRAME TESTING, HEURISTIC ANALYSIS
Unexcused/Unverified Absences
February
Number of D’s or F’s
Note Overall
We suggest focusing on testing the configure report and student report aspects of our prototype when continuing the process of developing an electronic version of the Check & Connect system. Receiving further feedback from users as to what they would want to see on the report is the next step to take.
Tardees
Tu, 2/4
Th, 2/13
Th, 2/21
Areas of Improvement Number of Missing Assignments (0)
February
↓ Tu, 2/4
Tardees (1)
HEURISTIC EVALUATION
Areas of Concern
The inclusion of areas of concern is important as well considering reports can be used to show students what they need to focus their attention on. WIREFRAME TESTING
Th, 2/21
Th, 2/28 February
Tu, 2/4
Th, 2/13
Th, 2/21
Th, 2/28
Areas of Concern 7 - High Risk)
Number of D’s or F’s (3)
Color coordinating the graph creates a visual that is easy to follow and decipher for any viewer. This feature is also most accessible for a student to understand if the mentor were to decide to show it to his or her mentee.
Th, 2/13
↓
↑
February
Tu, 2/4
Graph
Th, 2/28
Showing the Areas of Improvement section first is an emphasis every report should make on celebrating success. When communicating with mentors and coordinators through our testing process, we found that the most important aspect of Check & Connect to them is positive feedback and celebration of steps in the right direction. DIRECTED STORYTELLING, WIREFRAME TESTING
Th, 2/21
↑
Th, 2/28 February
Tu, 2/4
Areas of Improvement
Th, 2/13
Th, 2/13
High Risk
Including the immediate feedback of when students are at high risk within a data entry is an easy and immediate way to show mentors what areas are in need of emphasis. This direct visibility of system status was found to be needed through the Heuristic Evaluation HEURISTIC EVALUATION
Th, 2/21
Th, 2/28
Mini Graphs
Miniature graphs next to all data points also easily visualized what direction the data is heading to make the results of collection more recognizable without having the mentor do much work. Each data section selected to be on the report will have its own miniature graph.
Trending Arrows
Having trending arrows present shows mentors an immediate quick look at what direction the data is heading. Coloring negative trends in red is an easy way to understand that the student has an area of concern within that data section. HEURISTIC EVALUATION
19 | PROTOTYPE
Going Forward So much was learned from helping improve the check and connect system. We found that the student is the highest priority of Check & Connect mentors and coordinators. Therefore, we focused on improving the flexibility of the system to suit each employeeâ&#x20AC;&#x2122;s needs, while still improving the visibility of the data when it comes to using that information to aid students. We were able to tackle many issues which helped us make the system even better. However, there is so much more that could be done. These are the highlights of what we think would be the next steps in this redesigning process. Suggestions regarding the next steps in the redesigning process ++ How data is received and transferred: How do mentors and coordinators exchange data to use that information effectively? ++ How program is administered ++ Access and editing privileges: Between all of the C&C employees that would use this system, how could it be tiered so coordinators had editing privileges that mentors did not? What would each type of employee have access to? ++ Further testing of reporting capabilities ++ Further research into notification aspect: Notifications were planned to be administered to give employees advice on situations, most notably interventions with students. How this advice would be administered so that it doesnâ&#x20AC;&#x2122;t become a nuisance is still up in the air.
GOING FORWARD | 20
Appendix Mentor Persona Sarah Clarkson, Mentor
Age: 28 Case Load: Full Time, 40 students
Check & Connect Goals 1. Wants more time connect with her students and have more casual conversations with them 2. Puts a lot of focus on celebrating students accomplishments throughout the program to build their confidence 3. Strives to be that person for the students that they might not have at home to help them and make those connections so they can succeed and understand what they are capable of achieving
Favorites Loves her new French Bulldog Winnie, avid yoga goer, total movie buff, loves to travel, lives with her fiancé Jake, huge into music - lives by her record player
“The most valuable thing I get back from my job is seeing my mentees improve.”
Coordinator Persona Joan Erickson, Coordinator
Age: 43 Case Load: 20 C&C mentors
Check & Connect Goals 1. Wants to focus more time on helping mentors improve relationships with students instead of struggling to gather all the data from mentors 2. Would like to be able to provide mentors with more resources regarding situational issues and activities to do with students 3. Wants an easier way to keep track of mentors from all schools within her site
Favorites Hockey mom, has 2 boys and a girl, loves to scrapbook and craft, big into photography especially at kids sporting events, huge coffee connoisseur with a fabulous mug collection.
“If a few mentors don’t keep up with their data collection, it affects everything.”
i | APPENDIX
Student Persona Michael Long, Student
Age: 15 Case Load: 7 classes/day, 5 days/week
Check & Connect Goals 1. Really wants to graduate on time 2. Has a problem with late assignments, and wants to catch up on the assignments he’s missing 3. Sometimes feels like there is no one around who cares and as a result he loses his ambition and confidence in himself
Favorites He loves to play basketball with his friends after class and would love to be on the school’s basketball team but his current grades won’t allow it, he has a huge [basketball] shoe and jersey collection, lives with his mom and dad and three other siblings.
“Sometimes I feel like my mentor is the only one who really listens to me.”
Feature Desirability Analysis Data Feature Analysis Proposed Feature 1: Generate Report for Coordinator The ability to generate a summary report of student data that could be printed and shared with a coordinator. Answerer’s Role Mentor Coordinator
If present... Satisfied Satisfied
If absent... Dissatisfied Dissatisfied
Mentor/Coordinator Mentor/Coordinator Mentor/Coordinator
Satisfied Neutral Satisfied
Neutral Satisfied Dissatisfied
Comments It is very beneficial to see student data reports. This is absolutely essential or the process isn’t worth the effort.
This seems necessary to me; it’s a good synopsis to show the level of risk for purposes of support/advice for intervention assistance or to justify additional services or intervention.
APPENDIX | ii
Proposed Feature 2: Generate Report for Mentee The ability to generate a summary report of student data that could be printed and shared with a mentee (student) to review performance and progress. Answerer’s Role Mentor
If present... Satisfied
If absent... Dissatisfied
Coordinator
Satisfied
Dissatisfied
Mentor/Coordinator Mentor/Coordinator Mentor/Coordinator
Satisfied Neutral Satisfied
Neutral Satisfied Dissatisfied
Comments With our more visual student, this could be great! Based on my personal experience with them, they do like to see their progress or areas where they can make progress. Because this is a data-driven initiative and we want the students to take ownership of their own choices, we need to be able to have them look at their own trends--to address the needs and celebrate their successes (no matter how small)
This is something I already do with the school's data entry, but using C&C data would make it more clear and leave off some of the unnecessarily and sometimes confusing data provided by PowerSchool (the school's system). It also holds mentors accountable for keeping information current.
Proposed Feature 3: High Risk Flag See which students have been automatically flagged as high risk based on data entered. Answerer’s Role Mentor Coordinator Mentor/Coordinator Mentor/Coordinator Mentor/Coordinator
If present... Satisfied Satisfied Satisfied Satisfied Satisfied
If absent... Dissatisfied Neutral Neutral Satisfied Neutral
Comments
Proposed Feature 4: Progress Indicators by Data Type The ability to see up and down arrows on the same line as your data to show the most recent trend. Answerer’s Role Mentor Coordinator
If present... Satisfied Satisfied
If absent... Dissatisfied Dissatisfied
Mentor/Coordinator Mentor/Coordinator Mentor/Coordinator
Satisfied Neutral Neutral
Dissatisfied Neutral Neutral
iii | APPENDIX
Comments I want to explain my answer on the last one, but there was no comment box--the only time we would enter a student is after they've already been flagged as at-risk and referred to Check & Connect. Need to be able to disaggregate the data by type in order to more appropriately select interventions.
I think the current forms have more boxes for types of intervention than are necessary. Hard to tell if this question is asking about using the same exact forms vs. altering them or creating some new form altogether. I think the gridded month view is necessary, helpful, and shows changes over shortterm periods of time.
Proposed Feature 5: Spreadsheet View The ability to still enter and review C&C data in the same gridded month view used on the current paper forms provided by C&C. Answerer’s Role Mentor Coordinator
If present... Satisfied Neutral
If absent... Neutral Satisfied
Mentor/Coordinator Mentor/Coordinator Mentor/Coordinator
Satisfied Neutral Neutral
Dissatisfied Neutral Neutral
Comments As long as the data and reports are clear--it doesn't have to be in the same format as the paper forms--the flexibility to compare and contrast data on an individual student, across a building, and across the district are the most important--we aren't married to the paper format!
I don't really understand this question?
Proposed Feature 6: Intervention Suggestions See suggested interventions automatically, based on data entered. Answerer’s Role Mentor Coordinator
If present... Satisfied Satisfied
If absent... Dissatisfied Neutral
Mentor/Coordinator Mentor/Coordinator
Satisfied Satisfied
Dissatisfied Satisfied
Comments This would be extremely helpful!!! This is an awesome "nice to do" but the system would be fine without it. This would be EXTREMELY helpful.
Proposed Feature 7: Add Notes Have the option of adding notes to each academic/behavioral entry or intervention entry. Answerer’s Role Mentor Coordinator
If present... Satisfied Satisfied
If absent... Dissatisfied Dissatisfied
Mentor/Coordinator
Satisfied
Dissatisfied
Mentor/Coordinator Mentor/Coordinator
Satisfied Satisfied
Satisfied Dissatisfied
Comments Very important for an individualized system of interventions as each student will have a nuance to the entry. I would also like to add that it would be helpful to have a way to enter amount of time spent (minutes) with each students each time they meet with mentor either for a formal or informal Check. This is particularly important to us in our district as we are responsible for tracking ADSIS hours for program funding purposes. This would be much more helpful than a simple check mark in each box for each intervention. It would also be helpful if these minutes tracked could be complied in a report form. Thank you. Absolutely necessary. This holds mentors accountable to not just checking boxes... and it's helpful to look back and see exactly when this or that conversation occurred, what the results were, etc. APPENDIX | iv
Link to Interview with Brisa
http://www.youtube.com/watch?v=5DM5gNh7ZC8
v | APPENDIX
Wireframe Testing Script Mentor Scenarios
You recently met with one of your mentees, Michael, and want to fill in the data you retrieved after meeting with him. Find where you would enter in that he had 2 D’s and 1 F, 1 verified absence for this week and had 2 missing assignments. ++ ++ ++
From the mentor dashboard screen, find Michael in the list of students and click on the Add Data button. This brings you to the Check-in Data Entry page. If you can clicked the View Profile button next to Michael’s name, you would have seen his Student Profile page, where you could find the button at the top right of the screen to Add Data. You could also enter the data directly into the grid on Michael’s Profile page.
You received information from teachers late and as a result you now need to enter in Michael’s grade information from last Thursday. Find where you would enter in that data. ++ ++
From the mentor dashboard screen, find Michael in the list of students and click on the Add Data button. This brings you to the Check-in Data Entry page where you could change the date to last week and edit the entry. If you can clicked the View Profile button next to Michael’s name, you would have seen his Student Profile page, where you could enter the data directly into the grid on Michael’s Profile page.
You remember writing a note last Tuesday about why Michael was tardy to class but can’t remember what it said. Find where you could access this information from Tuesday, February 4. ++ ++
From Michael’s student profile page, you could click the Add Data button to open the Check-in page, change the date, and review any notes you appended to the entries from that date. From the student profile page, you could also click on the grid to review any notes attached to a particular entry.
Coordinator Scenarios
You are now logged in as a Coordinator who is reviewing both Mentors and their mentees. Later today you are going to be meeting with Sarah, a mentor at your site. You want to view a monthly summary of her total case load to see which students she has and the general trends of Sarah’s total case load. ++ From the Mentor Dashboard, find Sarah in your list of mentors and click the View Profile button. On Sarah’s profile click the Create Report button. You are reviewing data in preparation of renewing a school grant. Find how many high risk students Sarah has. ++ From the Mentor Dashboard, find Sarah in your list of mentors and click the View Profile button. On Sarah’s profile, click the High Risk sort option on the top right of the student list.
APPENDIX | vi