Usability Testing Results
JENN GREEN LATOYA HALL MAURA ROBINSON NICK SIRECI
Contents • Background • Testing • Goals • Process Overview • Participants • Tasks • Prototype • Participants • Findings & Results • Next Steps • Appendix
Curaspan The Company • The leading provider in patient transition software The Product • DischargeCentral® facilitates the process of moving a patient from one care facility to another • Provides end-to-end functionality for case workers • Consolidates information for better communication/transfer between case workers • Captures details that were not as transparent or accessible when case workers primary made contact with providers by phone
Challenges Technology within hospitals • Wide range of adoption of technology (fax, paper) • Integration with existing systems (Meditech, Midas) • Case workers often have to return to their desk to access DischargeCentral® (rely on pagers) Use case • Beds usually do not become available until day of • Patient’s needs (visiting nurse vs. post acute care) may not be fully understood until discharge • Target discharge date often changes due to medical necessity
Test Goals Determine a fundamental baseline for usability • Where are users running into the most problems? Rebuilding product on a brand new platform • What should change? • What should stay the same? • Data to validate that change isn't always bad for users Data Points of Most Value: • Ease of Use • Learnability • Discoverability • Aspirational: Dream Product: If this was your dream product what would it do?
Testing Process Overview Type: Usability Test Duration: ~1 hour Interaction: Remote testing using UXC Lab & GoToMeeting Data Collection: • Quantitative (included 7-point Likert Scale) • Qualitative (e.g. natural language descriptions) • Clickable Prototype
Participants: Overall
12FemaleCASEWORKERS Key Factors: • Working on 25+ patient cases on a given day. • Perform numerous context switches across patients in an environment that is busy and loud • Must be able to communicate across shifts (e.g. week and weekend staffs) • Deals with emotional cases where a balance between health and cost must be met
Participants: Existing vs. New
7
Participant Breakdown User Age Transition Software Experience
In Likert Results
In Pass/Fail Results
P1
49
Yes - Curaspan
No
No
P2
55
Yes - Curaspan
No
No
P3
35
No
No
Scenarios 1-4
P4
43
No
Yes
Yes
P5
57
Yes - Curaspan
No
No
P6
50
No
Yes
Yes
P7
61
Yes - Other
Yes
Yes
P8
53
Yes - Curaspan
Yes
Yes
P9
58
No
Yes
Yes
P10
33
No
Yes
Yes
P11
30
No
Scenarios 1-3
Scenarios 1-3
P12
30
No
Scenarios 1-3
Scenarios 1-3
Data was not included when its integrity was a concern. The following factors occurred: • WebEx failure • Moderator Guide/ Prototype mismatch found • No video recording (where audio not descriptive enough on its own) Results for specific tasks may have been 8 vs. removed when pass
Tasks Scenario Task 1
High Level Description
1.1 Pull patient from the census of patients (add to workbook) 2.1 Find the document in the system
2
2.2 Assign the document to the patient’s file 2.3 Create a referral packet 3.1 Identify providers that fit this set of needs
3
3.2 Send providers the referral package
4.1 Check status of the providers previously contacted 4
4.2 Send message to provider 5.1 Capture details (internal note)
5
5.2 Book the provider 9
Prototype Limitations
Benefits
• Prevented ability to see • Enabled remote full path of where testing participants would go • Forced talk aloud • Gave visual clues • Lack of other viable paths hindered ability to determine success or failure
Concerns • Not the best tool to serve as a baseline of existing system • Prohibited participants from being able to naturally explore the UI
• Dummy data caused frustration for existing users 10
Positive: Sending Referral
11
Positive: Find Provider
12
Other Positives Learnability • “Very easy. Once you get in and learn it, its simple. Not something you are tripping over.” Ease of communication with provider • Messaging used frequently, notes that stay out of the patient’s official record are appreciated (e.g. family dynamics) Key Services • “To have all those [key services] already in there and to know if there are beds available or not...I think is really a large feat to do because you are all linked together.” Better than the “old” way • Improved the process moving from fax to digital. Case workers commented on the improvement.
Observation: Participant Existing users • Had greater difficulty adapting to the prototype than nonCuraspan users • Struggled to perform some tasks using new methods possibly due to similarity to their current version • May reflect unwillingness to change • Automatic mode - not able to see screen holistically (tunnel vision) • Confused reaction may reflect a screen disconnect between system snapshots were taken from and what is deployed to the sites
14
Likert Results Scenario
Usable
Neither A or D
Agree Somewhat
Agree
Strongly Agree
P6, P7, P9
P12
P11
P8, P10
P4, P7
P6, P9
P11
P8, P10, P12
P6, P11
P7, P9, P10
P8
P6, P9, P11, P12
P7
P8, P10
Interface
P4, P6, P8, P11
P3, P7, P9, P10, , P12
Content
P4, P6, P8, P9, P11
P3, P7, P10, P12
P4,
P10, P11
P7, P8, P9
Content
P6,
P4, P11
P7, P8, P9, P10
Interface
P4, P6
P9
P7, P8, P10
P4, P6, P9
P7, P8, P10 15
Interface
Strongly Disagree
Disagree
Curaspan Non-Curaspan Disagree Somewhat
P4
1 Content Interface
P4
2 Content
P4, P12
3
Interface
P6
4
5 Content
Reflection: Quantitative Scale Likert incongruent with behavior observed • Many had an expectation that an adjustment period was needed to learn • Case workers and nurses may have more patience as a demographic - less likely to say something negative or rate something low • “Really easy if I do it the way I normally do it” –P2 Was not necessarily rating what they were always seeing.“ • “The issues I am having is my own unfamiliarity”-P5 Tendency to place blame on self rather than system
16
Task Results
17
Passes and Fails Per Task Passes and Fails Per Task: Actual Counts Curaspan Scenario
1
2
3
4
5
Task
Pass
Fail
Non Curaspan
All Participants
Pass
Pass
Fail
Fail
Total # Tested
Participant not included for specific task
1.1
2
0
4
3
6
3
9
2.1
2
0
0
7
2
7
9
2.2
1
1
5
2
6
3
9
2.3
2
0
0
7
2
7
9
3.1
2
0
5
2
7
2
9
3.2
2
0
7
0
9
0
9
4.1
2
0
3
3
5
3
8
P12
4.2
2
0
6
0
8
0
8
P12
5.1
0
2
0
5
0
7
7
P11, P12
5.2
1
1
2
1
3
2
5
P3, P9, P11, P12 18
Find Document
19
Assign Unassigned Document
20
Create Internal Note
21
Expert Review: Top 6 Findings Pill Navigation Small Text Patient Name Create Referral Packet (Save Patient Record) Save & Remove vs. Save & Keep Language
Findings of usability test support expert review results
22
Pill Navigation
23
Small Text
24
Create Referral Packet
25
Save: Remove vs. Keep
26
Language
27
Workflow Limitations Navigation from one task to another is not always clear • New users left hunting as no visual guidance is supplied • Language not directly related to contents Gets in the way of case workers getting job done • Not able to quickly find recently discharged patients • Referrals dismissed while there is still a risk that the patient plan might change Messages used through several parts of process • Users required to switch between views for complementary actions. Extraneous choices • Save: Keep vs. Remove leaves clutter • Multiple message buttons • Multiple search options that look the same but are not
Recommended
29
“Visiting nurse agencies or short term rehab want to know what the patient is going to need. Would be nice to have things clicked off that indicate what is going to be needed for example skilled nursing, physical therapy. Then you can go click click click and then I might not need to send so many messages, clicks are just easier. Wouldn’t always need to put a message in.” - P1
30
Request Criteria • Search currently filters criteria but information is not sent to provider (often part of messages). • Would include: key services, dietary restrictions, transportation access, POS, level, discharge date, insurance, private rooms, insurance authorization • Likelihood patient is going home vs. rehab • Checkboxes where possible with an open text message area for other comments.
31
Messages & Rank • Messages should be decoupled as being a specific area in workflow as communication occurs within several phases. • Ability to input messages should be available for all use cases where communication is common. • Rank (1st, 2nd, 3rd) was a common theme. Providers want to know how to prioritize their work. Case workers are regularly communicating this via messages.
32
Standby Requests Coordinate$ Booking Book$Provider$
Other$providers$go$ into$standby (freezes$referral)$
• Case workers do not want to have a referral canceled outright as there is high risk that the discharge date will change and they will be left without a bed for their patient. • This fear makes case workers less likely to book a request via the system when they can simply message the provider. This, however, increases the messages they need to respond too. • “Standby” allows providers to know that the request is on hold but allows communication between facilities to pickup where it left off in case it is needed. 33
Recently Discharged • Communication with provider often continues after discharge to make sure patient is all set. • Search does not behave the same way for recently discharged patients as it does with those in the hospital – makes it difficult to find the chart. • Having recently discharged patients listed in the workbook would give easy access. This list would be separated from active patients with less visual focus. Automatic removal from recently discharged would help keep workbooks up to date. 34
Navigation Existing navigation leaves caseworker unaware of the next step and confused by the language.
Recommend a stateful navigation that provides progress feedback using familiar language. It would allow caseworkers the ability to pick up where they left off and only click when they wish to return to a previous step.
35
Pager Example “People writing back thank you. Getting alerted on pager all day letting her know that she has a message. Open up in Curaspan to find out it is only a thank you. Courtesy is nice….” – P7 Audio from P1:
P2 indicates that Curaspan can be setup to receive as many or few pages as desired. Recommendation: Investigate further to verify use case is handled. Ensure that users are aware of any configuration available.
36
37
38
Next Steps • Continue Information Architecture review (Bentley team had limited access to fully functioning environment and testing had a limited scope) • Run usability tests on proposed navigation workflow • Feedback surveys about UX prior to training/after training (sit in on training as a silent witness if possible) • Proceed with A/B test to see how both user groups respond to other designs. • Competitive Analysis to focus on what will influence sales • Conduct benchmark with a live working demo environment • Contextual interviews to understand how application works under “hospital conditions”.
39
Appendix
Appendix: Secondary Feedback 1 User Issue/Comment
Recommendation
ER
P6
Preview for unassigned too small Add zoom to previewer
no
P10
Wasn’t sure how to preview document
yes
Refer to Expert Review Add text in preview screen when not populated
P5
“If I do save and remove and the facility hasn’t done all the things on their end it will disappear”
Make sure that the caseworker selections don’t prevent the provider from completing their work
no
P9
Confused about Test Wellness Center in prototype
See “Appendix: Image 1”
no
Question from Bentley: Why is the book referral button only visually associated with “Connected” providers and not “Unconnected” even though both can show in the?
41
Appendix: Secondary Feedback 2 User Issue/Comment
Recommendation
ER
P11
Unable to see “Return to workbook”…”the color blue and the white, you can’t really see. The green one <tabs w/text> you can see more.”
Refer to Expert Review
yes
Unaware that she is on the workbook
Refer to Expert Review
P4
The ability to distinguish the color blue diminishes with age. This makes reading small white text on top of blue difficult to read. yes
A text to the workbook page to let them know where they are. P6 P9
Confused by label over “pill”
See “Appendix: Image 2”
no
P9
Asked if there is anything that indicates date in patient search. Shows if beds are available now.
Consider enhancement that ties expected bed availability by date.
no 42
Appendix: Secondary Feedback 3 User Issue/Comment
Recommendation
ER
P11
Keep in mind in future UX research (e.g. user surveys) to determine if others find value.
no
Would like send receipt functionality currently in email for messages – to see if someone from the provider has read it. Even if they have not responded.
Bentley: Considered very low priority compared with other recommendations.
P8
Referring to internal note: “Once Design should indicate that I put a note in there how do I there is a note when there is see that I did that?” one.
no
P11
Felt pressing “sign and complete button for document was important for HIPPA purposes.
Evaluate accuracy of this statement.
no
P8
Like idea it flags estimated discharge date. Would tackle closer dates first
Verify discharge date is sortable in the workbook.
no 43
Appendix: Secondary Feedback 4 User Issue/Comment
Recommendation
ER
P1
“We had an infusion company that merged with Walgreens…so I had to refer to them.” Both company names were still in the system which was confusing. If the name changes they should get rid of the name as soon as possible. Bad for people who know but….
Risks wrong name given to patient or insurance company.
no
P8
Wants to be able to sort providers in Matching alphabetically
Verify that this is possible (prototype restricted ability to confirm).
P6
Thought that “Connected” provider is related to who she sent referrals to.
Keep in mind. Other participants no understood the terms connected and unconnected.
Evaluate the lag in the system between when the facility changes and when it is reflected in Curaspan. Provide functionality that allows the two names to be associated so both can be searched but act as one.
no
44
Appendix: Secondary Feedback 5 User Issue/Comment
Recommendation
P1
Assume that the patient’s no insurance is captured either in the product or a product it is integrated with. Allow search to filter based on insurance that a provider supports. Also maintain the ability to not have insurance factored into the search results.
“Some facilities are not contracted with different insurances.” Most participants called insurance out as part of their workflows on steps to work with a patient in the discharge process.
P2
Projected discharge and actual dates sometimes differ. “Sometimes you have to figure out and check off appropriate box.” Takes extra time as you have to go into Midas to figure out the reason. “In ER, the estimated date is really a guess.”
Evaluate the value in tracking the delta between actual and estimated dates – business perspective. If it is, consider adding an indicator about how certain the discharge date is.
ER
no
45
Appendix: Image #1
46
Appendix: Image #2
47
Tab Language: Acceptance User Participant response: what would you expect to find if you navigated there P1
<tangent - described how their Curaspan version differed>
P2
What facilities accepted which patients
P3
Your own personal list that you’ve already made
P4
Their admission work
P5
Not same as what they have. People accepted by a facility, home care agency, or vendor and then we would only fax the discharge paper if they were onsite.
P6
Patients that have been accepted to facility, but not ready to be discharged from the hospital.
P7
If facility accepted patients
P8
<never asked>
P9
No idea, maybe those that are present…”but verbiage doesn’t make sense to me”
P10
No idea
P11
the patient that will be accepted in your unit
P12
why the patient was admitted to the hospital, any facilities that accepted the patient 48
Tab Language: Discharge User Participant response: what would you expect to find if you navigated there P1
<tangent - described how their Curaspan version differed>
P2
The list of patients that were discharged
P3
Discharged would be patients being discharged
P4
for any discharge orders or documentation
P5
Those patients that are discharged
P6
Showing patients estimated discharge date.
P7
If they are gone
P8
<never asked>
P9
Those that have been discharged
P10
(bases off what she sees) So far looks like it looks like it lists all patients in one area, likes the est. discharge, currently doesnâ&#x20AC;&#x2122;t have this capability at her job
P11
the patient thatâ&#x20AC;&#x2122;s being scheduled to discharged
P12
display what the patient will need after the hospital, level of care, where their home is, where they 49 are in the hospital, when they were discharged. The patient levels.
Tab Language: Doc Mgr User Participant response: what would you expect to find if you navigated there P1
Documents, not sure
P2
A way to manage documents
P3
Would be where the referrals list would be
P4
not sure
P5
Documents that are outstanding and not assigned to a patient
P6
Doctors or specialists that are following a patient to the hospital or their primary care.
P7
No idea what doc manager is
P8
<never asked>
P9
How many patients are with each doctor
P10
Look up information provided by doctor information
P11
no idea
P12
patients listed by physician 50
Tab Language: Dispatch User Participant response: what would you expect to find if you navigated there P1
<tangent - described how their Curaspan version differed>
P2
where they’re going and how they’re getting there
P3
Not sure what this would be.
P4
notify other team members of departments
P5
No idea
P6
Patients that are going to out to different places for testing
P7
Doesn’t know
P8
<never asked>
P9
“in mind you are going to send them somewhere”
P10
not sure
P11
no idea
P12
unsure 51
Tab Language: Ordering User Participant response: what would you expect to find if you navigated there P1
<tangent - described how their Curaspan version differed>
P2
not sure
P3
Physician ordering
P4
orders
P5
Not sure
P6
new orders or current orders for patients
P7
What I have ordered for services
P8
<never asked>
P9
Maybe discharge orders
P10
Orders for the patient (Q. in hospital or post-acute?)
P11
ordering a johnny for a patient
P12
active orders for the patient 52
Tab Language: Reports User Participant response: what would you expect to find if you navigated there P1
<tangent - described how their Curaspan version differed>
P2
"Could be a Million different thingsâ&#x20AC;? 3:24; reports of how many patients went to a facility, grouped by insurance, location etc.
P3
Labs/radiology
P4
reports by position, consults
P5
attachments, lab work
P6
tests that have been done, imaging report, lab reports, etc.
P7
Reports sent to the facility/vendor
P8
<never asked>
P9
No idea...assume you went into a patient there would be reports inside the user. Unless this is more administrative
P10
Labs for the patient
P11
the patients reports
P12
any lab results or studies
53
Tab Language: Profile User Participant response: what would you expect to find if you navigated there P1
<tangent - described how their Curaspan version differed>
P2
“My Profile? I don’t know what Profile is”
P3
demographics of the patient
P4
the patients insurance, address all that kind of information
P5
My particular profile
P6
the profile of the patient - demographics, insurance info, etc.
P7
Not sure
P8
<never asked>
P9
Would think demographics for one patient but I don’t understand why that would be at the top
P10
patient demographics
P11
patient information
P12
patient demographics 54