Usability Test Report

Usability Testing Results

JENN GREEN LATOYA HALL MAURA ROBINSON NICK SIRECI

Contents • Background • Testing • Goals • Process Overview • Participants • Tasks • Prototype • Participants • Findings & Results • Next Steps • Appendix

Curaspan  The Company • The leading provider in patient transition software  The Product • DischargeCentral® facilitates the process of moving a patient from one care facility to another • Provides end-to-end functionality for case workers • Consolidates information for better communication/transfer between case workers • Captures details that were not as transparent or accessible when case workers primary made contact with providers by phone

Challenges  Technology within hospitals • Wide range of adoption of technology (fax, paper) • Integration with existing systems (Meditech, Midas) • Case workers often have to return to their desk to access DischargeCentral® (rely on pagers)  Use case • Beds usually do not become available until day of • Patient’s needs (visiting nurse vs. post acute care) may not be fully understood until discharge • Target discharge date often changes due to medical necessity

Test Goals  Determine a fundamental baseline for usability • Where are users running into the most problems?  Rebuilding product on a brand new platform • What should change?

• What should stay the same?

• Data to validate that change isn't always bad for users  Data Points of Most Value: • Ease of Use • Learnability • Discoverability • Aspirational: Dream Product: If this was your dream product what would it do?

Testing Process Overview Type: Usability Test Duration: ~1 hour Interaction: Remote testing using UXC Lab & GoToMeeting Data Collection: • Quantitative (included 7-point Likert Scale) • Qualitative (e.g. natural language descriptions) • Clickable Prototype

Participants: Overall

12FemaleCASEWORKERS Key Factors: • Working on 25+ patient cases on a given day.

• Perform numerous context switches across patients in an environment that is busy and loud • Must be able to communicate across shifts (e.g. week and weekend staffs) • Deals with emotional cases where a balance between health and cost must be met

Participants: Existing vs. New

7

Participant Breakdown User Age Transition Software Experience

In Likert Results

In Pass/Fail Results

P1

49

Yes - Curaspan

No

No

P2

55

Yes - Curaspan

No

No

P3

35

No

No

Scenarios 1-4

P4

43

No

Yes

Yes

P5

57

Yes - Curaspan

No

No

P6

50

No

Yes

Yes

P7

61

Yes - Other

Yes

Yes

P8

53

Yes - Curaspan

Yes

Yes

P9

58

No

Yes

Yes

P10

33

No

Yes

Yes

P11

30

No

Scenarios 1-3

Scenarios 1-3

P12

30

No

Scenarios 1-3

Scenarios 1-3

Data was not included when its integrity was a concern. The following factors occurred: • WebEx failure • Moderator Guide/ Prototype mismatch found • No video recording (where audio not descriptive enough on its own) Results for specific tasks may have been 8 vs. removed when pass

Tasks Scenario Task 1

High Level Description

1.1 Pull patient from the census of patients (add to workbook) 2.1 Find the document in the system

2

2.2 Assign the document to the patient’s file 2.3 Create a referral packet 3.1 Identify providers that fit this set of needs

3

3.2 Send providers the referral package

4.1 Check status of the providers previously contacted 4

4.2 Send message to provider 5.1 Capture details (internal note)

5

5.2 Book the provider 9

Prototype Limitations

Benefits

• Prevented ability to see • Enabled remote full path of where testing participants would go • Forced talk aloud • Gave visual clues • Lack of other viable paths hindered ability to determine success or failure

Concerns • Not the best tool to serve as a baseline of existing system • Prohibited participants from being able to naturally explore the UI

• Dummy data caused frustration for existing users 10

Positive: Sending Referral

11

Positive: Find Provider

12

Other Positives  Learnability • “Very easy. Once you get in and learn it, its simple. Not something you are tripping over.”  Ease of communication with provider • Messaging used frequently, notes that stay out of the patient’s official record are appreciated (e.g. family dynamics)  Key Services • “To have all those [key services] already in there and to know if there are beds available or not...I think is really a large feat to do because you are all linked together.”  Better than the “old” way • Improved the process moving from fax to digital. Case workers commented on the improvement.

Observation: Participant  Existing users • Had greater difficulty adapting to the prototype than nonCuraspan users • Struggled to perform some tasks using new methods possibly due to similarity to their current version • May reflect unwillingness to change • Automatic mode - not able to see screen holistically (tunnel vision) • Confused reaction may reflect a screen disconnect between system snapshots were taken from and what is deployed to the sites

14

Likert Results Scenario

Usable

Neither A or D

Agree Somewhat

Agree

Strongly Agree

P6, P7, P9

P12

P11

P8, P10

P4, P7

P6, P9

P11

P8, P10, P12

P6, P11

P7, P9, P10

P8

P6, P9, P11, P12

P7

P8, P10

Interface

P4, P6, P8, P11

P3, P7, P9, P10, , P12

Content

P4, P6, P8, P9, P11

P3, P7, P10, P12

P4,

P10, P11

P7, P8, P9

Content

P6,

P4, P11

P7, P8, P9, P10

Interface

P4, P6

P9

P7, P8, P10

P4, P6, P9

P7, P8, P10 15

Interface

Strongly Disagree

Disagree

Curaspan Non-Curaspan Disagree Somewhat

P4

1 Content Interface

P4

2 Content

P4, P12

3

Interface

P6

4

5 Content

Reflection: Quantitative Scale  Likert incongruent with behavior observed • Many had an expectation that an adjustment period was needed to learn • Case workers and nurses may have more patience as a demographic - less likely to say something negative or rate something low • “Really easy if I do it the way I normally do it” –P2 Was not necessarily rating what they were always seeing.“ • “The issues I am having is my own unfamiliarity”-P5 Tendency to place blame on self rather than system

16

Task Results

17

Passes and Fails Per Task Passes and Fails Per Task: Actual Counts Curaspan Scenario

1

2

3

4

5

Task

Pass

Fail

Non Curaspan

All Participants

Pass

Pass

Fail

Fail

Total # Tested

Participant not included for specific task

1.1

2

0

4

3

6

3

9

2.1

2

0

0

7

2

7

9

2.2

1

1

5

2

6

3

9

2.3

2

0

0

7

2

7

9

3.1

2

0

5

2

7

2

9

3.2

2

0

7

0

9

0

9

4.1

2

0

3

3

5

3

8

P12

4.2

2

0

6

0

8

0

8

P12

5.1

0

2

0

5

0

7

7

P11, P12

5.2

1

1

2

1

3

2

5

P3, P9, P11, P12 18

Find Document

19

Assign Unassigned Document

20

Create Internal Note

21

Expert Review: Top 6 Findings Pill Navigation Small Text Patient Name Create Referral Packet (Save Patient Record) Save & Remove vs. Save & Keep Language

Findings of usability test support expert review results

22

Pill Navigation

23

Small Text

24

Create Referral Packet

25

Save: Remove vs. Keep

26

Language

27

Workflow Limitations  Navigation from one task to another is not always clear • New users left hunting as no visual guidance is supplied • Language not directly related to contents  Gets in the way of case workers getting job done • Not able to quickly find recently discharged patients • Referrals dismissed while there is still a risk that the patient plan might change  Messages used through several parts of process • Users required to switch between views for complementary actions.  Extraneous choices • Save: Keep vs. Remove leaves clutter • Multiple message buttons • Multiple search options that look the same but are not

Recommended

29

“Visiting nurse agencies or short term rehab want to know what the patient is going to need.

Would be nice to have things clicked off that indicate what is going to be needed for example skilled nursing, physical therapy. Then you can go click click click and then I might not need to send so many messages, clicks are just easier. Wouldn’t always need to put a message in.”

- P1

30

Request Criteria • Search currently filters criteria but information is not sent to provider (often part of messages).

• Would include: key services, dietary restrictions, transportation access, POS, level, discharge date, insurance, private rooms, insurance authorization • Likelihood patient is going home vs. rehab • Checkboxes where possible with an open text message area for other comments.

31

Messages & Rank • Messages should be decoupled as being a specific area in workflow as communication occurs within several phases. • Ability to input messages should be available for all use cases where communication is common. • Rank (1st, 2nd, 3rd) was a common theme. Providers want to know how to prioritize their work. Case workers are regularly communicating this via messages.

32

Standby Requests Coordinate$ Booking Book$Provider$

Other$providers$go$ into$standby (freezes$referral)$

• Case workers do not want to have a referral canceled outright as there is high risk that the discharge date will change and they will be left without a bed for their patient.

• This fear makes case workers less likely to book a request via the system when they can simply message the provider. This, however, increases the messages they need to respond too.

• “Standby” allows providers to know that the request is on hold but allows communication between facilities to pickup where it left off in case it is needed.

33

Recently Discharged • Communication with provider often continues after discharge to make sure patient is all set.

• Search does not behave the same way for recently discharged patients as it does with those in the hospital – makes it difficult to find the chart.

• Having recently discharged patients listed in the workbook would give easy access. This list would be separated from active patients with less visual focus. Automatic removal from recently discharged would help keep workbooks up to date.

34

Navigation Existing navigation leaves caseworker unaware of the next step and confused by the language.

Recommend a stateful navigation that provides progress feedback using familiar language. It would allow caseworkers the ability to pick up where they left off and only click when they wish to return to a previous step.

35

Pager Example “People writing back thank you. Getting alerted on pager all day letting her know that she has a message. Open up in Curaspan to find out it is only a thank you. Courtesy is nice….” – P7 Audio from P1:

P2 indicates that Curaspan can be setup to receive as many or few pages as desired.

Recommendation: Investigate further to verify use case is handled.

Ensure that users are aware of any configuration available.

36

37

38

Next Steps • Continue Information Architecture review (Bentley team had limited access to fully functioning environment and testing had a limited scope) • Run usability tests on proposed navigation workflow • Feedback surveys about UX prior to training/after training (sit in on training as a silent witness if possible) • Proceed with A/B test to see how both user groups respond to other designs.

• Competitive Analysis to focus on what will influence sales • Conduct benchmark with a live working demo environment • Contextual interviews to understand how application works under “hospital conditions”.

39

Appendix

Appendix: Secondary Feedback 1 User Issue/Comment

Recommendation

ER

P6

Preview for unassigned too small Add zoom to previewer

no

P10

Wasn’t sure how to preview document

yes

Refer to Expert Review Add text in preview screen when not populated

P5

“If I do save and remove and the facility hasn’t done all the things on their end it will disappear”

Make sure that the caseworker selections don’t prevent the provider from completing their work

no

P9

Confused about Test Wellness Center in prototype

See “Appendix: Image 1”

no

Question from Bentley: Why is the book referral button only visually associated with “Connected” providers and not “Unconnected” even though both can show in the?

41

Appendix: Secondary Feedback 2 User Issue/Comment

Recommendation

ER

P11

Unable to see “Return to workbook”…”the color blue and the white, you can’t really see.

The green one <tabs w/text> you can see more.”

Refer to Expert Review

yes

Unaware that she is on the workbook

Refer to Expert Review

P4

The ability to distinguish the color blue diminishes with age.

This makes reading small white text on top of blue difficult to read. yes

A text to the workbook page to let them know where they are.

P6 P9

Confused by label over “pill”

See “Appendix: Image 2”

no

P9

Asked if there is anything that indicates date in patient search.

Shows if beds are available now.

Consider enhancement that ties expected bed availability by date.

no 42

Appendix: Secondary Feedback 3 User Issue/Comment

Recommendation

ER

P11

Keep in mind in future UX research (e.g. user surveys) to determine if others find value.

no

Would like send receipt functionality currently in email for messages – to see if someone from the provider has read it. Even if they have not responded.

Bentley: Considered very low priority compared with other recommendations.

P8

Referring to internal note: “Once Design should indicate that I put a note in there how do I there is a note when there is see that I did that?” one.

no

P11

Felt pressing “sign and complete button for document was important for HIPPA purposes.

Evaluate accuracy of this statement.

no

P8

Like idea it flags estimated discharge date. Would tackle closer dates first

Verify discharge date is sortable in the workbook.

no 43

Appendix: Secondary Feedback 4 User Issue/Comment

Recommendation

ER

P1

“We had an infusion company that merged with Walgreens…so I had to refer to them.” Both company names were still in the system which was confusing. If the name changes they should get rid of the name as soon as possible. Bad for people who know but….

Risks wrong name given to patient or insurance company.

no

P8

Wants to be able to sort providers in Matching alphabetically

Verify that this is possible (prototype restricted ability to confirm).

P6

Thought that “Connected” provider is related to who she sent referrals to.

Keep in mind. Other participants no understood the terms connected and unconnected.

Evaluate the lag in the system between when the facility changes and when it is reflected in Curaspan. Provide functionality that allows the two names to be associated so both can be searched but act as one.

no

44

Appendix: Secondary Feedback 5 User Issue/Comment

Recommendation

P1

Assume that the patient’s no insurance is captured either in the product or a product it is integrated with. Allow search to filter based on insurance that a provider supports. Also maintain the ability to not have insurance factored into the search results.

“Some facilities are not contracted with different insurances.”

Most participants called insurance out as part of their workflows on steps to work with a patient in the discharge process.

P2

Projected discharge and actual dates sometimes differ.

“Sometimes you have to figure out and check off appropriate box.” Takes extra time as you have to go into Midas to figure out the reason. “In ER, the estimated date is really a guess.”

Evaluate the value in tracking the delta between actual and estimated dates – business perspective. If it is, consider adding an indicator about how certain the discharge date is.

ER

no

45

Appendix: Image #1

46

Appendix: Image #2

47

Tab Language: Acceptance User Participant response: what would you expect to find if you navigated there P1

<tangent - described how their Curaspan version differed>

P2

What facilities accepted which patients

P3

Your own personal list that you’ve already made

P4

Their admission work

P5

Not same as what they have. People accepted by a facility, home care agency, or vendor and then we would only fax the discharge paper if they were onsite.

P6

Patients that have been accepted to facility, but not ready to be discharged from the hospital.

P7

If facility accepted patients

P8

<never asked>

P9

No idea, maybe those that are present…”but verbiage doesn’t make sense to me”

P10

No idea

P11

the patient that will be accepted in your unit

P12

why the patient was admitted to the hospital, any facilities that accepted the patient 48

Tab Language: Discharge User Participant response: what would you expect to find if you navigated there P1

<tangent - described how their Curaspan version differed>

P2

The list of patients that were discharged

P3

Discharged would be patients being discharged

P4

for any discharge orders or documentation

P5

Those patients that are discharged

P6

Showing patients estimated discharge date.

P7

If they are gone

P8

<never asked>

P9

Those that have been discharged

P10

(bases off what she sees) So far looks like it looks like it lists all patients in one area, likes the est. discharge, currently doesnâ&#x20AC;&#x2122;t have this capability at her job

P11

the patient thatâ&#x20AC;&#x2122;s being scheduled to discharged

P12

display what the patient will need after the hospital, level of care, where their home is, where they 49 are in the hospital, when they were discharged. The patient levels.

Tab Language: Doc Mgr User Participant response: what would you expect to find if you navigated there P1

Documents, not sure

P2

A way to manage documents

P3

Would be where the referrals list would be

P4

not sure

P5

Documents that are outstanding and not assigned to a patient

P6

Doctors or specialists that are following a patient to the hospital or their primary care.

P7

No idea what doc manager is

P8

<never asked>

P9

How many patients are with each doctor

P10

Look up information provided by doctor information

P11

no idea

P12

patients listed by physician 50

Tab Language: Dispatch User Participant response: what would you expect to find if you navigated there P1

<tangent - described how their Curaspan version differed>

P2

where they’re going and how they’re getting there

P3

Not sure what this would be.

P4

notify other team members of departments

P5

No idea

P6

Patients that are going to out to different places for testing

P7

Doesn’t know

P8

<never asked>

P9

“in mind you are going to send them somewhere”

P10

not sure

P11

no idea

P12

unsure 51

Tab Language: Ordering User Participant response: what would you expect to find if you navigated there P1

<tangent - described how their Curaspan version differed>

P2

not sure

P3

Physician ordering

P4

orders

P5

Not sure

P6

new orders or current orders for patients

P7

What I have ordered for services

P8

<never asked>

P9

Maybe discharge orders

P10

Orders for the patient (Q. in hospital or post-acute?)

P11

ordering a johnny for a patient

P12

active orders for the patient 52

Tab Language: Reports User Participant response: what would you expect to find if you navigated there P1

<tangent - described how their Curaspan version differed>

P2

"Could be a Million different thingsâ&#x20AC;? 3:24; reports of how many patients went to a facility, grouped by insurance, location etc.

P3

Labs/radiology

P4

reports by position, consults

P5

attachments, lab work

P6

tests that have been done, imaging report, lab reports, etc.

P7

Reports sent to the facility/vendor

P8

<never asked>

P9

No idea...assume you went into a patient there would be reports inside the user. Unless this is more administrative

P10

Labs for the patient

P11

the patients reports

P12

any lab results or studies

53

Tab Language: Profile User Participant response: what would you expect to find if you navigated there P1

<tangent - described how their Curaspan version differed>

P2

“My Profile? I don’t know what Profile is”

P3

demographics of the patient

P4

the patients insurance, address all that kind of information

P5

My particular profile

P6

the profile of the patient - demographics, insurance info, etc.

P7

Not sure

P8

<never asked>

P9

Would think demographics for one patient but I don’t understand why that would be at the top

P10

patient demographics

P11

patient information

P12

patient demographics 54