TCO Module Usability
Evaluation UED / Sept. 2015
01.
Scope & Key User Goals
02.
Usability Study Protocols Deployed
a. Information Architecture Study b. Usability Evaluation - (3 professionals) c. System Usability Evaluation - SUS scale (min. 10 users) d. Task Flow (15 users) e. Subjective Interviews
03.
Key Findings / Conclusion
Contents
01. Scope & User Goals To test the usability of the TCO (Total cost of ownership) tool on the XID page which aims to help users understand various expenses incurred during a property purchase. It provides cost break-up under major heads giving an indicative figure of total costs incurred to own a selected property & configuration.
What we tested: <http://www.99acres.com/aarcity-moontowers-greater-noida-west-npxidr80042?src=NPSRP >
02. Usability Study Protocols A. Information Architecture Study
B. Usability Evaluation C. User EvaluationTesting Task Flows with users
D. System Usability EvaluationSUS â&#x20AC;&#x201C; System Usability Scale
navigation Aâ&#x20AC;&#x2122; navigation A
navigation Aâ&#x20AC;&#x2122;
2nd fold
1st fold
navigation A
1
TCO
12 folds below
4th fold
3rd fold
4
5
A. Information Architecture Study It is found that the hierarchy is not clear. Users take sometime to find the course of action & sequence. It is necessary to relook at IA and simplify the same.
2
3
B. Usability Evaluation Usability of the TCO tool was evaluated and analysed on the basis of heuristic principles. No. Of Evaluators: 3 Rating done on a 5 step severity scale
Heuristics
Evaluator
1. Visibility of system status
a
Default selection is not understood by users
This part is not noticed by users
Not an issue
Cosmetic issue
Major
Critical
Fatal
Evaluation
System status is not clear - when there is a change in the indicative price range, or - when the options in the Area dropdown change - when a different BHK setting is selected
b
The default selections is not clear and its output is not obvious.
c
Once user lands on this page its not clear what is selected and on what bases the values are shown
Visibility of System Status Whether system is keeping users informed about what is going on
Results: The visibility of ongoing system status and change in state is poor. This issue is high on severity as it directly effects completion of user goals.
Heuristics
Evaluator
2. Match between system and the real world
Technical terms, users are not familiar
Graphical representation & legend are difficult to relate and understand
a
Not an issue
Cosmetic issue
Major
Critical
Fatal
Evaluation
The verbiage is technical/ uncommon - it may be difficult to understand for a common user. Also, there is no support text / tip, in case there are doubts at the userâ&#x20AC;&#x2122;s end.
b
The UI is not intuitive, content is in technical language.
c
Terms used in the tool are not clear to the users hence difficult to understand
Match between System & Real world Whether system speaks the users' language and concepts familiar to the user, rather than systemoriented terms
Results: There is an evident gap between the system, real world usage and metaphors. This issue is average severity as it impacts the users understanding of the actions available.
Heuristics
Evaluator
3. User control and freedom
Confusion between the sequence of the two dropdowns. Legend doesnâ&#x20AC;&#x2122;t mark the compulsory cost and additional cost
Not an issue
Cosmetic issue
Major
Critical
Fatal
Evaluation
a
Though toggles are available, there is no clear indication that the same is impacting the results on the right panel
b
Hierarchy is missing, which it makes it difficult to take the decision of selections
c
Left panel filters and right side graph do not clearly show the connection, so it doesnâ&#x20AC;&#x2122;t give any clarity for user about how to use and change the preferences as per their needs
User control and Freedom
Results : This is an average severity issue. The system has incorporated toggle to give control and freedom to users, but inappropriate hierarchy and visual lead are causing issues.
Heuristics
Evaluator
4. Consistency and standards
Not an issue
Cosmetic issue
Major
Critical
Fatal
Evaluation
a
The GUI elements used are not consistent, selected states are different for multiple elements.
b
The tool lacks consistent UI elements.
c
Selection states are not clear. Though the graphic is interactive, on-click events doesnâ&#x20AC;&#x2122;t show any info.
Consistency & Standards
. Dropdowns Buttons (supposed to be toggles) Radio buttons
Inconsistent UI elements confuse users
Results : This is an average severity issue. There is lack of consistency in the visual feedback of actions.
Heuristics
Evaluator
5. Recognition rather than recall
The selection is not obvious to novice user
First time user is unable to establish the connection between input and result. The animation may help user to recall it the next time he lands.
a
Not an issue
Cosmetic issue
Major
Critical
Fatal
Evaluation
The purpose of some of the actions is not appropriately marked / visible
b
Visibility of some information is doubtful. One would have to recall the interactivity of the tool when he arrives next time.
c
UI elements are not intuitive.
Recognition Rather than Recall User should not have to remember information from one part of the dialogue to another
Results : The is an average severity issue. Since the system uses technical terms to control and depict results, Instructions are missing for ease of use.
Heuristics 6. Flexibility and efficiency of use
No hierarchy defined for the interactions. Several options displayed at a time makes the tool less efficient
Evaluator
a
Not an issue
Cosmetic issue
Major
Critical
Fatal
Evaluation
The module is not built for repeated use; changing preferences is not “very simple” & “delightful” each time.
b
Too many elements and information displayed at a time reduces its flexibility and efficiency of use
c
Graphical elements are not efficient for use
Flexibility & Efficiency of Use Whether system caters to both inexperienced and experienced users
Results : This is an average severity issue.
Heuristics
Evaluator
7. Aesthetic and minimalist design
Distributed pieces of information displayed without clear hierarchy at a time hampers the visibility of most relevant information â&#x20AC;&#x153;At a first glance, mujhe bas donut dikh raha hai and Iâ&#x20AC;&#x2122;m trying to understand what the donut is aboutâ&#x20AC;?
Not an issue
Cosmetic issue
Major
Critical
Fatal
Evaluation
a
Lack of visual clues and hierarchy
b
Extra Information units displayed at a time is competing with the relevant information and hence the focus is lost
c
Too many elements used at a time which are from different families. Also too many actions displayed at a time confuses user. Hierarchy is missing.
Aesthetic and Minimalist Design
Results : A high severity issue, the system does not incorporate adequate visual clues and hierarchy in order to aid smooth completion of tasks.
Heuristics
Evaluator
8. Help users recognize, diagnose, and recover from errors
No proper feedback of the change in selection in the graphical information Users are reluctant to read such a large disclaimer. “Disclaimer hai matlab ye costing sahi nahi hai, isme additional ya hidden cost ayega onsite”
a
Not an issue
Cosmetic issue
Major
Critical
Fatal
Evaluation
Lack of proper feedback on user action
b
There is a lack of feedback after preferences selection. It confuses users.
c
Once the user lands on this tool, it doesn’t guide user about where to start from.
Diagnosis & Error Recovery Whether error messages are expressed in plain language, indicating precise problem Whether system gives feedback after committing error and constructively suggests a solution
Results : Low Severity Issue.
Heuristics
Evaluator
9. Help and documentati on
“How do I see information about these plans?” “What is PLC?” “What do these terms mean? “floor PLC, IFMS? I think I will need to gain knowledge first before coming to the website”
Not an issue
Cosmetic issue
Major
Critical
Fatal
Evaluation
a
Since the tool is not very common, system should have apt support in the form of quick help, the same is missing.
b
There is no help text provided in the tool. The tool contains technical terms for which a user would have to gain knowledge first.
c
There is no step wise guidance to the user also how efficiently this tool can be used and be of help to user is not conveyed.
Help & Documentation
Results : High severity Issue. There is no proper indication of how the tool can equip the user with the calculation, there is a lack of info tips and guidance.
15
34
Total no. of users
Average age
Real users
5
(Active home seekers for buying. 1/5 use cases has recently bought a property )
Other users
10
(Our Employees/ colleagues)
Male Female
7 8
User Demographics
Strongly Disagree
I think that I would like to use this system frequently (when itâ&#x20AC;&#x2122;s a real use case)
Strongly Agree
1
2
3
4
5
1
2
3
4
5
1
2
3
4
5
I think that I would need the support of a technical person to be able to use this system
1
2
3
4
5
I found the various functions to this system were well integrated
1
2
3
4
5
I thought there was too much inconsistency in this system
1
2
3
4
5
1
2
3
4
5
1
2
3
4
5
1
2
3
4
5
1
2
3
4
5
I found the system unnecessarily complex
I thought the system was easy to use
I would imagine that most people would learn to use this system very quickly I found the system very cumbersome to use
I felt very confident using the system
I needed to learn a lot of things before I could get going with this system
C. User System Usability Scale The System Usability Scale (SUS) is a simple, ten-item scale giving a global view of subjective assessments of usability. No. Of Users : 10
Score >
C. User - System Usability Scale HIGH
100 90
GOOD
80
70
Usability Results
AVERAGE
60 50
BELOW AVERAGE
40 30
LOW
20 10 1
2
3
4
5
6
7
8
9
10
Users >
6/10 users the TCO tool ‘Below in usability.
rated
average’
No.
1
Tasks While buying a property, ‘total cost of ownership’ varies from the ‘base price’ of a property. Can you find out the total cost of ownership of different apartments in the given project/society.
Success Criteria
• •
•
2
Explain what does the central graphical representation (donut and legend) indicate
3
Figure out the cost estimate of 3 BHK apartment of 1025 sqft area?
4
Find out the cost estimate of 3 BHK apartment of 1025 sqft area located on 1st floor with additional power back up?
•
• •
• •
User reaches the cost estimate section by scrolling Use clicks on ‘Click to view price list’ link on page
User understands what is the given price is for User understands the connection between top dropdowns and donut graphic without interacting
User notices top dropdowns of BHK and Area User uses the dropdowns and is able to tell the cost estimate User notices left panel of additional ownership charges, User is able to interact with the left panel and able to establish connection between input options and output cost
D. User - Task Flows The user’s were asked to perform 4 prime tasks using the TCO tool. The success criteria’s for the same were defined and the success rate is indicative of the tools usability performance.
No. of users : 15
66%
33%
Success Rate
Success Rate
TASK 1: You are interested to buy an apartment in Aarcity towers, Noida. Find out the price list for the ownership of the apartments in the same towers
D. User - Task Flows
TASK 2: Explain what does the central graphical representation (donut) and number indicate
Success Failure
86%
80%
Success Rate
Success Rate
TASK 3: Figure out the cost estimate of 3 BHK apartment of 1025 sqft area?
TASK 4: Find out the cost estimate of 3 BHK apartment of 1025 sqft area located on 1st floor with additional power back up?
Estimated time for all task completion: 15 min Average completion time by users: 8 min
The link interpreted as - it will open a new tab. Also link title “price list” is misleading – 6/15 users assumed to view a list, but when presented with an interactive tool it is confusing Title of the tool doesn’t convey what this tool does exactly - 4/15 And it is not consistent with the link titles above.
Launch Xid
D. User - Task Flows Link – “Click to view price list”
‘Price list tab’
Overview of configurations and Prices
Pain Points Pain points
Cost estimates and Price list Blind spot. Not noticed by 11/15 users. “Why there is a range?” Its not communicated
The graphical representation is difficult to figure out. The connection to legend is not obvious, or click on it every time. 12/15
Select ‘BHK’
Select ‘Configuration’ Read indicative price range
View expected cost and its break up Unnoticed by users 10/15 users. Also users were reluctant to download when asked. The text “For terms and conditions, download official pricelist” is misleading – Here the purpose of pdf is not clear, whether it is price list or terms & conditions document.
4/15 users were stuck at overview only and assumed that is the pricelist. Did not discover TCO or there is a detailed pricelist unless probed.
Change preference 1 Change preference 2
Download PDF
Change preference 3 Change preference 4
End
The default selected options are not intuitive (8/15 didn’t get default selection) The preference options are not easily noticed
1. Understanding of the term â&#x20AC;&#x2DC;Indicative price rangeâ&#x20AC;&#x2122;
Understood 6
E. Subjective interviews
Did not understand
9
2. Finding the exact price of apartment by this tool?
Questions were asked while users were interacting with the tool. Their understanding of certain terms and actions is studied. No. of users : 15
4
Don't know
11
Maybe by changing filters
3. Noticing the â&#x20AC;&#x2DC;Downloadable pdfâ&#x20AC;&#x2122;
7
8
Noticed Did not notice
4. Willingness to download the pdf 2 4
Would download to know more detailed information Would download only if really interested in to buy
4
No, its going to be an exhaustive brochure 5
No, this much information is enough
E. Subjective interviews
5. Understanding of â&#x20AC;&#x2DC;Official price list documentâ&#x20AC;&#x2122;
Don't know/ Not sure 4 5 3 3
Who is providing this list? Builder, developer or 99? Maybe the list is provided by builder Same information but in pdf list format
E. Subjective interviews
67% Of the users failed to connect the donut with legend
90% Of the users were unable to understand the logic for indicative price range
Key Concerns & Conclusion The system does not incorporate adequate visual clues and hierarchy in order to aid smooth completion of tasks
60% Of the users rated the system usability as below average
Mismatch Between System and Usersâ&#x20AC;&#x2122; understanding / verbiage, Lack of Help / Tips.
The system is not very flexible to use, it restricts the user while exploring the options in tool UI elements are not consistent and intuitive
It is evident from the study that clear hierarchy & action sequence needs to be established. Help text should be provided for users as they did not get the meaning of certain technical terms and sought help.
UI elements should be in harmony with familiarity & popular mental models (viz., selectors for payment plan/floor… are perceived as ‘action buttons’) Contextual tips for selectors/filters on hover would make the interface more efficient. Graphical representation of cost breakup should have supporting text. Representation of legend needs to be reconsidered. Default cost heads should be made clear upfront; users discover these cost items (viz., parking) after interacting with the tool leading to surprises.
Recommendations