Report Pilot User Testing of Mobile site of Shiksha (for MBA, Engineering)
Team UED Shiksha.com 24.07.2014
Objective: 1. To investigate various criteria that a user considers while searching & choosing a suitable MBA/ Engineering institute on mobile phone. 2. To observe how user interacts with Shiksha mobile version Product takeaways: 3. This may help in identifying the filter / search refine options 4. This may help in identifying the issues one finds while using Shiksha.com on mobile Scenario User who is master’s degree aspirant in his field of interest uses internet to shortlist 5 colleges for his Post graduation. The users are either MBA aspirants or Engineering aspirants
TASK FLOW • User research To give overview of the 3 personas to users and list down the criteria user thinks of while finding 5 institutes for each persona i) Task 1 Find 5 MBA/ Engineering colleges which you think are best for your profile ii) Task 2 Find the colleges which you can get if your CAT score is 80 percentile or AIEEE score is 190/360 iii) Task 3 Repeat tasks 2 and 3 on Shiksha.com- mobile android/iOS iv) Task 4 Find the MBA/Engineering colleges if you have a budget criteria e.g. Annual fees= less than 7 lacs
v) Understanding the labels: Check what users understand by labels currently used in hamburger menu 1. 2. 3. 4. 5. 6.
College predictor Engineering entrance exam Top ranked colleges Articles Request e-brochure Filter
vi) User feedback on the following interactions: [What users understand by given action and their response to the same] 7. 8. 9. 10.
Doing a request e-brochure Shortlist (Run time task: when user uses ‘shortlist’ feature, ask him why did he use it) Compare (Run time task: when user uses ‘compare’ feature, ask him why did he use it) Use of filters
vii) Feedback on visuals: 11. Pick one word to describe the color scheme : Dull, Pleasant, Vibrant 12. Pick one word to describe the mobile site- Complex/Messy, Simple n clear, Confusing, Interesting, Boring 3. Icons : Take time to figure out, Interesting, Can be improved
Pilot User testing- Shiksha mobile site Total no. of participants tested -
Participant type Senior software engineer Junior software engineer Graphic designer Design intern(design management)
Internet browsing skills 1 1 1 2
Beginner Intermediate Expert
1 3 1
Gender
Age 20-25
5
5
Men Women
2 3
UT Findings After conducting the Pilot user testing of Shiksha mobile site, the findings were analyzed and then categorized as follows
Confusion, minor problem This icon indicates minor problem in the flow while performing a task. It caused test participants to question and think for a few seconds
Roadblock a situation where the test participant cannot solve a reasonable task or which causes the test participant great irritation
Suggestions This is a suggestion from a test participant that could lead to a significant improvement of the user experience.
Insights Important points to be perceived deeply and further action points to be planned accordingly
Confusion, minor problem - results I. “What does ‘best match’ mean?”
Case: User searched through search box and arrives at SRP result. User didn’t understand what ‘best match’ and popular meant here.
II. Refine your results-
difficult to correlate with check mark as drop down menus are provided at other filter page Case : a. user searches for MBA colleges from home page and lands on category page. Couldn’t find relevant MBA colleges as per his desired course. Goes to Filters and refines it. Again disappointed by not seeing IIMs on category page. b. User goes to search box and lands to SRP. The filters found here are different from the earlier one.
a. Home > category page > filter
a. Home > open search > filter
III.
Duplication of the ‘refine’ links on filter page lead to confusion
Case: When user lands to SRP through open search Home > open search > filter
IV.
Home button is hard to find Case: User didn’t find relevant search results on category page, so he wished to go back to home page and change the preferences/ re-enter the details.
Home ?
V. Search bar is hard to find
Case: User didn’t find relevant search results on Category page, so he wished to type in search box (open search). It took a little long to find the search box as its not easily accessible.
Search box ?
VI. On institute detail page, MBA courses are not visible in a glance, one has to click on ‘other course’ then
it shows list of courses Case: User went through many institute detail pages so that he can get an overview of the institute and then shortlist them accordingly.
Roadblock- results I.
Management courses like Hotel management and Event management fall under ‘Hospitality’ and ‘Retail’ stream respectively in main stream drop down tab. But the results of hotel management and event management institutes also appeared in SRP when user searched for MBA institutes by open search. User felt that he is lost and wondered whether he selected wrong stream. Case: Home > Search Institutes(open search)> Full time MBA> Search> SRP
II. All users were disappointed to see the form which appears on clicking ‘Request e-brochure/ download brochure’. They disliked the long form.
III. 2 out of 5 users were diverted to ‘Institute finder’, then got disappointed to see the form which
appears on clicking it. Disliked the long form and skipped filling it. Users expected a tool and not the form.
IV. 3 out of 5 users couldn’t shortlist because they didn’t notice filter icon on category page/SRP. Possibilities: a. User didn’t understand what does filter do (function/term) b. User didn’t understand its icon (visual) c. User didn’t notice the filter button (location) This has to further studied and confirmed
v. User could not search institutes according to government/private college type
Suggestions I. On category page and Institute detail page, information about ‘placements’ is the most important thing for users. As suggested by user, one should be able to easily find reviews about ‘placements’ on institute page
II. On ranking page, Average/highest salary package should be included along with fees. III. On ranking page, shortlist and compare options can be provided to get quick overview IV. Filters suggested by users-top institutes for entrepreneurship -ROI -Infrastructure -Type of college-government/private
V. More information should be provided about the criteria institute considers while shortlisting the candidates
like- 10th 12th marks weightage, weightage to ‘years of experience’, girls marking criteria, etc which are important factors after CAT score
VI. Sub-stream should be provided at home page of Shiksha, so one can get relevant results without clicking on filters on category page
stream mode of learning location
“Where is sub-stream�?
Case:
User has decided his course as MBA- IT. Once he reaches to SRP through home, SRP shows irrelevant lists to him. So user has to go to Filter and select the specialization/sub stream to get desired search results. This increases 4 no. of clicks.
Insights- results I. Users have following 4 priorities while selecting the institute a) b) c) d)
Ranking of the institute PLACEMENTS- brands and offers Cutoff of the institute Fees
II. Users said that they would not trust Shiksha.com because
e) 1. Users have existing knowledge of the top institutes which they didn’t find on the 1 st page of results(category page), e.g. IIMs don’t appear on category page f) 2. They think that only institutes which pay for their listings are shown on Shiksha g) 3. “Coaching institutes like ‘time4education’ website provides genuine rankings of the institutes as its their business” h) 4. They would trust on google and offline sources like friends, etc to get first hand information about the institute
III. Shiksha didn’t help in making decision or shortlisting because SRP didn’t provide ranking and ranking page(‘top institutes’) didn’t provide important info to shortlist the institutes
IV. Users shortlist the institute by calculating its ROI(Return of investment/value for money) factor- Fees vs Placement package. Providing ROI factor on the category and ranking page would make the job easiest
V. Users who were unaware of the exams to be appeared for getting into MBA, couldn’t find exam guide or MBA preparation guide
VI. Following are the keywords which users typed in the search box-‘top institutes of MBA’ -‘top MBA colleges’ -Top MBA colleges in India’ -‘ Top private MBA colleges’ -‘ Good private college+ best placement offer’ -’Preparation of CAT’ -’How to apply for MBA’
Conclusion Quick fixes
1. Duplication of ‘refine’ on filter page has to be fixed (ref: slide 9) 2. Drop-down tab of Sub-stream/specialization may be shown upfront on home page moving from ‘filters’ to get relevant search results in first step. Right now, users are forced to ‘refine’ after preliminary search (ref: slide 18) 3. Discoverability of Home button is not intuitive. (ref: slide 10) 4. Discoverability of ‘Search bar’ is also not intuitive; may be introduced either on the top navigation bar or as global search bar for quick access (ref: slide 11) 5. Suggestion: Rankings of the institutes to be displayed on category page/ SRP and Ranking page. 6. The UI of Filter options of Category page and SRP is not consistent; may need attention. (ref: slide 8)
Near future fixes 1. 2. 3. 4. 5. 6.
Nomenclature - ‘best match’ on SRP can be revised to make it more self explanatory (ref: slide 7) Long form of the ‘request e-brochure’ needs attention (ref: slide 14) Long form of ‘Institute finder’ has to be resolved(ref: slide 15) ‘Filter’ tab being unnoticed by users- need more research to figure out the reasons (ref: slide 16) ‘Placement reviews’ tag may be introduced on ‘institute detail page’ (ref: slide 17,i) Average/highest salary package along with fees may be displayed at category page/SRP to give quick overview of the institute
Long term fixes 1. Categories/ main streams of courses to be categorized appropriately (ref: slide 8) 2. Feature Proposal: ROI factor may be introduced on category page/ institute detail page to help/guide users make decisions (more research needed) because users often calculate the ROI before shortlisting the institute
Thank you.