Report User Testing of Mobile site of Shiksha (for MBA, Engineering)
Documented by Pradnya Patil Team UED Shiksha.com 1.09.2014
Objective: 1. To investigate various criteria that a user considers while searching & choosing a suitable MBA/ Engineering institute on mobile phone. 2. To observe how user interacts with Shiksha mobile version Product takeaways: 1. This may help in identifying the filter / search refine options 2. This may help in identifying the issues one finds while using Shiksha.com on mobile
Scenario User who is bachelors or master’s degree aspirant in his field of interest uses internet to shortlist 5 colleges for his Post graduation. The users are either MBA aspirants or Engineering aspirants
TASK FLOW • User research To give overview of the 3 personas to users and list down the criteria user thinks of while finding 5 institutes for each persona
i) Task 1 Find 5 MBA/ Engineering colleges which you think are best for your profile (free to search through google or any specific website known to them)
ii) Task 2 Find the colleges which you can get if your CAT score is 80 percentile or AIEEE score is 190/360 (free to search through google or any specific website known to them)
iii) Task 3 Repeat tasks i and ii on Shiksha.com- mobile android/iOS iv) Task 4 Find the MBA/Engineering colleges if you have a budget criteria e.g. Annual fees= less than 7 lacs
v) Understanding the labels: Check what users understand by labels currently used in hamburger menu 1. 2. 3. 4. 5. 6.
College predictor Shortlist Filter Articles Institute finder Request e-brochure
vi) User feedback on the following interactions: [What users understand by given action and their response to the same] 1. 2. 3. 4.
Doing a request e-brochure Shortlist (Run time task: when user uses ‘shortlist’ feature, ask him why did he use it) Compare (Run time task: when user uses ‘compare’ feature, ask him why did he use it) Use of filters
vii) Feedback on visuals: 1. Pick one word to describe the color scheme : Dull, Pleasant, Vibrant
2. Pick one word to describe the mobile site- Complex/Messy, Confusing, Simple n clear, Interesting, Boring 3. Icons : Take time to figure out, Can be improved , Interesting, Usual
Participants details Total 12 in number
MBA 6 no. ENGINEERING 6 no.
Internet browsing skill (based on observation)
No. of users who prefer mobile to browse over desktop
3/12
9/12
No of users who are specific to location
Specific to location 6/6
Specific to location 4 /6
MBA (4/6)
ENGINEERING (all)
Social sharing preference
Whatsapp 6/12 Twitter 1/12 Facebook 2/12 Whatsapp
None
No. of users who have used Shiksha before Shiksha user
1/12
UT Findings After conducting the user testing of Shiksha mobile site, the findings were analyzed and then categorized as follows
Roadblock a situation where the test participant cannot solve a reasonable task or which causes the test participant great irritation
Confusion, minor problem This icon indicates minor problem in the flow while performing a task. It caused test participants to question and think for a few seconds
Suggestions This is a suggestion from a test participant that could lead to a significant improvement of the user experience.
Insights Important points to be perceived deeply and further action points to be planned accordingly
Roadblock a situation where the test participant cannot solve a reasonable task or which causes the test participant great irritation
1. Disappointment to see results on Category page
A. Resulting colleges are not familiar to users Users have pre-acquired knowledge of the best institutes which they expect to see on first search result page. Shiksha’s category page or SRP shows a list of random colleges which disappointed (12/12) all the users
-Picked quotes“I have never heard of these colleges” “Its not showing reputed colleges” “There is no sequence, no ranking, they are showing some
random unknown colleges” “FMS to top college hai, top mein kyun nahi aa raha?”
“Jo ranking wala page tha, wo jyada sahi tha” “Apko nahi lagta, yaha pe ranking ke basis pe ayega to jyada
MBA
Engineering
sahi rahega?”
Case: when users were prompted to go to home page and explore the colleges
1. Disappointment to see results on Category page B. Couldn’t get results specific to their branch Users didn’t know how to narrow down search results to see colleges of their course.
10/12 users did not notice filter Hence the idea of filtering out colleges specific to their branch didn’t touch their minds.
-Picked quotes“ I don’t know how ‘to open search’)
Category page/ SRP
to find only civil engineering colleges here” (went
2. Wrong Information Even with 1 wrong information users loose credibility of site 2 out of 12 noticed wrong info displayed on category page -Cut off -Affiliation -Fees And said that they would not like to trust Shiksha
-Picked quotes“ Its showing me wrong
affiliation”
“ Its cut off is not ‘x’, its higher, also the fees are wrong”
3. College predictor-unavailability of seat at particular location is not communicated
Results page V Filters V Location tab >
Delhi?
Case: When a student enters exam rank on JEE college predictor and filters it as per his desired location, he doesn’t find his location. It happens because the location doesn’t have any college which would have seat availability for the given rank. This information is not communicated. It is misunderstood that Shiksha doesn’t have information for the given specific location -Picked quotes“ Yahaan pe toh Delhi hai hi nahi”
College predictor> JEE > Filters > location
4. Failure of Task 2- because of ‘Filter’ being unnoticed Task 2- finding list of colleges according to cutoff/percentile was failed Reason 1 : Filter being un-noticed (cut-off filter being hidden) Reason 2 : Search by cut-off option/filter is not provided anywhere else 4 out of 12 users went to Google to perform task 2 and rest couldn’t perform the task Recommendation: Coach marks can be added to this page
-Picked quotes“ Why there is no option of entering cut off on Home page” ?
Category page/ SRP
5. Data Source: Naukri.com being unnoticed, ignored Users are unaware of the Naukri.com website and its credibility of being data source for placement information. The reason is, users are in student stage and hence have not yet come across Naukri.com as they have not yet sought for a job. Adding the tagline of Naukri i.e. ‘India’s No. 1 Job Site’ would be helpful to understand the value of data source.
-Picked quotes“ It is a source as they say, but I don’t know Naukri.com”
Location: Comparison chart
Confusion, minor problem This icon indicates minor problem in the flow while performing a task. It caused test participants to question and think for a few seconds
1. Confusion whether Shiksha is updated or not One can not figure out whether information on Shiksha is updated or not. e.g. it shows cut off (ranking page, category page) but its not clear whether its showing last year’s cut off or not Fees: whether it is revised/updated
-Picked quotes“Shiksha is not offering me last
year’s cut off”
“I don’t know whether this fee is revised or not, I would visit the college’s website to check revised fee” “Career360 is good, it says best colleges in 2014”
Is it last year’s cutoff?
Fees revised or not?
Ranking page - cut off, Category page – fees,cutoff
2. Fees
Fees:
a.
Fees are not provided on ranking page
b.
It is not mentioned whether the fees are per annum, per semester or for entire course on category and SRP b. Fees per annum or per sem?
-Picked quotes“Fees is missing here”
“These fees are per
a. Fees details are missing
Ranking page,
Category page
annum or per semester?”
3. Difficulty in criteria selection Location selection- Home page Users found it difficult to select ‘ALL CITIES’ location on home page 2 out of 12 (all who were not specific to location)
Reason 1. Repetition of ‘All cities’ 2. Visually confusing, because one of the ‘All cities’ tab is already shown in colour which is assumed as ‘selection’ and hence users clicked back button thinking that its already selected but it is not
Visually confusing
Home page > location dropdown
4. Difficulty in criteria selection Location selection- Filter tab in category page
Users didn’t find ‘ALL CITIES’ option while changing location preference under Filter tab 3 out of 12 (all who preferred Delhi as location but would not mind to relocate if they get a very good college) Case: on Home page, user has selected ‘Delhi’ location and now he wants to check colleges all over India. Now to select all cities user has to go back to Home page again to change location preference
-Picked quotes“how do I check colleges in all over India?”
‘All cities’ not provided under location preference of filters
Category page > filters > location
5. Difficulty in ‘refining’
Refine button: Users (2/12) who clicked on ‘Filter’ tab, were confused about what to do next after selecting criteria. They were prompted to click on ‘REFINE’ tab and then back tab on top to see the results Problem : duplication of ‘Refine’ tab
-Picked quotes“next what
to do?” “I don’t know how to see colleges now”
Category page/SRP > Filter page
6. Confusion in ‘Compare’ When you click compare, It gets added into My List, but comparison doesn’t appear. One doesn’t know what else has to be done to see comparison chart. Terms like ‘Add to compare’, ‘click to compare’ would make it more clear.
-Picked quotes“oh its added into list, but now I don’t know how to compare” (beginner)
Category page > My lists
Suggestions This is a suggestion from a test participant that could lead to a significant improvement of the user experience.
Suggestions
1. It is good to see list of companies coming for placements, but both Engg/MBA users are more interested to know what package they had offered -This information is available currently only for MBA -It appears only in compare option
9/12 users
2. Quality of the faculty in college is one of the important factor for students and their parents while selecting the institute. Providing the details of Educational background and experience level of the faculty is essential. 7/12 7/12 users
Suggestions
3. Users would like to see other academic eligibility criteria along with ‘Cut off’ Like, 10th 12th percentage, CBSC score, bachelors degree percentage, etc This information is hidden under ‘Admission procedure’ tab on Institute page which is not easily accessible. “for good college, you should not only get good percentile but also good marks in 10th,12th and bachelors” “only 90+ CAT score wont assure you good college” “information about academic score eligibility is necessary” “they have not mentioned other criteria of eligibility other than Cut off “now colleges have clubbed 12th and JEE score”
5/12 users
Suggestions
4. Users would like to know what is the refundable fee amount of the institute, because many students cancel their admission if they get admitted to better college. 3/12 users
5. Students would like to see whether college is offering any scholarships 2/12 users
Suggestions
6. It is good to see fees of displayed colleges in separate column which is easier to compare. User gives following example of career360 website’s ranking page 1 user
Desktop version of Career360.com (Shiksha.com desktop site has such columnar version, which is not incorporated in mobile version)
Insights Important points to be perceived deeply and further action points to be planned accordingly
Insights Insights
1. All 12 users said that they prefer to get suggestions from their parents, relatives, friends, college students, peers and faculty as they are experienced and they have acquired knowledge in the field. Primary research source
Secondary research source
Parents, Relatives,
Internet,
friends,
Google,
Students,
Online reviews
peers
Decision making
faculty
2. 5 out of 12 students said that they would trust students’ reviews about colleges than the reviews given by faculty(faculties are biased to their colleges sometimes) or college representative. “There should be open reviews like flipkart” “ I would not trust reviews on college website. They remove negative reviews” 3/12 MBA
Insights Insights
3. 3 out of 6 MBA students would choose college considering what extra that college offers to them than other colleges. Extra = extra courses, project, internships, facilities, programs, extra curricular activities “I will see what extra do they offer to me to add to my CV” 3/6 MBA
4. 4 out of 6 engineering students who had location criteria as Delhi, had appeared for IP University’s entrance exam. Users expected to see that exam in the list of exams under Filter on category page. “it doesn’t show IP entrance exam in the list” “they have not listed all the exams, they have listed only top ones” 4/6 Engg
Insights Insights
5. 2 out of 6 engineering students said, “college which is older is preferred”, “It is important to know year of establishment” , “pata chalta hai, kitni batches bahar nikli hai” 2/6 Engg
6. 2 engineering students would like to search only government colleges and 1 only private colleges 3/6 Engg
7. 7/12 of students who would trust college rankings which are generated by some trusted party or if it is according to some survey conducted 7/12
Insights
8. 4 out of 12 users said that they visit college’s website to check admission and counselling dates. They don’t know any website which gives such updates. 4/12
9. 2/3 users who came across sharing email id said that they will not like to share email id because Shiksha will start sending emails and might spam too
2/3 email id sharing
Insights
10. 3/12 students got confused with the term institute. They thought it means ‘coaching institute’. 3/12
11. 1 user who came across register window while shortlisting more than 3 colleges, said that he would not register as it will take time to fill up the form and then do the verification process as they will mail him for verification
Insights
12.
Google search text entered by users
MBA • • • • • • • •
Times top MBA colleges in India MBA colleges cutoff Top MBA colleges in India Top 10 MBA colleges after IIMs Best mba colleges in India Top mba colleges in India Top 5 mba colleges in Delhi NCR Top 10 mba colleges in NCR
ENGINEERING • • • • • •
Mechanical institutes in delhi Top 10 engineering colleges in delhi NCR Top 10 civil colleges in delhi Top 10 hotel management institutes in delhi Top colleges of b.tech in delhi Top 5 colleges of b.tech
The images show an autocomplete text field, and the pick list of search predictions supplied as the user enters the search query. All these auto complete options by Google say ‘Colleges’ instead of ‘Institutes’. So it is highly recommended to use word ‘College’ than the ‘Institute’ on Shiksha.
Mobile google search
Desktop google search
Insights
14. Links/ websites users have used or refer MBA • • • • •
Indiabix.com Glassdoor.com Stupidsid.com Pagalguy.com UGC website (to know colleges and their affiliation) • HT horizons (CAT articles) • Time website and Time faculty
ENGINEERING • • • • •
IP University website (check cutoffs, important dates etc) Studentchacha.com Faadooengineers.com Yahoo answers (college reviews) Facebook pages like – Education world, Education India (follow pages to get updates) • Seniors’ blogs (where they talk about college ratings) • FITJEE rank predictor
15. SELECTION CRITERIA - MBA 1st criteria
4
2
2nd criteria
1
2
2
3rd criteria
1
2
2
4th criteria
1
2
5th criteria
1
1
1
1
1
3/6 users are location specific
SELECTION CRITERIA -
1st criteria
2
ENGINEERING 2
2 Government/private college
2nd criteria
3
1
1
1 Government/private college
3rd criteria
1
4th criteria
1
3
1
1
1
1
1 Transport
5th criteria
1 Transport
6/6 (ALL) users are location specific
What do these TERMS mean to them
What do these TERMS mean to them
1. COLLEGE PREDICTOR “Will help me out in deciding college” “No idea” “Acc to my JEE rank, it will tell me which college I’ll get” “Will tell me which college I’ll get when I feed rank”
What do these TERMS mean to them
2. SHORTLIST “Is is kind of Bookmark” “To save it for next time reference”
3. FILTER “Is to change your preferences” “To narrow
down your results”
What do these TERMS mean to them
4. ARTICLES “News”
5. INSTITUTE FINDER “It will show location of the college when you enter college name” “It is a tool to find the colleges as per your rank”
What do these TERMS mean to them
6. REQUEST E-BROCHURE Once I click, it will download the brochure It will get downloaded Will display campus, placement etc information Have to fill
the form and then they will display brochure
They will mail me brochure once I click Will send brochure on my email id Get detailed information
You receive brochure soft
copy. Maybe it’s payble
Feedback on visuals 1. Pick one word to describe the color scheme : Vibrant - 1/12 Dull - 0
Pleasant - 11/12
Feedback on visuals 2. Pick one word to describe your interaction experience on mobile site
Interesting 2/12
Boring 0
Confusing 0 Complex/messy 1/12
Simple and clear 9/12
Feedback on visuals 3. Icons design
Usual 2/12
Takes time to figure out 2/12
Interesting 0
Can be improved 8/12
COMPLIMENTS that Shiksha received “It has good
features like college predictor” “Shiksha has provided all required information at one place!” “I’m really amazed ki itna kuch diya hua hai, ye pehle mil jata to, It would have been helpful” “Call option is good, it’s like JustDial” “They have displayed no. of credits per semester which is very helpful” “Yahaan images bhi di hai infrastructure ki, nice. now I can believe that this college has this infrastructure” “Shiksha has very good and crisp information” “Top ranked colleges information on Shiksha is very helpful” “This website
is good, maybe it is not much popular” “They have provided important things about the college (institute page)”
THANK YOU