PORTFOLIO INTERAC TION
DE SI G N
PORTFOLIO CONTENT
1
Multi-channel In-vehicle Interaction Design for Detection Enhancement
Exploration in hardware and software intergrated design Individual Graduation Design Mar. -Jun. 2017
2
HexAqua
An Intelligent Drink Hub
Exploration in IoT smart devices Course Project Apr. - Jun. 2016
Award: Excellent Bachelor Thesis of Shanghai Jiao Tong University ( Top 1% of 2017 ) Suggested Reading Mode: Two Pages
3
PAT-PAT
AI Chatbot to Connect with Empty-nest Parents
Exploration in multimodal interaction with AI assistant Personal Virtual Project Jul. -Sep. 2018
PORTFOLIO CONTENT
4
Holo-Piano
Remote Piano Teaching & Traning
Exploration in AR and computer supported collaboration Course Project Nov. 2019 - Feb. 2020
5
Speakat
Serious Game against Social Anxiety
Exploration in gamification and psycho-therapy Course Project Nov. 2019 - Feb. 2020
Suggested Reading Mode: Two Pages
6
Gallery of Automobile Design
7
Gallery of Photography
Multi-channel In-vehicle Interaction Design for Detection Enhancement
Type of Project
Individual Graduation Project
Mar -Jun
2017
Awards
Excellent Bachelor Thesis of Shanghai Jiao Tong University ( Top 1% of 2017 )
1 2
User Research
Benchmark Analysis
3
Design Reasearch
4
Design Process
5
Final Exhibition
6
Conclusions & Takeaways
About the Project This project is an experiment and exploration on one aspect of driving assistance (ADAS) --- blind spot warning, along with multi-channel interaction, aiming to redesign a hierarchy of multi-channel warning system that enhances driver’s overall detection of outside world while perceived in an aesthetical manner. Both interaction and industrial design method applied, the project is a trial on hardware and software blended design. An interactive scale model demo(1:4) was finally presented on the graduation exhibition.
User Research Blind Spot Detection is the most wanted technology
Statistics · 80% information drivers needed is from visual sense. 。 。 · 60/360 , as 1/6 of in-vehicle area is blind zone.
According to a survey*, blind spot warning ranked top on the chart, which took 42% of the vote, even higher than front collision assistance.
· 500, 000 traffic accidents globally every year are resulted from blind spot.
* Nerdwallet
https://www.nerdwallet.com/blog/insurance/survey-consumer-fears-self-driving- cars/
Benchmark Analysis
Hierarchy of warning Drive straight
Mecedes Benz System main flow-chart Speed >30km/h system standby
Multi-channel
Abrupt feedback
Turn signal ON
Weak Warning
caution light on
Visual:
Side vehicle in blind zone
Mecedes has widely recognized multi-channel warning system, but the simultaneous strong warning from three channels may cause panic and misjudgment.
Visual:
light flickering cluster warning
Auditory: di-di alert
Haptic:
steering wheel vibration
Turn signal OFF
Strong Warning
Design Research Channel selection
Initial Concept
Multi-channel communication helps better and faster decoding for receiver (the driver), provided that certain sensory channels are properly selected.
A warning system of two feedback channels enhancing driver’s detection of the outside
Visual cue Saccadic machanism Selective neglect on weak warning Sensitive to sudden signal, e.g. flicker
Auditory cue 24H active Direction and distance implication Sensitive to sudden diviance, e.g. squeal
Surround atmosphere + warning light strip (glanceable display)
Stereo headrest speakers
Haptic cue Sensitive and instant rapid reaction Sudden diviance may cause panic A third channel may disrupt driver
Abandoned for potential of distraction
Design Process
Interaction Flow-chart based on scenarios Vehicle in blind zone, system activated
How driver can detect more Visual and auditory cues to help perceive and locate side car
1. Driving straight Indicate position as side cars passing by Visual weak warning Glanceable light strip follows side car to indicate position only on the door panel.
Auditory weak warning Stereo amplifies environment sound to implicate side car’s position. 1 Vehicle passing by in adjacent lane, the light strip follows to indicate position on each side, which driver doesn’t have to notice.
Visual strong warning Light strip shines all the way to steering wheel and flickers to alert driver.
Auditory strong warning Harsh alert with amplified environment sound.
Visual: Weak
Auditory: Weak (could be shut for comfort)
2. Indicator ON and ready for lane changing
3. Side car too close
System activated and reacts accordingly whether safe to change lane
Warning of side collision
2.1 Safe to change lane
2.2 Not safe to change lane 2.1 When it’s safe to change lane, system stays in weak warning.
Visual: Weak
2.2 Side vehicle insisting on accelerating or no enough space to move, system will warn strongly from both channels.
Auditory: Weak
Visual: Strong
3 System warns from either side when there’s potential collision.
Auditory: Strong
Visual: Strong
When safe to move Indicator OFF Distance back to normal
Auditory: Strong
Design Process Interior design facing demostration Considering the interacitve demo on exhibition, the interior design was intended to be an open sculptic cabin with concentrating festure and minimal other gadgets and decoration.
I leak light concept
Abandoned for low accuracy indication.
II boomarang light concept
Abandoned for unnecessary complexity and distraction.
III fianl U-shape concept sketch Simple, elegant and effective.
Artistic rendering
In this rendering, I tried to convey the idea that light strip can act as not only warning indication, but also atmosphere/ambient light which can be perfectly blended into the interior.
Digital modeling and rendering The model was later milled out of engineering plastic and then assembled with electronical parts embeded.
Visual channel
Ambient warning light strip
Auditory channel
Stereo headrest speakers
Design Process
+
Prototyping and iteration for interaction The prototype ran with Processing motion tracking algorithum, in collabration with Arduino sensors and outputs of both visual and auditory channel.
Camera + Motion tracking
Sensors + Multi-channel output
Initial edition
Tracking dot indicating distance by size and color
Smaller and darker means farther
Bigger and brighter means closer
How it indicated
The tracking dot meanwhile implicated the relative distance of the object by size and color: smaller and darker means farther; bigger and brighter means closer; then the position and lightness of dot was projected onto light strip.
What to modify
I Distance indication cancelled due to signal bounce (technical issue ), left only position implication II Add remote control to switch mode (driving straight or lane changing)
Final edition
Arduino main program flow-chart Two main modes with a standby third mode
Main program with three modes and all sensors integrated
Remote Control Switch to simulate turning signal ON/OFF
Video camera
Processing
Arduino board and sensor module
Sacle model with light strip Mode1
The sensor module
Buzzer
All sensors were integrated into one turret-like module, built with acylic sheet cut.
RC receiver
路 Buzzer : auditory warning 路 RC receiver : mode switch by RC 路 Video camera : motion tracking 路 Sonar : distance throttle
Mode2
Video camera Sonar
Mode1: Driving straight Mode2: Ready for lane change Side collision warning ( standby any time )
Final Exhibition
On graduation exhibition The visitors were invited to stir about in the designated BLIND ZONE to act as if the side vehicle and the model will response accordingly. A remote control was also handled to the visitor to switch the main vehicle’s driving dicision, whether to drive straight on or change lane, hence experiencing different warning scenarios.
PROF. GU AND ME
One minute demo https://www.youtube.com/watch?v=rqke4hR8YAw Full version https://www.youtube.com/watch?v=IhxKvN1mPso&t=134s
Conclusions & Takeaways To be expected in the future Multi-channel in-vehicle interaction is the trend
As vehicles getting smarter, it is crucial that how we present rich infomation to the driver in a more intuitive, multi-channel/model way to assist decision. Meanwhile, car provides a unique space for journey, where occupants can expect more diversity of entertainment.
Adaptive feedback with machine learning to be applied
Considering different drivers, novice or skilled, the system should response accordingly. By learning drivers’ reponse over a certain period, the system should be able to adjust itself to adapt and grow with user.
Hardware and software blended design in a new paradigm Design in intelligent era requires designers to be dedicated in both hardware and software areas, especially in car industry. And also design method needs a paradigm shift due to the change of product form. This project is an initial personal experiment and I will keep on exploring.
HexAqua An Intelligent Drink Hub
About the Project HexAqua is an intelligent system for water drinking. Applying IoT technology, HexAqua can be managed on smart phone. The system is especially designed for families to meet various drinking needs, and to strengthen affection in a subtle way. Data will be shared among family members, therefore users are encouraged to check their beloved ones’ data and send remind to each other, hence bringing the family closer.
Type of Project
Course project (Product Design III)
Apr -Jun
2016
My Roles in Team ( 5 members ) System Design, Interaction Design, Industrial Design
Background of project In China, people love warm beverage, like hot tea or hot coffee, or simply warm water. Thus kettles and cups are widely adopted across country. As more people are getting concerned with drinking water, whether a fit habbit or the quality of drink itself, drinking data nowadays is of growing importance to everyone. In this project, we set our goal as connecting users with drinking data through intelligent drinking devices that most Chinese will use on a daily basis.
Research
Drinking data to connect people
Market
Users
We searched the market for smart devices for drinking like cup and kettle, from which we got rough idea of functions that should be contained.
Smart kettles
Smart cups
We interviewed a few families and individuals using intelligent cups or kettles. Their opinions on using these devices will help us shape our project in the future.
· My daily overall drinking data shall not be collected only by my kettle in kitchen. But other smart cups can’t share data with it.
· Dechlorination (by boiling twice or more) · Timing and reservation
· Drinking record · Reminder of prompt water compensation
+
Smart kettle users · To keep precise temperature of boiled water with my smart kettle is a brilliant function. How I wish that my cup can do so.
· App controlled boiling and perservation
Inspiration
Smart cup users · Intelligent cup can only perserve but not heat up water. The battery only supplies the sensor. · I bought one intelligent cup for my girlfriend as a gift, but I can’t remind her when to drink because only herself can check the record. We don’t feel further connection brought by it.
=
？
Opportunity 1st: NONE product has kettles and cups integrated together so far.
+
+ Data = ？
Opportunity 2nd: With drinking data to connect people and devices, our product will serve better our users with their life.
Persona
Bob
Annice
Bobby
Senior manager in a hi-tech corporation. Keen on new smart device.
Office lady working in community service organization. A qualified mother who cares her family.
Primary school student in Grade 5th.
Drinking habits:
Drinking habits:
Drinking habits:
Pain point:
Pain point:
Pain point:
37 years old husband, father
A typical family of three Our protagonists are from a typical family of three living in the urban area, Bob, Annice and their son Bobby. The three of them love each other. They are leading a mid-class life.
35 years old wife, mother
· Coffee in the morning. · Warm tea for after-meal reading time.
· Cup gets cold yet tee needs to keep warmth.
· Hot coffee in the mornng. · Eight cup of pure water during daytime. · Warm beverage for special period.
· Too devoted in work to forget to drink somtimes.
10 years old son
· Warm milk for morning. · Nutrition pills after lunch, sometimes medicine pills.
· Frequently forgets to take pills.
Yet as a family, each of them is conerned about ... whether the others are drinking promptly and satisfied.
Initial concept After researching and persona setting, we decided to design an integrated itelligent system of drinking water for families. Users are connected with full-chain drinking data collected from smart cups and kettles.
Basic function
Refined feature
Bob
Bobby
· Boiling and heat keeping with remote control · drinking data inspection on app
Sample topolpgy: Annice’s family
· Overall drinking data recorded by both kettle and cup · Users connected with shared drinking data within group
Each user, within a group, can: 1, operate their own cup(s) and shared kettle; 2, read the other group members’ drinking data; thus connected with each other.
Components of HexAqua
Read data only Device to operate
Annice
Two main scenarios of HexAqua I.
HexAqua Cell of base(s)
Smart kettle(s)
Smart cup(s)
II.
Phone APP
Charge and heat
Heat and communicate
Record and operate
Base plate can heat water and charge cup/kettle up. Each in the shape of hexagon, the cells of bases can be extended.
Kettle and cup are each equiped with heater strip and IoT module. Battery powers IoT modules to communicate. Only when placed onto bace can cup and kettle start heating.
With smart phone, users can get access to and control their devices in remote.
85°C
85°C
Extendable cells with more devices to be added. Personalised LED strip as status display.
Each base can work individually as charger and heating power.
Storyboard 85°C @ 7.am
85°C
GOOD QUALITY
The story shows one day of Annice’s family and how this family interact with HexAqua and each other.
1
The night before, Annice filled the kettle and set it to keep 85°C after boiled at 7a.m, which is perfect temperature for tea and coffee for morning.
2
7 o’clock, the kettle had already boiled and carefully kept the water at exact 85°C. To Annice’s satisfaction, the water had been dechlorinated to a good quality.
ANNICGEG’OSAL DRINKIN
BOBBY’S PILL TIME
20x4=80 20x5=100
REMIND HIM
4
The three took their cups with themself and set off for work and school.
5
By 12 a.m, Annice checked Bobbby’s data and there’s no record during the pill-takng period. Annice sent Bobby remind and his cup flashed as well.
35% ACHIEVED REMIND HER
6
3
Annice made tea for Bob and coffee for herself with the perserved water. She then warmed some milk for Bobby in his cup.
Honey, time to relax and drink! Thx darling
Bob found Annice’s record far below her goal. He reminded Annice to drink promptly to meet the goal and not be too busy to take care of herself.
7
After dinner, Bob started his routine reading at the desk to relax himself. Annice brought him a cup of light tea with a base plate to keep heating.
Prototype Sketches of concept
Main hardware components of system
Structure details of extendable cells of bases
Modeling and rendering
APP
select device
My Device - CUP
My Device - KETTLE
Information structure heat up and keep now filled percentage Control
temperature charging state, battery
Cup 1
drinking data, goal and remind heat up and keep reminder (coffee, pill, etc)
Reservation
My Device (Homepage)
list of reservation
cup page (default homepage)
start heating
keep after heating
select device
kettle page; set and heat
confirm whether to boil first
boil/ Heat up/ Keep now ongoing order Control
Water processing features
filled percentage temperature
Shared kettle 1
charging state, battery Reservation
boil/heat and keep list of reservation
Personal data & analysis
ongoing history
Add device drinking data
drinking record
Annice
remind her
pill reminder
Bobby
remind him chat
Main flow wireframe
Add friend My account Me
add heating reservation
DATA to connect people
chat
Friends
reservation list; new reservation
Device management
configure device authorisation reservation list; new reservation
add new reminder
add heating reservation
Bob’ s phone app is taken as an example. In the flow chart, it shows how the cup and the kettle are operated and reserved. Meanwhile, a remind sent by Bob to his wife regarding the shared drinking data among family members, along with Annice’s feedback is also demonstrated.
Friends
Me
UI Design Main pages with UI design
friends list
my page
cup reservation list
kettle page
friends list
drinking data
kettle reservation list
edit and send remind
chat (remind accepted)
Shared data & remind
edit and send remind
cup page when heating
my page
Conclusion remind received by Annice
remind accepted by Annice
The Hexaqua drinking hub integrated smart devices with data collection and application. As one possible node of IoT, HexaAqua is an exploration in how smart devices and data could serve people better in life, concerning both function and emotion. With the blooming of IoT technology, a burst in numbers of IoT devices and size of data is inevitable. How we arrange and present all the infomation to our users in a friendly manner, and how these data can benefit users more, has set crucial challenge for us designers, which has also been my persistent pursuit in the future.
Bachground
PAT-PAT AI chatbot to connect with empty-nest parents
Facts about empty-nest parents 1. Empty-nest parents seeing tremendous growth Empty-nests, in which children have moved apart from their parents, either for study or work, or migration, have now taken a huge part of family types in China. It is estimated that there will be over half of Chinese families with middle aged and elderly empty-nest parents.
2. Most parents receive a CALL every week, 100%
About the project Witnessing many of my friends moving far away from home, leaving their elderly parents alone in hometown, I decided to carry out a virtual project in exploration of possible form of AI assistant to help relieve those mothers and fathers from missing and worrying of their children.
family call 80%
75%
yet most families MEET over once a month. 100%
family meet
75%
70%
50%
50%
25%
25%
0%
5%
10%
3%
2%
0%
10%
5%
15%
3. Smart devices, might not be that smart to connect with parents... Due to aging, declining memory and eyesight may keep discouraging them from interacting with these screens. Yet we are still far from relieveing them of missing and worrying...
Thereby our aging parents, deserve courtesy of modern technology,
Type of Project
Virtual Project
Jul - Sep
2018
My Roles User Research, Interaction Design
to relieve their missing and worrying of the children in distance.
User research
Interview and observation of two families
Key findings
Family1 : Zhang’s family
Video call device: wireless digital frame
Father: Zhang Genbao, 60 (retired) Mother: Wang Yali, 56 (retired) Daughter: Zhang Lili, 28 (works in another city)
Digital frame is much larger than phone which is friendly to us... we love the photos and videos that our daughter shared with us... how I wish I could touch her... ---Wang(mother)
“
”
Voice interaction is welcome.
Visual playback helps relieve missing.
Parents are afraid to bother the child.
Video call device: iPad
Family2 : Gao’s family
“
We use iPad just for video call for it’s a bit complicated to operate... Siri does help but she don’t really understand us sometimes, so we never aroused her again... ---Deng(mother)
Father: Gao Kang, 53 Mother: Deng Ling, 51 Daughter: Gao Huan, 23 (studies in France)
Persona
”
Mums miss and worry about their children more than dads.
Proactive attention from child is the key to comfort parents.
One typical week of Lucy Events
2 Check Zoe’s social media
Casual chat with Zoe in daytime (not daily)
update in the morning
Misses or worries about Zoe when not heard for long
Routine video call on the weekends
weekdays
Lucy Lin
weekends
Age 56; retired; Lives with husband; Daughter in France. Due to aging, screen gazing gets difficult.
Challenges
Can hardly read small letters Can’t stare at screen for long
Being afraid to bother “Most of conversation is started by her... we are afraid to disturb.”
Frequent but not skilled user of social media. The only daughter, Zoe, studies alone in France, which has 6 hours time difference with China.
Eager to know about Zoe and worries about her “...can’t help worry about her from time to time...”
Deng
Opportunities
Voice broadcast of updates “How I wish someone could read it out loud...” Wang
Remind the child to say something from time to time
Deng
Status indication (available or not) visual playback of memory
Lack of physical interaction “... how I wish to reach out and touch her through the glass...” Wang
Physical interaction and AR feedback
Initial concept of PAT-PAT AI chatbot that helps relieve missing Inspired by user research and persona, PAT-PAT is an AI chatbot that helps senior parents, especially mothers, to relieve their missing and worry of children. PAT-PAT is not a chatty toy or a spy. It knows best what you may be thinking about and it will tell you exactly what you want.
1. Activity broadcast
2. Visualised memory play
3. Interactive video call
According to phone and biometric data, PAT-PAT tells you whether your loving one is happy or sad, busy or not.
Once detected yearning or worrying, PAT-PAT will invite you to a projected slideshow of the photos and videos to relieve missing.
PAT-PAT acts as if an avatar of the one on the other side of the screen, with whom you can interact physically.
to wash away worry and anxiety
Two ways to arouse PAT-PAT 1. voice arouse PAT-PAT? Yes, Lucy? Did Zoe have lunch? According to her activity and biometric data, she has absolutly had a great lunch.
active album to relieve user’s yearning
to feel closer by physical feedback
Head pat means to praise and comfort 2. just pat it! Zoe’s calling! Lucy pats on it and says...
Answer it, please. Right away!
PAT-PAT invites user to pat on it as if patting on the head of their children and to encourage them like the good old days. With both physical and visual feedback, the experience is even more real.
Conversation design Sample dialog flow
Status broadcast: occupied
This flow shows the most frequent conversation, when Lucy asks about her daughter Zoe. PAT-PAT detects if Lucy is in a possible nastalgic mood. If so, PAT-PAT then will suggest a slideshow of recent days’ pictures and videos that Zoe shared.
She seems to be in the office and fully devoted at work now.
PAT-PAT, how’s Zoe right now?
?
Suggest for album review
Ask too frequently?
Ask about Zoe
Here we go! Yesterday, Zoe sent you this photo saying, “What a wonderful day of intern!” She looks fulfilled. Last week, ...
Dont’t worry, she’s doing fine. What about a look back on her recent days?
in a possible missing mood
Yes
Slideshow
Yes, I’d love to see it.
No
Yes End
She’s doing pretty good.
End
PAT-PAT’s suggestion PAT-PAT suggests user according to different status, so as to relieve the user from possible mood of missing.
Decision tree This tree library shows the main flows of how PAT-PAT replies to, and then kindly suggests user, either to make contact or to relieve.
User intention
PAT-PAT? Yes, Lucy?
?
PAT-PAT’s reply
Ask about Zoe What is Zoe doing right now? What is Zoe busy with? What’s going on with Zoe? How’s Zoe? Tell me about Zoe. Is Zoe busy? Is Zoe free now? ...
Possible activity Location She seems to be
at office at school at home in the park in the gym ...
+
Test and iteration of the tree Wizzard-of-Oz tests were carried out for iteration, as feedback from parents were collected and considered.
relaxing watching TV dinning busy at work concentrating in study sleeping taking a nap jogging exercising ...
Video call
Start video call
Call Zoe, please. Please call her if she is available. How I wish to see her right now. ...
at work /... Ooops, sorry she is now sleeping /... exercising /...
Right away!
Send message
Record and send message
Can you tell Zoe...? I want to say something to Zoe. ...
Ok, I will send her a message goes... OK. Say it as if you are facing her. ...
End
End
Status
What about then ...?
Available
Video call / Message now -have a video call right now? -tell her about what you are doing? -ask her what’s up? -say something? ...
Occupied Asleep Back soon
Visualised memory replay Don’t worry. What about we... -check what Zoe shared recently? -look back on her recent days? -take a look at the album? -listen to some music? ...
Respecting privacy, child’s activities and statuses to be shown are set on his/her own phone app. Social
relief for missing when unreachable
Wait for Zoe
-wait for a while? I’ll inform her when she’s available. I’ll tell her when she’s back. Her alarm clock will be on in 1hr. ...
Activity
Ask too frequently? End
View photos What had Zoe shared these days? I want to have a look at the photos. Show me Zoe’s recent days. ...
Photo/video slideshow Would you browse it one by one? I have prepared a slideshow, shall we? ...
Location End
TV
!
Confirm again in case of emergency If needed, I will send her a message right away. It’s fine, Lucy, I will call her right now. ...
Yes
User journey
Ok, I’m listening. Just say it as if you are talking to Zoe.
1. Zoe’s social media update broadcast Activity broadcast
Morning Lucy !
I’d love to. Sure, please
Glad to hear that. Thanks, PAT-PAT.
Zoe’s said: “Another busy working day. Great experience for intern LOL.”
Zoe updated her facebook yesterday. Would you like me to read it out for you?
With pleasure. Would you like to say something to her then? She will receive your message when she’s up.
My pleasure , Lucy.
PAT-PAT reads out Zoe’s update and projects the picture from inside onto its transparent shell.
PAT-PAT invites Lucy to say something back to Zoe. A text message with sound record will be sent.
2. Zoe’s receiving Lucy’s message
7:30
a.m 6hr ago
I’m really proud of you Zoe on... 3’
Ok. She will receive a text message saying “I’m really proud of you Zoe, on hearing your wonderful day. Take care of yourself dear ! ” A record of your voice will reach her as well. Oh it’s so sweet of you PAT-PAT. Thank you.
Yes, sure, please.
PAT-PAT detects that Lucy’s up. It shines to greet and asks if Lucy wants to hear about Zoe’s update.
PAT-PAT
Umm... ok. I’m really proud of you Zoe, on hearing your wonderful day. Take care of yourself dear !
view
Instructive conversation “Siri don’t really “talk” with us. When she Deng can’t understand me, she wouldn’t guess or guide us... I do appreciate it if she can instruct or guide us promptly.”
Under PAT-PAT’s instruction, Lucy speaks out her message. PAT-PAT then records and translates it into text message.
3. When Lucy misses or worries about Zoe “ I’m really proud of you Zoe on hearing your wonderful day. Take care of yourself dear! ”
7:30 PAT-PAT
a.m 6hr ago
I’m really proud of you Zoe on...
PAT-PAT, what’s Zoe’s doing?
PAT-PAT, what’s Zoe’s doing?
Activity broadcast According her location and biometric data, she is taking a nap in her office.
Don’t worry Lucy, she is doing well at work. Umm... What about we have a review of her recent days? Oh, I’d love to. Please. Thanks.
After a while... Several hours later, Zoe’s up.
Zoe receives a message from PAT-PAT.
Zoe plays the voice record before she unlocks the phone.
Lucy asks PAT-PAT what’s going on with Zoe during the day.
If Lucy asks too frequently, PAT-PAT will transfer to another topic as to relieve her.
User journey 4. PAT-PAT reminds Zoe’s to say somthing from time to time Visualised memory playback Yesterday, Zoe posted this on Twitter and said... Last weekend, Zoe went hiking and shared... Last Wendesday, Zoe... ...
Wang
+6h
Proactive attention
“... I would take out the digital frame and browse photos and videos she shared. When I saw her vividly in screen, I did feel much better .”
4:30 PAT-PAT
p.m
Zoe’s
Lucy’s
just now
Zoe sent you a new message, Lucy! Having tea dinner break with with my friends. Good night mum and dad~ dad!
Time to say something~
After a while... PAT-PAT may invite Lucy to a playback of Zoe’s recent days with photos and videos that Zoe shared.
PAT-PAT reminds Zoe to say something if mom is detected worrying, or if Zoe’s been away for too long.
Zoe’s having a tea break together with her colleagues at work while Lucy is about to sleep.
Zoe takes picture of their snacks and says good night to her parents.
Conclusion
5. Routine video call at weekends Interactive video call Wang
Zoe’s calling!
“... how I wish I could reach out and touch and feel her through that piece of glass...”
Answer it, please.
12:21
Mom is patting you on head.
Sunday morning, Zoe starts the video call. PAT-PAT tells Lucy that Zoe’s calling. Lucy comes up and pats it to answer.
PAT-PAT shines to tell Lucy that Zoe has sent a new message.
Lucy sits down with PAT-PAT in hand. The family have a joyful video talk.
Lucy pats PAT-PAT as if patting Zoe’s head. PAT-PAT shivers with Zoe’s laughter and the her cheeks will be full of hearts .
On the other side, Zoe will also see screen filled with love as PAT-PAT tells her that mom has just patted on her head.
Application of AI for seniors’ care There is a clear and promising future of AI technology applied in senior care, from big data health analysis to emotional bond. Proactive inquiry-based interaction With deeper understanding of human emotion and intention, AI chatbot would carry on even longer and more encouraging conversation. Anti-addict design for digital wellbeing Life proceeds in reality. Intelligent device should serve as only life assistant but not to spoil user, respecting both mental and physical health, in case of the ampification of sorrow resulted from device error.
HOLO - PIA NO Remote AR Piano Coach About the project Learning piano as an amateur could be effort taking regarding the communiting back and forth from the coach’s place. Yet a remote session in the form of video call will lose a lot of essential information compared with side by side tutoring. We decided to explore and utilize the features of Hololens and MIDI, under the notion of Augmented Reality and Groupwork to build a remote teaching and training platform.
Type of Project
Course Project (MR&TUI, Groupwork)
Nov. 2019 - Feb. 2020
My Roles
User Research, UX/UI Design
System Diagram According to the user findings, we designed a system utilizing HoloLens to project coach’s hands and key pressed onto the learner’s keyboard. The virtual hands are captured by Leap Motion. Meanwhile, both can also see each others’ upper body.
Crucial User Findings
Learner’s image
We interviewed some beginners and have some skilled piano player reflected on their training. Here are the findings:
Coach’s image
MIDI signal
1. Fingering is important. Beginners often get lost in fingering and make mistakes in initial stages. A close-up view of coach’s hands is crucial for learners, either being remote or on the spot. 2. For learners, a piece of continuously played melody is rewarding yet requires a lot of practice. Coordination of two hands is hard for beginners and often results in discouragement. The technique of accompaniment play is often adopted by the coachs in order to help students catch the tempo.
MIDI signal
Coach’s virtual hand
Coach’s devices
Learner’s devices
Leap Motion
HoloLens
Webcam
Webcam
MIDI(Piano) Monitor
MIDI(Piano)
1. Setup and detection Using Vuforia SDK, we assigned a little standing pad printed with HOLO-PIANO logo as marker so that Hololens can map the corresponding UI onto the note, midi and certain position in space. Idealy the marker is fixed on Piano. Since we don’t have an 88-key MIDI, we use the 25-key instead and in unity, the relative position is claimed.
2. Feature1: Remote coaching Coaching contains two sessions: A dashed square in HoloLens will guide user to aim at the marker. Once the marker is recognised, the app will launch.
Demo: the coach plays and the virtual hands will float over learner’s keyboard hence the fingering can be observed in a close-up view. Accompaniment: the coach plays along with the learner so that the coach and give a tempo and help the learner establish an initial sense of the tune.
Evaluation and Conclusion We evaluated our prototype on semester project exhibition. The idea was generally praised as appealing and unique. Yet due to the complexity of system network, the experience was influenced by the accumulated delay in signal transmission.
3. Feature2: Self practicing The note is encoded in XML and user can practice with the instruction of corresponding dropping bars over the keys. In addtion, certain sections can be selected and practiced repeatedly.
By the end of the project, the global pandemic burst out and profoundly changed our life. More events are hosted online and remote education and tutoring has seen the biggest surge throughout the years. Given the condition granted, we are looking forward to our design’s positive impact. In one way we can replicate and even enhance the experience of conventional piano coaching, and in another way we would happily see if more coaches’ business influenced by the social distancing will get back on track with a new modality norm.
Speakat
Serious Game against Speech Phobia About the project A meta-analysis on serious games for mental health shows effect favoring serious games over no intervention controls in reducing disorder-related symptoms[1], including the evaluation of computerized CBT (CCBT)[2]. By simulating the embarrassing scenarios, the game intentionally and repetitively expose the user under the imaginary fears. Then Speakat will empower the user with corrsponding “magic power� and help user guide himeself/herself out of the dilemma. Unlike conventional thought training, a visualised instant feedback is expected to help user establish new cognition and evaluation more quickly and solidly. The times of power use are not limitless and an anti-addiction and exit mechanism was also discussed over.
One of the magic power: Cabbagizer. By turning audience temporarily into cabbages, Speakat helps the user out of distration.
[1]Lau, H.M., Smit, J.H., Fleming, T.M. and Riper, H., 2017. Serious games for mental health: are they accessible, feasible, and effective? A systematic review and meta-analysis. Frontiers in psychiatry, 7, p.209. [2]Fleming, T.M., Cheek, C., Merry, S.N., Thabrew, H., Bridgman, H., Stasiak, K., Shepherd, M., Perry, Y. and Hetrick, S., 2014. Serious games for the treatment or prevention of depression: a systematic review.
Type of Project
Course Project
2019 - Feb. 2020 Nov.
My Roles UX/UI & Animation Design
Demo Video https://youtu.be/_Iy0khjQlqY
An overall evaluation is proposed after each course of training.
Gallery of Automobile Design I was trained as a professional automobile designer for two years in the university, during which I experienced systematic practice of sketching, rendering, clay making and digital modeling. Many of the projects were supervised by professional designers from OEM. Later during my two-year UX designing experience in OEM, I worked closely with exterior and interior deigners and thanks to my previous training experience, I could communicate with my colleagues within the styling design discourse which they are familiar with and exchange both sides’ idea efficiently.
Grand Hongqi Concept The concept redesigned the classical Grand Hongqi sedan for state leaders. Inspired from the stance of the design from half a century ago, the new design has demonstrated confidence in both aesthetic aspect and political field of modern China.
GTW Grand Tour West Concept
GTW Concept was designed for National Geography photographer to ride in the enormous land of west China. The vehicle will be the photographer’s best working and living partner throughout the trip, where the photographer can have a rest, move around and set up his camera to capture the amazing moment of natural beauty. The project was sponsored and supervised by Changan Auto and finally presented in tape graph and clay model (1:5).
Some other skethes...
Gallery of Photography I love observing and documenting the world, especially through my lens. I started from scenary shooting and gradually I found that the most touching moments are from those people with stories. For recent years, I turned my frame to people and capture their happiness, sorrow or thrill. I took a second major of Communication in direction of New Media in university and as more photos I take, I came to realize that there’s an invisible responsibility lying on my shoulder, that is, as a photographer and a media source, I must be loyal to the truth and be vigilant to possible rumor that my picture may cause due to deliberate interpretation, since from time to time photographers tend to exaggerate their discovery consciouly or unconsciouly in order to absorb more attention. Behind the camera, I will keep exploring the world and inner me as a life long hobby and mission.
Eyes collection since 2014
Ratha-Yatra Hougang Singapore 2015
In Class SJTU China 2017
Thank you In good experience, I believe.
Copyright © JUNHANG YU 2020. E-mail: jeremiah.y@hotmail.com