VIRTUAL REALITY MEDICAL SIMULATION
VIRTUAL INTERACTIVE HUMAN ANATOMY
VIRTUAL INTERACTIVE SIMULATION ENVIRONMENT
I worked on VRMedSim from September 2014 onwards in Keio-NUS CUTE Center where my official job title is research assistant. This project is developed under the Augmented Learning Group in Keio-NUS CUTE Center in collaboration with the National University Hospital Singapore, also the source of our funding. VRMedSim is a virtual reality simulation for educational use by the NUS Yong Loo Lin School of Medicine. Under the umbrella of VRMedSim are two projects - Virtual Interactive Human Anatomy and Virtual Interactive Simulation Environment. Our team expanded from its early days of one designer (myself) plus one programmer into our current team of three programmers, two designers, and a few interns for 3d modeling and animation. Due to the project’s long term and on-going nature, kindly keep this portfolio document private even though I have ensured the contents here are not NDA sensitive.
VIRTUAL INTERACTIVE HUMAN ANATOMY Replace the current anatomy atlas that are 2d in nature by providing every student with a digital 3d cadaver of their own as a measure to supplement the current lack of cadaver donors, limiting the amount of interaction each student has with a real cadaver. Leverage on the spatial-relationship nature of anatomy learning with VR and kinesthetic interactions.
CHALLENGES AND OPPORTUNITIES. As virtual reality is still in its infancy, there is no clear rules and mental model to follow, which also means there is a lot of space for exploring new possibilities. And while motion sensing devices have been aplenty in the market for a while now, tracking consistency, ease of detection, and visual occlusion remains unstable in most, especially when its comes to small granular gestures. Combining these two emerging technology into a fluid and seamless experience is still quite a challenge that requires experimentation and testing, which we iterate repeatedly. We make mistakes, tonnes of them, but we invest minimal time in each iteration and then move onto a new solution swiftly. Anatomy learning in an interactive VR environment holds lots of possibilities which could not be realized otherwise. While we dived into anatomy, and extracted few essentials that could be achieved with VR after few iterations working closely with our medical school stakeholders, much remains to be explored regarding teaching methodologies and techniques for learning anatomy.
OUR APPROACH
IMMERSION IN ANATOMY. To better understand how best to learn anatomy, I started learning anatomy myself. Even tried an online anatomy course by HarvardX which was nothing short of sleep-inducing. But nothing beats having an anatomy professor as my own (and my team’s) private tutor! And yes, we had fun trying not to “merlion” looking at the cadavers.
anatomy atlas
mobile & desktop apps
learn through sketching
physical models
interactive flash puzzle
cadavers
OUR DISCOVERY
INFORMATION = KNOWLEDGE. Even with the myriad of tools and resources for learning anatomy, helping students build a 3-dimensional mental model of human anatomy remains the biggest hurdle in anatomy teaching. Most students resort to mnemonic memorization, only to forget everything when the new semester begins. Anatomy intertwined with physiology for understanding the purposes behind the structure, animated musculoskeletal movements in simplified mechanical joints, and helping students connect 2-dimensional radiology images to a mental model of 3-dimensional human anatomy, are some of the main features in VIHA. The emphasis is on visualization by providing enough flexibility for manipulation by the users, and providing a simplified version to help with conceptual understanding before moving onto the full granularity of human anatomy. START AS SIMPLE AS POSSIBLE
As learning anatomy can be overwhelming due to the sheer amount of information, one of the learning techniques we incorporated into the software is chunking. For example, the cross section feature allows the user to cut through any part of the body for a 2d view, a simplified view, where the arrangement the different functional systems are grouped into chunks with color coding. The understanding of how the functional systems are placed relative to each other can then be elaborated further into each individual part, as well as rolled across the whole human body as a general principle.
skin
skin sternocleidomastoid platysma trapezius sternohyoid omohyoid thyrohyoid
muscle infrahyoid
mylohyoid strap muscles atlas (c1) axis (c2) cervical verterbra (c3) skeleton
cervical verterbra (c4) cervical verterbra (c5) cervical verterbra (c6) cervical verterbra (c7) trachea thyroid cartiladge laryngopharynx esophagus
airway viscera digestive
common carotid artery internal jugular vein external jugular vein nerves
vessels
BRAINSTORM
AND THEN TEST THEM QUICKLY. A three-dimensional user interface in VR? How do we leverage on spatial memory? What mental model should VR borrow from? How do we create a sense of depth in VR? How to create a sense of presence in VR? What form of feedback can supplement the lack of tactility? Where do we draw the line between flexibility and usability? There is a fair amount of uncertainty and insufficient evidence to answer most questions. We often have to make assumptions during development, so sticking to agile practices is crucial.
STORYBOARD
MAKE IT FLOW. From sketches to vision video.
HIGH FIDELITY MOCK-UPS
WIREFRAMING. Going through each feature step-by-step sheds light on various unforeseen interaction issues, such as coherence between different sets of controls, switching between different features, a constant escape hatch etc. Wireframing serves as a visualization to concretize the ux and ui for communication purposes. More importantly, it unites the team’s envisioning of the software down to the details so we could build towards one coherent software.
PROTOTYPES
KEEP CALM AND ITERATE ON.
ITERATION .1
ITERATION .2
ITERATION .3
ITERATION .4
ITERATION .5
screen display
leap motion
Proof of concept. Anatomy puzzle game.
*Completed before I joined. Established the workflow for generating 3d models with anatomical accuracy for integration in Unity. Comprehensive 3d model of head and neck with complete list of objects with on off toggle.
Groundwork for oculus rift integration with leap motion. Laid down basic principles such as visual occlusion, camera movement, scaling of virtual objects and interactions/gestures, constraining interactions to within the range of leap motion, optimal font size and legible writing zone in the oculus rift, borrowing real world affordance for virtual interactions...
oculus rift
Divergent phase where we translated learning methodologies into software features and explored novel possibilities for learning anatomy in virtual reality, and then narrowed down to few prominent features and basic navigational interactions that still offers enough flexibility.
razer hydra
Stable release for future database integration and interoperability with other motion sensing devices. Models and animations for the upper limb region is almost complete. Comprehensive features offers more flexibility but usability is compromised in this iteration.
VIRTUAL INTERACTIVE SIMULATION ENVIRONMENT
Competing priorities versus time pressure, decision point despite incomplete evidences, organized chaos, and unpredictability. These are but any emergency department physician’s daily grind. These situational skills can only be trained through first hand experience. With VR and motion sensors, simulation of an ED can now be immersive and case parameters can be edited on the fly.
OUR DISCOVERY
KEEP YOUR EMOTIONS IN CHECK. The ER is an overwhelming place, and it is more than just sensorial. Yes, with various beeping machines, the patient’s biological reactions (or lack there of), people constantly talking and shouting, everyone crammed around the patient fighting for space, all happening at the same time, it is sensorial intensive. On top of the sensorial input, the physician has to keep track of what everyone on the team is doing, make mathematical calculations in his head, keep track of multiple statistics, make sense of all these information and make decisions under time pressure, then communicate clearly and concisely. But above all these, the emotional impact a life-and-death situation can command is perhaps most implicative, immediately as well as over time, affecting the staff’s psychology well-being. And I believe it is this emotive side to the emergency department environment that will be the measure of success for this simulation.
OUR PLAN
START SMALL THEN MULTIPLY. In the first iteration, everything was stripped down to basics. We focused on few specific tasks and the aim of this simulation was to use the right tools in the correct sequence. The plan is to eventually multiply this by all the ER procedures to create a open ended simulation, where the real test was in timely decision making rather than the technicalities and skills in execution as with most medical simulations available right now.
FIRST ITERATION
WALK THROUGH AS A START. Starting with basic procedures,namely setting up intravenous drip and airway intubation, we made a vision video combining minimal prototyping, bad acting, and kick-ass video editing, as the starting point for the project. On hindsight, the gestures we designed would have made the simulation impossible to complete. The mental model based on real world physical affordance simply does not translate into gestures, especially without tactile feedback and depth perception in VR right now. tools
tools on tray near patient’s arm
drip chamber + tubing
action
automated
feedback
grab (grab)
grab
saline bag
rotate
tourniquet
grab tie tourniquet green veins
alcohol swab vein dustbin
cannula vein
grab swipe twice (grab)
grab inject/puncture
hold down vein
zoom in view blood flashback at catheter hub
catheter
advance forward
lower angle stabilize with index finger
tourniquet
grab release tourniquet
needle
pull
dustbin
(grab)
iv tubing catheter hub
bandage over catheter
roller clamp
grab rotate
grab tap/swipe
roll
FOLLOW UP
FLOW DIAGRAM TO STATE DIAGRAM. After establishing the pipeline for building each procedure, we worked on the state diagram, to connect these procedures into a network of events. Depending on the physician’s actions, the patient reacts and enters a different state, leading to feedback in the form of his statistics (blood pressure, pulse rate, SpO2 etc.) and/or bodily movement on the rigged patient. All these procedures are timed, and when not fulfilled within the time line, will trigger a different set of events. Ultimately we aim for this state diagram to be content managed by the medical school themselves so they can set case parameters on the fly.
*This project has been put on hold till we completed the stable release for the anatomy project recently, and hence is still in its infancy.
MAY 2014 AT RETAIL ACCESS
SEASONAL PROMOTION SPACE
LAUNCH PAD AT SEPHORA ION Combining lab instruments as symbolic references to scientific research with philosophy’s products in fuss free yet information packed packaging reinforces its scientific heritage since inception by renowned dermatologist, Cristina Carlino.
Workshop area surrounded by petri dishes, chemical flasks, lab stools, blackboard, and dated overhead projector transports one into the nostalgic days of science lesson during schooling while subconsciously instilling scientific confidence behind the brand.
MAY 2014 AT RETAIL ACCESS
PRESS EVENT FOR NEW PRODUCT LAUNCH
FULL OF PROMISE AT BISTRO SOORI Philosophy’s new product launch, the “full of promise” range is a serum with two ends, one for day use and the other for night. The venue is translated into spaces that highlight the product’s duality of day and night, through lighting and manipulating the black-white composition, creating contrasting immersive spaces while still upholding philosophy’s brand image.
OCT 2013 AT RETAIL ACCESS
dimmed lighting for one-on-one interaction
OCT 2013 AT RETAIL ACCESS
bright open space for the offi
fficial event representing the “day”
“night” space with products under spotlight
cropped visuals focus on the sensual NOV 2013 AT RETAIL ACCESS
glorified giant factice
PUBLIC RELATIONS EVENT FOR NEW FRAGRANCE
SEE BY CHLOE AT THE SPACE HONG KONG Various visual and olfactory stimuli tease the visitors, creating a layered sensorial journey of uncovering, to build up anticipation before the official reveal of the new fragrance. Allowing self-discovery in a casual environment not only caters to the younger target audience of See by Chloe, but also builds on the free-spirited nature of the fragrance. While ensuring the setup is capable of piquing the visitors’ interest in the new fragrance, enough visual interest and optimal lighting throughout the venue is another key deliverable to encourage sharing of the event through social media by the visitors, mainly influential instagrammers and bloggers.
the trail of discovery
cozy ambiance with products casually scattered
entrance
visuals coupled with personal anecdotes by the spokesperson
large factice glorified under spotlight
NOV 2013 AT RETAIL ACCESS
end the event with a photo shoot for memories and PR of course
finally! a whiff of the perfume
brochures and little artefacts related to the making of product is introduced in a cozy ambiance over tea and dessert with the products scattered around casually
FLAGSHIP DEPARTMENTAL STORE
SK-II AT GMS AEON JAPAN GMS Aeon is a convenience stop usually targeting housewives, but SK-II is given the opportunity to propose a generic fixture to blend mid-range brands and prestige brands together seamlessly. One of the important features of the space is a good balance of shopping through self-selection for fast, independent shoppers versus personalized counseling and services for those looking for a prestigious experience with time to spare.
NOV 2013 AT RETAIL ACCESS
STOP ATTRACT ME.
APPROACH HOLD ME.
Brand logo, visual composition consistent with SK-II branding, disruptive color, clear promotion signage. Welcoming and open, inviting to touch and interaction.
NOV 2013 AT RETAIL ACCESS
HOLD GUIDE ME.
CLOSE DELIGHT ME.
Clear categorization of products for navigation, easy to update product information and pricing, promotion display, call to action, and cartons for quick pick and go.
Counseling by beauty consultant and personalized services and treatment procedures to pamper.
Accentuated with the right props, the space transports customers in time and space into a dream character, that of Johnnie Walker as an luxurious traveler in the yore. MAY 2013 AT RETAIL ACCESS
TRAVEL RETAIL
JOHNNY WALKER AT BRISBANE AIRPORT More than hard liquor, the Johnnie Walker Explorer’s Club Collection sells a desired lifestyle, and hence leverages heavily on interactivity and unique in-store experience.
CHAN MIN YUN
chanminyun@gmail.com