3D Hologram
-1-
KUOPING CHANG interaction designerr ckuoping@gmail.com +886 921 649 041
RESEARCH INTEREST
User Experience / Interaction Design / Augmented Reality / Human Computer Interaction / Game Design / Museum Interaction / Exhibition Design / Children Education WORK EXPERIENCE
Interaction Designer
Industrial Technology Research Institute, 2017-2019 Design front-end user interaction for a smart showcase research project supported by Ministry of Economic Affairs.
Visual Designer
National Taiwan Museum of Fine Arts, 2016-2017
Co-work with curators of NTMOFA to design educational exhibitions for physically challenged visitors.
EDUCATION
Scholarship Program
Design for Interaction, TU delft, NL, 2015-2016
Master
Ergonomics & Interaction Design, NCKU, Taiwan, 2016
Bachelor
Industrial Design, NCKU, Taiwan, 2013
IT SKILL
Interaction
Unity / Arduino / Processing / C# GUI
UX Design
Axure / Principle / SPSS
Visual Design
Adobe Photoshop / Illustrator / Indesign / After Effects
Programming
Flash/ javascript / actionscript
Product Design
Solidworks / Keyshot
ACHIEVEMENTS
Best Research Award
The Research of Directive Information Fusion Technology, ITRI
Patent
A Method for Information Display to Mulit-user A Design of Head Mounted Device using PMOLED A Visual Ability Training Device A Design of Scale with assitant instruction integrated
Publications
Hoe, Z. Y., Lee, I. J., Chen, C. H., & Chang, K. P. (2017). Using an augmented reality-based training system to promote spatial visualization ability for the elderly. Universal Access in the Information Society, 1-16. Lee, I. J., Chen, C. H., & Chang, K. P. (2016). Augmented reality technology combined with three-dimensional holography to train the mental rotation ability of older adults. Computers in Human Behavior, 65, 488-500.
Chang, K. P., & Chen, C. H. (2015, August). Design of the Augmented Reality Based Training System to Promote Spatial Visualization Ability for Older Adults. In International Conference on Virtual, Augmented and Mixed Reality (pp. 3-12). Springer International Publishing. Chang, K. P., Huang, Y. W., Hsueh, S. Y., Chen, Y. T., Huang, S. N., Chen, C. H., & Chien, S. F. (2014, October). Hidden lion: a location based app game of sword lion searching. In Proceedings of the first ACM SIGCHI annual symposium on Computer-human interaction in play (pp. 323-326). ACM.
Smart Showcase
-4-
AMOLED Display
Smart Showcase
Interactive Transparent Display System using Directive Information Fusion Technology In response to the demands of future smart museum, we have established an interactive transparent display system called smart showcase. The smart showcase is based on its flexible transparent AMOLED display technology while integrating eye tracking with directive object recognition and information fusion technologies. Once a user touches the window, the window can show the exhibitrelated information at the right location on the display by identifying the exhibit that the user is looking at and the position of the user's eyes. This innovative showcase creates a whole new interactive and intuitive experience for users at museum. It is believed that the system can also be applied for entertainment, retail, and medical applications and brings more advantages to them. Research Project supported by Ministry of Economic Affairs, Accomplished at Industrial Technology Research Institute(ITRI).
-5-
Smart Showcase
Museum Showcases
Free-Standing Case
In most museums, the typical way to display exhibits is to put them in showcases with transparent glass or plastic. However, although the showcases function as good protection for the exhibits, the poor design of current showcases indirectly intervenes the interaction between visitors and the exhibits, resulting in much negative visiting experience to users. For instance, if a visitor wants to get the description of an exhibit while observing it, normally he needs to check a sign card attached beside the showcase, which distracts his attention from the exhibit . This issue is even more serious in large showcases. Large showcases are used to display plentiful of exhibits. However, it will cause a visitor to feel difficult to distinguish and identify a specific exhibit while reading the description. In these situations, not only the visitor’s experience is negatively influenced, but the information of the exhibits is also hardly conveyed to the visitor.
Wall Case
Display Case System -6-
AMOLED Display to conduct field studies at museums to see how current exhibits are demonstrated and how visitors interact with the exhibits
to define personas to focus on the unmet needs that should be solved at museum interaction and make service blueprint to organize a better user flow at museum
to study the resolution, the color, the lighting and the size of transparent display in order to offer a suitable graphical user interface proposal for the smart showcase design
to te st u s e r p refe re n c e o n t h e interaction, the reaction time, the accuracy and the visul effect of the smart showcase
to implement the GUI and user interaction proposal by programming in C# language
integrate with the hardware modules, software functions and GUI content into a well-structured system.
-7-
Smart Showcase
Design of Exhibition Signs The design of exhibition signs are pretty poor at National Museum of Natural Science. For example, the signs are directly attched on the showcase window, seriously blocking visitors' eyesight toward the exhibits. What' s worse, visitors might feel hard to identify certian exhibits since too many items are there in one showcase. In this way, they are unable to check the exhibit and the information of the exhibit simultaneously.
Visitor Interaction Museum is a good place for visitors to come with their families. When a visitor is checking an exhibit through showcase window and wants to explain to his families, he is likely to touch the window to locate the exhibit for his families. This behavior provides a very great insight for us to realize a fact that the showcase window might be a good platform to communicate bewteen users and exhibits.
Museum Guided Tour Museum offers many professional guided tours in order to create more interactive experiece for its visitors. Usually, one museum guide needs to service up to 100 to 120 visitors in a day, which is very exhausting for the guide. Also, museum guides are not provided with useful guiding tools for them, so sometimes it is diffult to present a dynamic tour to visitors.
Contextual
Inquiry
at
Science
Museum -8-
AMOLED Display
We conclude three m a i n u s e r g ro u p s as the persona o f m u s e u m : families(parents and their kids), teachers and students(primary education) and museum guides. After defining the diffculities they are facing at museum, we list three functions that would be required in order to fulfill their needs: real time informatino of exhibits, interactive visiting experience and diverse content.
-9-
Smart Showcase
Service Blueprint
new museum experience Based on the user needs defined in persona, we create three innovative museum experiences. "Directive Touch Technology" can be best applied to the user group: parents and their kids. "Client-Content Management" is designed for the user group: teachers and students. "Information Creation" is the service
that the museum guides need the most. These three experiences are analyzed in details by using service blueprint.
-10-
AMOLED Display
-11-
Smart Showcase
wireframe of
Directive Touch Technology
-12-
AMOLED Display
System Integration In order to finish the service of directive touch technology, a well-organized showcase system needs to be done by integrating the following four components. Hardware means the physical modules in the showcase. Software includes AI and detection functions. Form means conventional product design. Interface means UX research and GUI design.
-13-
Smart Showcase
AMOLED (active-matrix organic light-emitting diode) is a display device technology. AMOLED can illuminate without the support of other light sourses(the mechanism that LCD requires). A n o t h e r st re n gt h o f AMOLED more power efficiency and less power consumption. Thus, This is absolutely an advantageous display technology nowadays.
AMOLED
Transparent Dispay Illumination
Color
To test the illumination of AMOLED display in a bright interior space.
To see how RGB colors are presented in AMOLED display. Black in normal display will be transparent in AMOLED display. Resolution
To test the illumination of AMOLED display in a dark interior space.
With only 50 ppi, it is important to decide which size and style of graphics can best fit the low resolution of AMOLED.
T h e g r a p h i c s te n d to be more blue in dark surroundings.
Transparency
To see the degrees to h o w i n fo r m a t i o n w i l l be interferenced by the objects placed behind.
-14-
AMOLED Display Touch Module : Airbar
Airbar is an IR touch sensor to get touchscreen on most display or notebook. It can detect many gestures like point, zoom, swipe. When plugged in a device, Airbar sends IR and receives touch signal if IR is interrupted. Hence, it is possible for Airbar to locate the touch point from users. The effective area of Airbar is 345mm*195mm, which is suitable for the use of 17"AMOLED display (345mm*260mm). Thus, Airbar is a good choice in this project.
Camera Module : OV-580
OV-580 is a stereoscopic camera with FOV(D/H/V) 109ยบ 88ยบ 72ยบ. The video resoltuion and the frame rate of OV-580 is 1920*1080 (1280*720)and 30 fps(60 fps). In the system, two OV580 are used. One is used for capturing the image of exhibits, and the other is used for capturing the users eyes. OV-580 is a great camera module for the use of object recognition and depth detection.
Other Hardware Modules -15-
Smart Showcase
The Structure of Software Functions
a proper position on the display. In doing so, the user will always see the information on his eyesight to the object no matter how he moves. As shown above, A user from left side wants to realize more about the butterfly, and another user from right side wants to get the information of the dragonfly. After they both touch the display, the system is able to respond correct information to the corresponding users because the users’ eye positions are also calculated. Hence, although the positions of their touch point are close, due to the difference of the eye positions, the information will also be different and show to its correspondent target users.
Two important software functions are adopted in the system using directive touch technology: object recognition and eye tracking. The system keeps recognizing objects behind the AMOLED display and calculating their locations. After their locations are identified, the system will create initial ROI areas of the objects by the process of coordinate transformation. Meanwhile, the system is also detecting users in front of the AMOLED display, and calculating the eye position of a user. The calculated eye position will be used as a variable to calibrate the ROI areas of the objects. Next, the system matches the user’s touch point on the display and the accurate ROI areas of the objects to show the information of the objects at
-16-
AMOLED Display
GUI
pixel art
Pixel art is adopted as the main style of UI in AMOLED Display due to the compromise of low resolution. UI is presented in the form of animation, making it more attractive to visitors.
22"LCD Testing The location and the size of UI information are tested by LCD display before it will be presented in AMOLED display.
-17-
Smart Showcase
Phase I: Slide Show
Phase II: Eye Tracking
Phase III: Directive Touch
Once the smart showcase starts, the system will first step into slide show mode. In this mode, an a n i m a t i o n d i s p l ay i n g forest scenarios will be played continously until mode-changing buttion is clicked.
In Eye-Tracking mode, the system will detect users eyes when they are looking at exhibits. Once their eyes positions are located, the exhibitrelated information will show on the display.
In Directive Touch mode, the information will show on the display after the user "touches" the display. Apparently, the location o f t h e i n fo r m a t i o n i s integrated with user's eye position and position of touch point.
-18-
AMOLED Display
The final version of smart showcase was demonstrated at Taipei Nangang Exhibition Hall from Aug 29th to 31th, 2018. As shown above, the smart showcase system can allow two users to use simultaneously. Information in yellow means the information for the right user, and red means
the left user. In order to make all exhibit's detection accurate for both users, the exhibits are arranged in a triangle area which is the overlap of both users FOV region.
-19-
Smart Showcase
Evaluation of Smart Showcase 32 Visitors were invited to conduct the evaluation of smart showcase. They were asked to finish a subjective questionnaire after they had tried the smart showcase. The questionnaire was divided into 3 sections: overall review of the system, guiding function of the system and system stability. Each section consisted of 4 to 7 items, and each item was a five-scale rating scheme ranging from “very disagree (1)”to “very agree (5)”
Basically, the visitors were satisfied with the guiding function of the system. They could directly get the information of the exhibits on the display. Moreover, since the system was able to detect a visitor’s eye position, the information was always showing at a proper position, which was a strength they agreed the most. However, some visitors replied it was hard for th em to focus on both the information on the display and the exhibits behind the display.
A visitor is testing the function of smart showcase
-20-
AMOLED Display
Smart Showcase
Directive Touch Technology Service
Compared with the other three typical tours in musuems, Smart Showcase with Directive Touch Technology service brings more convenience to both visitors and guides. Without the need to take any portable device, visitors are able to get the exhibit-related information right from the showcase itself. It is expected that smart showcase can increase intelligent and interactive experience for visitors in museums.
-21-
For the video: https://goo.gl/b1Bmb9
Goodlock
3D Hologram Based Mental Rotation Training Tool
-22-
3D Hologram
Goodlock
3D Hologram Based Mental Rotation Training Tool for the Elderly
Master Thesis Design, Accomplished in the 2nd year of Graduate School of Industrial Design, NCKU
Mental rotation, one of cognition abilities, is related with a sense of spatial direction and perception. However, the decline of this ability results in poor spatial awareness on the elderly, further influencing their lives. Goodlock, which applies 3D hologram technology, is designed as a mental rotation training tool for the elderly. The use of augmented reality(AR) and leap motion of Goodlock could enhance the training effect and trainers' interests and thus improve the sense of spatial direction of the elderly. Also, compared with conventional mental rotation stimuli which are abstract, the training content of Goodlock, which is inserting keys into different lock holes, is more interesting and easily understandable. Goodlock is implemented in public spaces such as senior citizen centers to provide another training option rather than physical fitness for elderly to use and play.
-23-
Goodlock
Decline of
Mental Rotation Ability
Mental Rotation Ability
a cognition ability of spatial psychology Mental rotation, one of spatial cognition abilities, is defined as an ability to mentally rotate a 2D or 3D object in an imaginative space. In a more professional term, it refers to the mental manipulation of spatial information to determine how a given spatial configuration appears if they are being transformed[1]. Human beings are endowed with this ability to help orient themselves in physical space and navigate from place to place. For instance, map reading, way finding and direction judgement are all involved with the use of mental rotation ability [2][3][4]. Hence, mental rotation ability has a significant influence in one's daily life.
mentally rotate an object in space
-24-
3D Hologram lateral prefrontal cortex
However, mental rotation ability, just as most cognition abilities, will also decline with human's increasing age. This fact has been corroborated by numerous neuroscience studies. Under computerized tomography, lateral prefrontal cortex and hippocampus volume, which have a stake in cognition perception and the ability of processing spatial information, reveal a significant decreasing pattern on older adults[5]. Also, older adults tend to show a low white matter integrity in their brain, suggesting that they have poor performance on spatial memory and attention [6].
hippocampus
Therefore, it might be tough for the elderly to mentally comprehend what the objects will be after they are spatially transformed. Since mental rotation ability has so much to do with one's daily life, the decline of it will result in the poor sense of direction and environmental psychology, causing the elderly people to have spatial problems and low spatial awareness in real life [7][8][9].
-25-
Goodlock
motivation one:
Weaknesses of Current Training In fact, the decline of mental rotation can be delayed by undergoing considerable mental rotation training[10]. Usually conducted in 2D presentation modes, this training requires participants to judge whether two stimuli are the same or mirrored [11], and then write down their answers on paper. Yet, this test-like training might be inappropriate for the elderly due to the following reasons:
cognitive load spatial training in 2D m e d u i m g e n e ra t e s mental load.
interaction
there is no much sensory interaction involved in the training.
paper test
this training is testing rather than gaming.
boringness
the training, similar to testing, makes trainers boring.
-26-
3D Hologram
motivation two:
Brain Fitness
No in Senior Citizen Center Although many community and senior citizen centers have been established and serve as well-organized places for the local elderly to socialize and exercise, few cognitive training facilities are provided. The physical fitness training is still the main service in senior citizen centers; for instance, treadmills, stationary bikes and steppers are the most common equipments. Since cognition and mental rotation
ability are also important to the life of the elderly, to design and implement a mental rotation training system in the senior citizen centers is essential and innovative. In this light, the elderly will have more opportunities to take cognition training in usual time, and gradually prevent the cognitive and mental rotation from declining in the future.
-27-
Goodlock
Augmented Reality Effect 3D hologram is a projection system which displays virtual images inside its pyramid-like glass structure. The virtual images produced by reflection and overlapped in real space are vivid and dynamic as if they completely exist in real world, creating an augmented reality(AR) visual effect. The AR effect presents correct spatial information and promotes a concept of visualization which is important to mental rotation ability[12]. Moreover, AR can effectively reduce the cognitive load and help facilitate trainers' learning interests in the training[13].
motivation three:
Strengths
of 3D Hologram
-28-
3D Hologram
Direct Manipulation In some 3D hologram projects, viewers are allowed to manipulate the virutal images through the use of a gesture sensor-Leap Motion. The gesture control of direct manipulation easily engages the viewers during interaction and promotes a great immersion for the viewers. Also, the use of gesture is proved to facilitate mental rotation process since trainers can easily predict how the mental presentations look after they are rotated and helps reduce the cognitive load produced from mental rotation process [14][15].
360 Perspective With the strength of 360 degree perspective of 3D hologram, viewers can observe the images from any angle rather than from only the front side of it. This advantage allows not only the front viewers but everyone to enjoy the visual effect. Therefore, 3D hologram will be a good installation settled in public spaces like senior citizen centers.
-29-
Goodlock
Goal of Goodlock x 3D Hologram x manipulation x gamification x senior center
The goal is to integrate 3D hologram, augmented reality, gamification and manipulation into one system which will be implemented in senior centers as a mental rotaiton training device. In this way, senior citizens have other training options rather than physical fitness and thus the decline of spatial cogitition can be delayed successfully.
-30-
3D Hologram The first part of the research was to analyze gestures older adults used when confronting mental rotation te st s . T h e ge st u re s were examined by guessability theory [16] and the most representative ones were selected. These representative gestures were applied in the final system as the input for direct manipulations.
-31-
Goodlock
The second part of the research was to generate new training content based on the gestures defined in the first part of the research. A design workshop was held and five interaction designers were invited in an effort to contribute the ideas of training games. Final outcome, which was scored as the highest by the designers, was called Goodlock, which was a set of key and lock. Players rotate the key in a correct angle to make it pass through the hole on the lock. The key-lock components in Goodlock are different from t ra d i t i o n a l m e n ta l ro ta i o n stimuli and thus can be easily apprehended by older adults.
-32-
3D Hologram
Solidworks was used to construct the form of the key and lock, and 3ds MAX was used to attach textures and color to the models. To differentiate the levels by difficulties, three types of key: iron keys, bronze keys and gold keys were made. The more expensive material the key was, the more complicated the shape on the key would be.
Gesture control was done by the use of leap motion. Three types of control were implemented: Rotation, which was to rotate one's palm around his wrist; Translation, which was to move one's hand with hand open; Standby, which was to make a fist to stop any motion of the key.
-33-
Goodlock
The game was implemented in Unity3D, a game development engine by programming in C#. The game rule was to make the key pass through the correct hole to touch the first collider and further touch the second collider to move into next level.
The split screen was made in order to create 360 degree perspectives in the pyramid hologram. It was achieved by creating cameras in one scene. So viewers were able to see three perspectives simultaneously in the screen canvas.
-34-
3D Hologram
The biggest challenge for hardware implementation was to create the pyramid form. The requirement of a perfect pyramid is to make each side and plane lie at an angle of 45 degree. This depended on accurate geometry calculation and prototype testing. After the form was done, cover th e pyrami d wi th s em i transparent film to improve the reflection. The figure shown below is the steps of creating the standard threesided pyramid.
-35-
Goodlock
Final refinement of the device was done by fortifying its intensity, polishing the pyramid, and painting the supportive structure in black. Painting the structure in black was a way to prevent itself from being reflected inside the pyramid. The reflection of unnecessary images rather than the game would disturb the view and distract the viewers. By doing the refinement, the device looked more in a unity; also, this could improve the quality of reflection and thus made the augmented images more vivid and real.
A supportive structure made from wood and stainless steel was used as a platform to demonstrate the pyramid and also for the convenience of transporting.
-36-
3D Hologram
One strength of 3D hologram is the augmented effect, which, by the study of cognition science, is beneficial to the understanding of spatial relationship. Another strength is 360 degree perspectives. It makes the game playing not limit to only the players right in front of the device but open to everyone around the device. The game of Goodlock is more appealing to older adults since it is tranformed with a more contextual and interesting rules other than traditional mental rotation stimuli.
3D hologram
Mental Rotation
Training System
-37-
Goodlock
-38-
3D Hologram
Game Scenes Scene 1-2 Opening scene of Goodlock. Three types of key (iron, bronze and gold) indicting three levels (novice, medium and advance) would be presented in this scene to inform players of level difference. In addition, a practice session was included in this scene allowing players to try gesture control.
Scene 3-8 Different scenes in iron key level. First, players needed to reply the hole on the lock they think matched the shape of the key (scene 4). Then, they were able to insert the key into the hole they answered. If the key passed through the lock, a green correct circle would appear on the lock; o t h e r w i s e t h e key wo u l d b e blocked by the lock until the player rotated the key to the correct angle to pass it through the lock.
Scene 9-10 The medium level (bronze key) and the advance level (gold key). Basically the game rules were the same in these two levels, but the only difference was the complexity the key was. A more complex key meant more corners on the shape, so playsers might take more effort distinguishing the same shape or the mirrored shape.
-39-
Goodlock
Evaluation
pretest & posttest design To test the training effect and older adults' preference on Goodlock, an evaluation of pretestposttest design (A-B-A design) was set up. In a total of 14 older adults with average age of 67.21 were recrutied. During the intervention phase, they received Goodlock training 12 times within 1.5 months. During the pretest and posttest phase, they were given mental rotation paper tests for 8 times. In each time, the error rate was recorded so the result of training effect would be clear if we compared the mean error rate of pretest and posttest. In the setting, participants sit in front of the 3D hologram to play with Goodlock. A researcher also sit beside the participant to observe the whole playing process.
-40-
3D Hologram
Outcome The paper test consisted of 2D and 3D mental rotation tasks. The error rates of the two types of tasks were recorded individually. As seen from the figure below, the mean error rate of 2D mental rotation tasks decreased significantly in the posttest. The error rate of most 3D tasks also
decreased. The improvement of their performance showed a remarkable training effect of Goodlock. In addition, the participants enjoyed the play of Goodlock and replied a postivie feedbacl toward this training device.
-41-
Goodlock
Vision in the future Create a Community Network for the Elderly to do Brain Fitness Nowadays, senior centers play an important role for older adults. They serve as a friendly place for older adults to study, exercise and socialize. The installation of Goodlock in senior centers will enrich the value of senior centers because it is an innovative cognition training system which utilizes 3D hologram technology, augmented reality and motion sensor. It is expected that more and more centers can install Goodlock so that older adults have a new training option of "brain fitness" other than typical physical fitness.
More importantly, one strength of 3D hologram is 360 degree perspectives, which allows not only one player but many audience to get involved in the training session. They are more likely to discuss the game, share ideas, interact with other people while training, increasing the socialization function of senior citizen centers.
-42-
3D Hologram
Reference [1] Salthouse, T. A., Babcock, R. L., Skovronek, E., Mitchell, D. R., & Palmon, R. (1990). Age and experience effects in spatial visualization. Developmental Psychology, 26(1), 128. [2] Lawton, C. A. (1994). Gender differences in way-finding strategies: Relationship to spatial ability and spatial anxiety. Sex roles, 30(11-12), 765-779. [3] Parsons, T. D., Courtney, C. G., Dawson, M. E., Rizzo, A. A., & Arizmendi, B. J. (2013). Visuospatial processing and learning effects in virtual reality based mental rotation and navigational tasks Engineering Psychology and Cognitive Ergonomics. Understanding Human Cognition (pp. 75-83): Springer. [4] Malinowski, J. C. (2001). Mental rotation and real-world wayfinding. Perceptual and Motor Skills(0031-5125 (Print)). [5] Bird, C. M., & Burgess, N. (2008). The hippocampus and memory: insights from spatial processing. Nat Rev Neurosci, 9(3), 182-194. doi: 10.1038/nrn2335 [6] Hedden, T., & Gabrieli, J. D. E. (2004). Insights into the ageing mind: a view from cognitive neuroscience. Nat Rev Neurosci, 5(2), 87-96. [7] Beni, R. d., Pazzaglia, F., & Gardini, S. (2006). The Role of Mental Rotation and Age in Spatial Perspective-Taking Tasks: When Age does not Impair Perspective-Taking Performance. Applied Cognitive Psychology, 807-821. [8] Jansen, P., & Heil, M. (2010). Gender differences in mental rotation across adulthood. Exp Aging Res, 36(1), 94-104. doi: 10.1080/03610730903422762 [9] Lee, S., & Kline, R. (2011). Wayfinding Study in Virtual Environments: The Elderly vs. the Younger-aged Groups. International Journal of Architectural Research, 5(2), 63-76. [10] Hertzog, C., Kramer, A. F., Wilson, R. S., & Lindenberger, U. (2008). Enrichment effects on adult cognitive development: Can the functional capacity of older adults be preserved and enhanced? Psychological Science in the Public Interest, 9(1), 1-65. [11] Shepard, R. N., & Metzler, J. (1971). Mental Rotation of Three-Dimensional Objects. Science, 171, pp. 701-703. [12] Kaufmann, H., & Schmalstieg, D. (2003). Mathematics And Geometry Education With Collaborative Augmented Reality. Computers & Graphics, 37(3), 339-345. [13] Shelton, B. E., & Hedley, N. R. (2004). Exploring a cognitive basis for learning spatial relationships with augmented reality. Technology Instruction Cognition and Learning, 1, 323-357. [14] Chu, M., & Kita, S. (2011). The nature of gestures' beneficial role in spatial problem solving. Journal of Experimental Psychology(1939-2222 (Electronic)). [15] Chu, M., & Kita, S. (2012). The role of spontaneous gestures in spatial problem solving Gesture and Sign Language in Human-Computer Interaction and Embodied Communication (pp. 57-68): Springer. [16] Wobbrock, J. O., Morris, M. R., & Wilson, A. D. (2009). User-defined gestures for surface computing. Paper presented at the Proceedings of the SIGCHI Conference on Human Factors in Computing Systems.
-43-
For the video: https://goo.gl/8Hpky1
ARotation
-44-
Mobile AR
ARotation
Augmented Reality Based Spatial Visualization Training Game for the Elderly ARotation, which is an APP version of Goodlock, is also a spatial visualization training tool designed for the elderly. Similar to mental rotation ability, spatial visualization is one of spatial cognition abilities which will decline with human's increasing age, resulting in much inconvenience to elderly's life. However, compared with Goodlock which is a fixed installation in public centers, ARotation is implemented as an APP which allows the elderly to train their visualization ability through tablets and smartphones, increasing the portability and accessibility of training. Therefore, it will be easier and more convenient for the elderly to do cognition training at all times. Project of Interface Design, Accomplished in the 1st year of Graduate School of Industrial Design, NCKU
-45-
ARotation
Spatial Visualization & Mental Rotation what's between these two spatial cognition abilities Spatial visualization ability is the ability to mentally m a n i p u l ate 2 - d i m e n s i o n a l , 3-dimensional and 4-dimensional figures(Lohman, 1996). It is typically measured with simple cognitive tests. According to Lohman's definition(Lohman, 1988), these tests are mental rotation, paper folding, block d e s i g n , p a p e r fo r m b o a rd , surface development and
-46-
mechanical principle. In other words, mental rotation is one of spatial visualization abilities which centers on rotation of an object in an imaginative space(McGee, 1979). They share the similar pattern of cognitive operation and also the similar function of spatial concepts. Hence, these two abilities relate to each other.
Mobile AR
Goal of ARotation
delay the decline of visualization ability
1.Elderly Friendly to provide a visualization training option for the elderly
2.Portability to enhance the accessbility and portability of training. So users can do the training without the restriction of specific places.
3.AR technology to apply interactive technology to enhance the training engagement and the digital interaction
4.Manual Control to activate motor operation in human's brain which is beneficial to visualization. Also, this is easier for the elderly to play and use.
-47-
ARotation
Augmented Reality
AR is the real environment whose elements are augmented by computer-generated sensory input. Viewers can see the virtual figures through the devices that are equipped with camera. When the camera is targeting toward marker patterns, the virtual figures will appear from the patterns in the screen of the device.
SPATIAL INFORMATION AR provides correct spatial information and faciliates spatial cognition. People can better realize spatial relationship in AR rather than in 2D platforms.
REDUCE COGNITIVE LOAD AR increases germane load and reduces extraneous load. By training in AR environment, AR can help users reduce intrinsic load; therefore, the overall load is down (Shelton & Hedley, 2004).
-48-
ENHANCE IMMERSION With the features of immediate interaction and digital manipulation with the virtual figures, AR can enhance users' engagement and interests when they are in the training.
Mobile AR
Design Process In order to create user centered game and controller, an experiment was conducted to collect older adults' behaviors when they were handling visualization problems. Next, transform these behaviors into the design of controller and games.
-49-
ARotation
Idea Generation
-50-
Mobile AR
Implementation create Augmented Reality by coding in C# in Unity 3D
-51-
ARotation
The game was implemented in Unity3D and tested in laptop and APP version.
The form and the arrangement of the controller were also improved in order to achieve fluent use.
Evaluation Process
Older adults were recruited to test the usability of ARotation.
This section summarizes the process of implementing the software of ARotation and reveals how the improvement of the game and the manual controller were done. Last but not the least, evaluate the training effect of ARotation by single subject research.
-52-
Mobile AR
There were 3 phases in the evaluation. In the 1st phase, 8 elderly participants were tested their baseline performace of spatial visualization by taking paper tests. In the 2nd phase, they received ARotation. In the 3rd phase, they were taken the paper tests again to see if they made an improvement or not.
-53-
ARotation
3D Printing model making A reflection device was made to redirect the camera route of the tablet, and a stand was provided to keep the tablet in vertical angle. These two items were done by 3D printing, which saved quite a lot of cost and time.
The material used to print was ABS. The time for implementation took around 6 hours.
A reflection device whose angle was adjustable could be equipped on the tablet.
playing / training process User rotates the cube marker to control the virtual figure that is above the virtual floor.
Next, press the round marker which presents a red virtual button in the screen, and the figure will fall down
-54-
By reflection, the camera route was redirected toward the ground so the markers could be scanned.
Open Sources
Mobile AR
ARotation will be open sources accessible through cloud sharing. Older adults and their families can download the marker pictures from ARotation websites and print them out. The original 3D file of iPad stand is also available. Therefore, older adults can directly start training at all times!
Reference [1] Lohman, David F. "Spatial ability and g." Human abilities: Their nature and measurement 97 (1996): 116. [2] Lohman, D. F. (1988). Spatial abilities as traits, processes, and knowledge. [3] McGee, M. G. (1979). Human spatial abilities: Psychometric studies and environmental, genetic, hormonal, and neurological influences. Psychological bulletin, 86(5), 889. [4] Shelton, B. E., & Hedley, N. R. (2004). Exploring a cognitive basis for learning spatial relationships with augmented reality. Technology Instruction Cognition and Learning, 1, 323-357.
-55-
For the video: https://goo.gl/K9a583
Hidden Lion
-56-
Location Based APP
Hidden Lion An Location Based APP Game of Sword Lion Searching
Hidden Lion is a Location Based Service(LBS) APP game which is related to sword lion culture in Anping, Taiwan. We develop a storyline and theme that has a connection with the local culture. While playing the interactive games at each sword lion sites with this APP, visitors can find sword lions and experience the background story of each sword lion. Furthermore, Hidden Lion creates a service system in Anping. From the support from Anping district office, more and more visitors can come here and play the Hidden Lion. After completing all the interactive games, the visitors will receive a coupon of sword lion model coloring from Sword Lion School as a reward, which gradually forms a business cycle and culture connection. Project of Human Computer Interaction, Accomplished in the 1st year of Graduate School of Industrial Design, NCKU
-57-
Hidden Lion
Sword Lion Statue
Tainan is a place full of traditional culture and atmosphere in Taiwan, which is also the reason why many visitors are attracted to come here and travel. Among all activities, the most special traveling activity in Tainan is Sword Lion Searching in Anping. In the past, Anping was a crowded region near the sea; therefore, local people built sword lion statues in front of their houses to keep bad evils away. Sword lion statues are the stone plates installed on the wall. The appearance of sword lions is different from each other, representing different protecting abilities and stories. Nowadays these sword lion statues decay with time and weather and remain few intact in Anping.
Different sword lions have different protecting capabilities. However, sword lions decay with time and remain only few intact nowadays.
-58-
Location Based APP
Sword Lion Searching
discover remaining sword lions in Anping Recently, with the promotion of Anping District Office, many travelers are attracted to come here to find out remaining sword lions and experience the cultural stories of these protectors.
Activity Theory In order to analyze sword lion searching more specifically, a research method: Activity Theory is adopted which is clear for us to collect data and details of sword lion searching.
According to the interview and observation, most travelers like to share pictures and experience on social media and their personal blogs.
-59-
Hidden Lion
Problems of sword lion searching
However, current sword lion searching has some problems. First, exisitng searching maps have too many versions, which may confuse visitors and make them hard to locate lions. Second, sword lion searching results in disturbance to the locals. Third, visitors do not realize the cultural story of sword lions.
Vision of our project
We are devoted to creating Hidden Lion which is a LBS APP helpful for sword lion searching that includes interactive games to make visitors learn the sword lion story in a more interesting way.
-60-
Location Based APP
SERVICE SEEKER
Mrs. Hsueh loves to travel to other places in the weekends with her families. She likes Tainan for its cultural value and fascinating history. She always has fun with her son during each traveling not only for teaching the local culture but for improving their relationship.
-61-
Hidden Lion
Dramatic Elements
save sword lions from the evil's invasion in Anping
In order to make the APP similar to a game, we create the game story. Since Anping is an island in the past and sword lions fend off evils, a plot “Anping is attacked by evil spirits� is created. Powerful lions are imprisoned, so players need to locate and play interactive games to free them. Sword lions will be more powerful throughout the game by defeating evils and bringing peace in Anping again.
Formal Elements
We follow the formal elements to plan the game procedures. The detailed information and functions of Hidden Lion are listed through 4W1H questions. Hidden Lion is suitable for everyone to use. The basic reqirement in each procedure is shown in the left table.
Dramatic Arc
-62-
Location Based APP
APPLE iBeacon
iBeacon is a location sensor to detect the distance between iphone and iBeacon itself. When the players touch the sensing area, an interactive game will be on.
Location Based APP
Implementation GOOGLE GPS
Our system uses the google map service as the main guide view and connects to the Parse database for getting the annotation data.
ELECTRONIC COMPASS
We set up compass functions in the APP so that user's walking direction is detected.
MULTITOUCH
Mapping function is used in X i Pe n g Pe n g ' s m i s s i o n , s o multitouch coding is required.
IMAGE RECOGNITION
P l ay e rs t u r n o n c a m e ra t o recognize sword lions so that the game will get started.
-63-
Hidden Lion
Island of Sword Lion
A. empire city The most powerful sword lions are mainly located in this area.
B. military camp In the past, soldiers and general lived here. Many historic sites are also in this zone.
C. harbor zone This area is near the canal and Anping harbor.
D. ghost town -64-
This zone is named Ghost Town due to many tombs located here.
Location Based APP
Empire City
Basic Function of the APP Since the whole area is too large, we only implement the empire city area in the APP. Here we show the detailed map view of empire city. Icons on the right side are the basic functions users can use to help them in searching process such as the location of sword lion, tourist spots, other players, restaurants, mayor 's home and temples.
sword lion site
other players
the location of sword lions in empire city area
the location of other p l aye rs w h o a l s o have Hidden Lion A P P. P l a y e r s c a n invite them to join the searching
tourist spots
restaurant
the location of famous tourist spots in Anping
Anping is famous for its delicious snack. The location of these restaurants is also noted
mayor's home
temple
mayor serves as a searching instrutor so players can ask him for help
temple is also where user can ask for the searching help
-65-
Hidden Lion
Once upon a time, sword lions lived peacefully in the sword lion island.
Suddenly, a devil invaded and destroyed the island. Some lions were even imprisoned.
Luckily, one little lion escaped from the attack safely.
He was devoted to bringing peace back again and saved his lion friends.
First, the lion introduces the background story and asks players for help.
Then, the player selects t h e re g i o n w h e re h e wants to challenge.
Press the camera icon, and the interface shows the introduction of the tourist spot.
Press the noodle icon, and the interface shows the introduction of the famous restaurant.
P re s s t h e swo rd l i o n icon, and the interface shows the location and introduction of the lion.
If the player discovers any sword lion in empire city, he can press the catch button.
Once the button is pressed, the player is able to scan the lion discovered at the spot.
For example, the player can target the Yu Jian Er sword lion to start the game.
-66-
Location Based APP
The story of Yu Jian Er: once upon a time, he was resting.
H o we ve r, a g ro u p o f ghosts was approaching.
They attacked the resting Yu Jian Er sword lion.
Yu Jian Er was defeated and even got imprisoned.
Fortunately, Yu Jian Er left a magic star and awaited the player to save him.
Next, the game introduces how to finish Yu Jian Er mission.
The player should run the route of the star to release Yu Jian Er.
Once the mission is completed succeddfully, a congratulations page will show.
Bagua & Magic Star Bagua models with each iBeacon inside are placed individually at five corners in a yard in front of Yu Jian Er statue. The player completes the mission by running the star route to touch the bagua models.
-67-
Hidden Lion
T h e sto r y o f X i Pe n g Peng: Once upon a time, Xi Peng Peng was strolling along a street.
Suddenly, weather got bad and a water devil appeared.
Xi Peng Peng decided to fight against the devil.
However, the devil was too strong and Xi Peng Peng was hurt greatly.
Unfortunately, Xi Peng Peng was imprisoned.
First, the game will d e m o n s t ra t e h o w t o finish the mission.
T h e p l aye r m a p s t h e screen back and forth to remove bubbles.
Once the mission is completed succeddfully, a congratulations page will show.
Mapping on the screen The player needs to map the screen back and forth to finish the mission: remove the bubbles and clear the screen.
-68-
Location Based APP
In the achievement page, it shows how many missions the player has accomplished.
If all missions in one region are completed, t h e p l ay e r r e c e i v e a coupon.
Lion Models Mock Up If the player received a coupon, he can go to the sword lion school in exchange of sword lion model making and coloring in lower price. It is expected that the player enjoys the model mocking up process and is impressed with cultural story of sword lions by playing Hidden Lion.
-69-
Hidden Lion
conference of game competition in Toronto Hidden Lion was also published in CHI PLAY 2014 in Toronto Canada. We were invited to attend CHI PLAY conference and introduced Hidden Lion APP in Canada. I n a d d i t i o n , H i d d e n L i o n p ro j e c t wa s presented in the exhibition held by the Fundation of Historic City Conservation and Regeneration. We were devoted to cooperating with local governments and promote its into real markets.
-70-
Location Based APP
Through Hidden Lion, which has a storyline and theme that connects the local culture, we can link the government, sword lion promotion organization, local restaurants and stores. We not only make it an ideal medium for exploring cultural history b u t a l s o a p e r fe c t a p p ro a c h fo r revitalizing local economic.
Future Goal of
the Service System -71-
