Portfolio
Real to Virtual
To optimize interfaces and interactions between reality and virtuality Selected Works 2017-2021
Apply for XR Interaction Designer Xinyi Zhou
Content Virtual World
is a new form of human civilization, including a new supply-demand virtual content relationship, continuous interactions, and multifunctional users, which carries past memory, recent experience, and future imagination. It is a space that combines virtuality and reality and communicates through terminal equipment. In the past 20 years, 2D interface personal computers (PCs) and mobile terminals have dominated information transmission. With the development of Extended Reality(XR) technology and head-mounted displays(HMDs), people access Information, learn knowledge, and meet friends in the virtual space through the 3D mobile interfaces or these HMDs. The new interface allows human experience moves from indirect towards direct, static towards dynamic, fragmented towards continuous. Such an innovation motivates users to freely generate(create) and consume(experience) customized content in virtual space. In this process, users have escaped from the traditional role of the viewer and become a hybrid role that integrates content production, experience design, and platform construction. Therefore, such a brand-new system is subverting our traditional self-identity, social networks, and human civilization, giving us unlimited possibilities for future life.
01 Toward a New Era: Virtual Gallery - Including Pingshan Virtual Gallery & Virtual Gallery at Sansar [VR, Metaverse Platform]
02 "Treasure Hunting" [AR, App, Photogrammetry]
03 Remenbering Urban Village -The written Paper has been submitted to CAADRIA 2022 Conference and is under review (As 1st author) [Photogrammetry, Panoramic video, AR, NVIDIA CloudXR technology]
04 VR Offline Entertainment Experience Module [0-1 Product R&D]
05 AR Cube [AR, App, Object Tracking]
06 Interplay between the virtual and physical heritage [Hololens, Arduino]
07 Meta-Emo [Deep Learning, Pix2Pix GAN, App]
08 Taida AR Exploration [AR, App]
09 Shenyang K11 Kaos VR Experience Space Design [VR Offline Experience]
10 Exhibition: 2021 Asia Digital Exhibition Expo [VR Product; Product Display & Sale]
11 YOMOV VR Experience Hall [VR Offline Experience; Spatial Design]
Toward a New Era: Virtual Gallery
New attempts with two virtual gallery including Pingshan Virtual Gallery & Virtual Gallery at Sansar Location: Shenzhen, China Date: 08/2020 - 12/2020 & 06/2021-08/2021 Team members & Role: Each part has a specific description The form of Contemporary art can be diverse. The transformation of art in different spaces will give the audience a different experience and inspire them to think about the meaning of work from another perspective. Technology is one of the most important labels of Shenzhen. This young city has the best soil for new media art due to its advanced high-tech manufacturing abilities in this background. When the COVID-19 ravaged the world in the first half of 2020, the past life patterns of human beings were affected entirely and changed. This project discussed how traditional art could break through the limitation of space and region to continue the dialogue with the audience. Virtual reality technology and Metaverse platform may be the answer.
Social attention & impact
Physical & virtual art communities
Virtual Exhibitions
Artworks
Digital version preservation
Physical Exhibitions
Conceptual drawing by Mengqi Tu
s
n tio
to bi ct xhi tra l e t a A tu vir
Virtual exhibition open to audiences on an ongoing basis
Attract to physical exhibitions
Physical & virtual art exhibitions Digital Art Trading Physical Art Trading
Research: Research on the history and case studies of virtual gallery on the metaverse platform
Problem caused tbal pandemic of Covid-19 In 2020, The Covid-19 global pandemic led to the closure of offline public places one after another, and galleries are no exception. Activities that people are keen to socialize in art galleries and museums, participate in workshops, and appreciate works of art have also been canceled. I hope that more people can see our works of art and participate in more exchange activities.
Different types of art galleries
How should our museum community operate due to the closure of the pandemic? What kind of activities can keep the museum alive?
Main Technologies -Physical Metarial
Features
Pros and Cons
-Construction
-All exhibits are placed in the physical exhibition hall -Monotonous presentation
-Transportation
-Limited information resources
Really need a way to see good artworks
Cons: -Strong sense of communication
-High topic certainty
Head of Art Museum -Audio, video, games, animation, etc. -VR technology enhanced visual effects -Huge variety of information resources -Thematic diversity
-Gesture Tracking -Image Processing
Artwork collector
-Scene understanding -Input technique -Mobile headset It’s really sad to lose my friends in the museum community. I hope that the museum’s operations can be restored as soon as possible.
Artist
Participants related to the museum put forward their own new ideas, and hoped to find innovative solutions in the constant discussion.
Let me find a way out!
Internet
-Edge computing
-Audio, video, games, animation, etc. -Various interfaces, various visual effects and new social connections -Tradable content
-5G Network
-Iteratively user-generated content
-Extended Reality -Telepresence
Research: Research on the history and case studies of virtual art museums Mobile APP
Augmented Reality Virtual Reality
Pros: -Uncomfortable Device -High requirement of network Cons: -Unlimited Space -Feeling of Presence
-Automatic digital twin -Blockchain
Designer
Visitor
CD-ROM
Pros: -Content is difficult to keep -Not universal
Pros: -Not yet mature -High requirements for various technologies
Cons: -Strong sense of social connection -Secure virtual trading system
Proposal: Tripartite cooperation of two virtual art museums + 2Proposal Metaverse Financial Support
Artist
A three-dimensinal computer-generated museum comprising an immaterial constellation of rooms and exhibits
A virtual museum of science, humanities, and culture that uses infomation, virtual experiments, and handson activities
1991
1999
App utilizes mobile phone devices in order to enrich the visitors experience in the museum
2013
An "Interpertive Gallery" that allows visitors to explore Magritte's "the paradoxes at work in reality".
2018
Features 74 Dutch and Flemish Old Master paintings from the Dutch Golden Ager in a fantasy space exists only in VR
Virtual artwork presented in VR
2021
Metaverse Platform
Virtual Art Gallery
Collector
Designer
DSL Collection
Timeline Digital museum/gallery
Coperation
Visitor
A place where galleries are able to connect with international audiences in real time.
2018
Museum
Physical Art Gallery
Proposal: Virtual Art Gallery by VR 2D Artworks
Platform: Unity Engine
Proposal: Virtual Art Gallery at Metaverse
3D Experiences
2D Artworks
Platform: Sansar
Interactions
Interactions TATE Modern
National Palace Museum
Modigliani VR: The Ochre Atelier
Mona Lisa: Beyond the Glass
Roblox
Fortnite
Pingshan Gallery
Human
Technology
Virtual Gallery
Multiplayer Muséum national d'Histoire naturelle
Kremer Museum
The Scream
The Spirit of Autobiography
Decenterland
Sansar
Physical
Interface
3D Experiences
Interface
Avatar
Narrative Virtual
Virtual Gallery
Physical
Virtual
Spatial Sequance | Pingshan Virtual Gallery Virtual Gallery Entrance Hall
Pingshan Virtual Art Gallery
Virtual Exhibition
Virtual Exhibition Interaction
6 Choose a showroom
1
7
3
2 Offline VR experience
Experience sequance
5
4
Enter the buffer gallery of the virtual art gallery to give users t i m e t o prepare
The virtual art gallery exhibition hall built at Pingshan Art Museum is equipped with three Meta Quest 2 devices, which can support multiperson experience and is the physical entrance of the virtual art gallery.
Specific exhibition s p a c e , taking the work "Meat" a s a n example
Exhibition h a l l entrance platform: users can choose their favorite exhibition to go
Spatial Sequance | Virtual Gallery at Sansar Sansar Hall
Virtual Exhibition
Virtual Gallery Entrance Hall
Transfer space
7 3
4
Original Artwork
Entrance of single exhibition
5
1 8
2
Exhibition Hall
Entrance of DSL Collection
Sansar Experience C e n t e r, f o r u s e r s to choose their favorite activities and participate
The entrance hall of the virtual art gallery provides space for users to communicate and discuss (Virtual social). Eight exhibition halls of different works are waiting for users to explore
6 Back to Entrance Hall
The eight virtual spaces are all designed according to the same space experience system, including spaces with Original artwork space, Transfer space, View platform, and Main exhibition and other functions
Production of Pingshan Virtual Gallery STEP2: 2D Artworks to 3D Virtual Experience 3D Virtual Experience:
Pingshan Virtual Gallery
Re-creation of eight independent physical works of art into the content of the virtual exhibition hall, creating a virtual art gallery through the production of new visual systems and interactive methods
An Offline Virtual Gallery; Walkable Virtual Experience; New Form of Artwork Exhibited Way Location: Shenzhen, China Date: 08.2020 - 12.2020 Team members: Xinyi Zhou, Liquan Liu, Rufeng Xiang, Mengqi Tu Role: 90% Pingshan Virtual Gallery Design, Group Technology deployment, Group Virtual Exhibition Production
Jump into next work
STEP3: Interaction
Entrence
Interaction: Teleport： Human-interface interaction: For example, in this experience, the user can hit a 3D object to make the object produce a unique movement trajectory
Audio & Text: Virtual Exhibition Unity
VR
Digital Preservation
Offline Virtual Gallery Walk into the mind of artists
Artworks
Audio:Text introduction voice, BGM, etc., rendering a better immersive experience Te x t : G i v e t h e w o r k a b a s i c background introduction
Physical art communities
Physical art Trading
Physical Exhibition: Cooperate with Pingshan Art Museum to build the offline exhibition hall entrance of the virtual museum
STEP4: Communication ta
Production of Pingshan Virtual Gallery
va -A
n
tio
ac
er
nt rI
tar
Human-Avatar Interaction
STEP1 | 2D Artworks: Eight Chinese artworks from DSL Collection
a Av
Wireless VR streaming 4 1
5
6
2D Paintings
7
Related Arkworks:
2D Videos
In the context of the global pandemic of covid-19, collectors of DSL Collection hope to bring back the original collection of artworks (in the form of oil paintings, videos, etc.) to people in a new way.
Technology Support | Wireless VR Streaming Data Cable
Successful Streaming
PC
2
Oculus Quest 2
3 8
The eight virtual spaces are all designed according to the same space experience system. The set experience sequance will be presented one by one with the narrative experience
Production of Pingshan Virtual Gallery
Pingshan Virtual Gallery - Multiplayer experience
In-site photo of Virtual Gallery Meta Quest (wireless streaming) uses positioning cameras to identify the environment and assist in positioning, which leads to the need for visually different designs on the walls around the exhibition hall so that the headset is easier to locate.
Meta Quest 1 / Meta Quest 2
Positioning camera
Layout and technical deployment
VR experience area Cable
WIFI x 2 Headset area
1
2
3
4
Display The server
Data cable
Waiting area
Virtual experience sequence 1
2
3
4
Pingshan Virtual Gallery - Multiplayer experience
Interaction: Animation
Trigger
Virtual Gallery at Sansar A new experiment in the Metaverse platform: exploring new forms of existence and trading methods for artworks Location: Sansar, Metaverse Date: 06/2021-08/2021 Team members: Xinyi Zhou, Liquan Liu, Fukai Chen, Junxiao Ma Role: 65% Production of Virtual gallery at Sansar, 40% Virtual exhibition design at Sansar Virtual Exhibition Sansar
Sansar
Online Virtual Gallery
Digital Preservation
Other virtual events
Artworks
Interactive Target of Entrance
Entering from the exhibition hall to each exhibition sets interactive a n i m a t i o n c o n t e n t , prompting the user to enter a new exhibition
Entrance of Each Exhibition & Interactive Target
UI Design G u i d a n c e system: including UI design of origital artwork, transfer space, main exhibition, viewing space
Virtual art communities
Center Space with UI Guide Virtual art Trading
Single Exhibition With eight recreated artwork experiences i n a r o w, t h i s section takes the shock of time as a example to explain what kind of interaction , text, sound, and animation have integrated in the space.
Production of Virtual Gallery at Sansar Virtual Gallery Entrance Hall
Artworks from DSL Collection
1
Entrance of exhibition hall
2D Videos Artworks
Recreate at Sansar platform
Animation
Recreated Artwork - Take the shock of time as example
2
In the context of the global pandemic of covid-19, collectors of DSL Collection hope to bring back the original collection of artworks (in the form of oil paintings, videos, etc.) to people in a new way.
2D Painting Artworks
Entrance of each exhibition
Teleport Space
Transfer Space with Gates
T h e s e t experience sequance will be presented one by one with the narrative experience
Cases of Virtual Exhibition at Sansar | Responsible for Spatial Architecture Design
Exhibition Hall | Main Designer: Liquan Liu
The Engine | Main Designer: Fukai Chen
Flirt | Main Designer: Xinyi Zhou
The Big Era | Main Designer: Liquan Liu
"Treasure Hunting" Build up two experience systems for architectural heritage - Take Zhou Family Courtyard as example Location: Yongzhou, China Date: 11/2020-03/2021 Individual Work
With the loss of the population and the intensification of aging, the idle and dilapidated indoor and outdoor spaces in the Zhou family Courtyard are increasing yearly. Precious buildings and non-cultural heritage are gradually declining. Problems such as lack of protection, single transmission channels, insufficient funds for repairs, and lack of protection awareness among local residents have become more apparent. The purpose of this project is to use the portable and exciting features of AR technology to disseminate and preserve cultural heritage and try to spread the cultural content of the Zhou Family Courtyard in the concept of "4D museums" locally and remotely so that more People participate in the protection of architectural heritage.
Triangulation
HERITAGE DATASET
IN-SITE System
Windows x 3
Photogrammetry
Vuforia
Door x 3 Wall x 3
ROMOTE System
Historical development of Zhou Family Courtyard
Game Procedure
Architectural Heritage Zhou Family Courtyard are in disrepair for a long time
The Old Courtyard, the earliest of the six existing courtyards was built during the Jingtai period in the middle of the Ming Dynasty (1450-1500). 1450
The New courtyard, first built in 1841, Daoguang, the 21st year of Qin Dynasty, covers 4000 square meters in the floorage. 1638
1841
1611
The Four Family Courtyards consist of Central Courtyard, Old Courtyard, Academy and Defensive Tower. It was built from the 30th year of Guangxu to the early years of the Republic of China (1904-1912). 1904 1894
Timeline
Based on the historical timeline, collect data on the three significant elements of doors, windows, and walls in the Zhou Family Courtyard
"Treasure hunt" map
Three levels related to WINDOWS
Unity Engine as a platform for resource integration and AR content production
Three levels related to DOOR
Tourist Ta r g e t u s e r : To u r i s t s . Design and develop from the tourists' perspective
Three levels related to WALL
Use Vuforia to make AR content presentation
"Treasure hunt"
Solve puzzle
Scan target
Reward
Solved Animation: Structure explosion
The Red-door Mansion, first built in Wanli of Ming Dynasty (1611-1628), was built by Zhouzuo's younger son, Zhou Xisheng.
The Black-door Mansion, first built in Tianqi of Ming Dynasty (1638), was built by Zhou Xisheng's oldest son, Zhou Zisu.
The Ziyan Mansion named by Zhou Chongfu was built in Guangxu of Qing Dynasty (1894-1902) by his sons and nephews.
Current problems in Zhou Family Courtyard
Successfully solve the puzzle, open the next level
Virtual Button
Physical to Virtual (Data Collection) Point Cloud - Model - Targeted by Vuforia
The old houses are in disrepair for a long time, and the architectural heritage does not have enough funds for protection
The house that grandpa left behind can no longer be the owner. The young people in the village no longer have reliable labor after they go out.
As a well-known heritage tourist area, the guide sign system in the village is very old and unclear, making it difficult for tourists to obtain effective information and unable to have a good understanding of the architectural heritage.
Clear all levels
AR
AR APP | UI Design & Two System
Landscape screen
"Hunting Treasure" brings vivid digital heritage to visitors through lightweight and pervasive mobile devices, creating a new concept of "4D Museum". The boring and monotonous knowledge of ancient buildings is also better spread with this AR Game.
Rewards by completing all levels of the game will promote tourists' desire to explore and compete in the game. The setting of two different experience systems enables more users to participate in the reflection and discussion of the heritage proservation.
The final rewarded 360 panoramic video also leads the viewer to further understand the relationship between the three elements of doors, windows and walls in the indoor environment.
“Hunting Treasure” In-SITE System
"Hunting Treasure" begin!
In the doors and windows of Ming and Qing Dynasties, a large number of plant patterns appeared. Trees and flowers are often used in ancient auspicious patterns. The ancients gave life to plants and paid attention to the inherent qualities of various plants.
Hunting
Introcution
Seeking
Animation
Interaction
REMOTE System
In Song Dynasty, it was called waist board, and "Building Fayuan" was called Jiatang board. According to Liang Sicheng, the double ring board was called Jiatang board.
Hunting
Introcution
Environment
Animation
IN-SITE | Game Procedure
IN-SITE | User Experience
STEP1: HUNTING Getting hidden introduction by clicking different site icons on digital map
STEP2: GENARATION Generate the corresponding digital model f r a m e s b y scanning the AR markers found
STEP3: HINT Using virtual button technology to show that how many pieces of the puzzle composed of this object
IN-SITE | User Experience
The support window is a kind of window that can be raised and taken off.
STEP4: PUZZLE
The upper section can be pushed out and the lower section can be taken off.
Use virtual button technology to show the unique information of each structure, thus guiding users to solve puzzles successfully
S T E P 5 : R E WA R D PANORAMA 360
Using camera and smartphone to catch almost 360 degrees' photos for building up 3D models
REMOTE | Game Procedure
REMOTE | User Experience
STEP1: HUNTING The Zhou Family Courtyard is located at the source of the Xianshui River, a tributary of Xiaoshui River, Fujiaqiao Town, Lingling District, Yongzhou City.
Getting hidden introduction by clicking different site icons on digital map
It is generally divided into upper and lower sections.
STEP2: Experience C l i c k t h e " Tr e a s u r e " p o s i t i o n , and the corresponding target will be displayed. By observing the relationship between "treasure" and the surrounding environment, we have a preliminary understanding of In the windows of Ming and Qing Dynasties, a large number of plant patterns appeared. Trees and flowers are often used in ancient auspicious patterns. The ancients gave life to plants and paid attention to the inherent qualities of various plants.
on
STEP3: PUZZLE
The upper section can be pushed out and the lower section can be taken off.
Use virtual button technology to show the unique information of each structure, thus guiding users to solve puzzles successfully
On
ama
Off
STEP4: Animation
360
Use the button to control whether the "treasure" shows an explosion animation, so as to better interact with people around you
S T E P 5 : R E WA R D PANORAMA 360
Using camera and smartphone to catch almost 360 degrees' photos for building up 3D models
U s i n g camera and smartphone to catch almost 360 degrees' photos for building up 3D models
1
2
3
4
5
6
7
8
Remembering Urban Village Written as paper Remembering Urban Village, accepted by CAADRIA 2022 Conference Using CloudXR Technology as an Enhanced Alternative to Visualise Heritage Interactively Location: Shenzhen, China Date: 8/2020-4/2021 Team Member: Xinyi Zhou, Rufeng Xiang, Mengqi Tu, Haida Sun Role： 80% Innovation & Technology Plan; 100% UI Design; 50% Project landing Urban villages are strictly related to urban growth. It reflects the era characteristics and memory of a region in urban growth, which has significant value in heritage documentation and dissemination. Due to the continuous development of urbanization and the shortage of urban land, many urban villages will be replaced by more valuable functions. Therefore, how to better preserve the digitalization of urban villages and make more people understand the value of urban villages is particularly important. However, the existing technology still has shortcomings in disseminating digital heritage. For urban villages, usually a large-scale and complex environment, the hardware requirements will be very high for high-precision visualization. Most of the existing solutions use large hardware devices, such as the virtual sand table. Unlike hand-held devices, such devices are expensive and not portable, limiting better dissemination of such heritage. Due to the hardware limitation of hand-held devices, neither the display resolution nor the interaction effect is satisfying. Therefore, this project proposes a new workflow by NVIDIA CloudXR streaming technology to achieve high-precision visualization and a rich interactive experience on hand-held devices. Such heritage can be promoted and popularised more conveniently.
Computational Process
Documentation Photogrammetry Modeling
Digital Reconstruction
Dissemination High-fidelity
NVIDIA CloudXR Technology
Interactivity High Accessibility
Research of Chinese Urban Village
Proposal
Urban villages have become a powerful booster for the urbanization pro ce ss an d p r ov ide aff or dable housing and jobs for the low-income migration for the rapid urbanism of China over the past years.
AR
Interaction
Designer
Sandbox DOCUMENTATION
BaiShiZhou Renewal Project
TECHNOLOGY
Government
Digital Heritage
Text
Contractor
Movement
Animation
COMMUNICATION
Recall
Communication
Citizen
Ray-casting
Interior Models INTERACTION
For urban villages, usually a large-scale and complex environment, the hardware requirements will be very high for high-precision visualization.
Baishizhou Urban Renewal Project
DISSEMINATION
Child: Wow, I can see where my parents lived before, and the food at the roadside stall looks so delicious.
Contractor: It’s great to protect our urban heritage while promoting urban development.
Participants
Delegate Phase I construction Phase II construction Phase III construction Phase IV construction Phase I demolition
Government
Contractor
Win-Win
To promote urban development, carry out urban renewal.
Phase II demolition
Contractor
Phase III demolition
Economic Goals
Connecting the government and citizen, implement urban renewal plans, and obtain economic benefits.
Contractor: Presenting Baishizhou in this way, I believe it will retain more precious memories of the urban village.
Phase IV demolition
Baishizhou Urban Renewal Project
Urban Development
Government
Citizen Urban village provides them with convenient housing and affordable living needs.
Provide Housing
Designer Consider the needs and interests of all parties and seek creative solutions. Understand needs
Daily needs
Designer
Residents
Preservation and solutions
Social Development
Development and Construction Land Scope Line
Improve the quality of life
Remove scope line
Firstly using tablet to scan target Villager Own real estate and commity network in urban village.
Main Workflow Methodology:
Computational Process:
UI Design and Presentation:
There are many methodologies for architectural heritage preservation. This
Compared with using traditional localized high computing power
NVIDIA ClouxXR technology significantly promotes the accessibility and fidelity of the content. Meanwhile, the
project uses a methodology with documentation-computation-dissemination
equipment, NVIDIA CloudXR technology can remotely control
Unity engine enables visualized information, amusing animation, sound effects, and UIUX design to be linked to
(DCD) workflow, focusing on technology to promote high accessibility, fidelity,
server equipment for data processing. It is innovative digital heritage
the digital heritage to enhance the fun and interactivity of the content.
and interactivity in disseminating digital heritage. This methodology proposes
preservation and can replace localized high computing power equipment.
UI Design:
a new perspective of the preservation of digital architectural heritage.
DCD Workflow:
both sides: the device and the server, respectively.
Documentation
Digital Reconstruction
Photogrammetry & Modeling:
Softwares:
1.UAV Oblique photography 2.Panoramic camera 3.Sketch and modeling
When using NVIDIA CloudXR technology, it needs to use devices at
1.Agisoft Metashape 2.Unity 3.Adobe PR 4.3D MAX
Computational Process
Dissemination
Technology:
Features:
NVIDIA CloudXR Technology
1.High fidelity 2.Interactivity 3.High Accessibility
SERVER
CLIENT Network
Documentation:
Seek & Scan Target
Ray-casting interaction & Swipe up
Get infomation & Swipe up again
Get Model & Move around
Data of Texture:
NVIDIA CloudXR Server
NVIDIA CloudXR Client
A. Activates the device camera's Ray casting algorithm. When the user aligns the device camera at a specific area where will be highlighted.
Outcome of Photogrammetry:
B. Slide the screen upwards: the specified animation and sound effects emerge. C. Swipe the screen up again. D. Based on the technical support of bi-directional audio in CloudXR, users can remotely have real-
How many devices can be tested at the same time
time communication and discussion.
Triangulation Modeling:
Click to experience again Presentation:
Test Result: Single Clean Channel
Vi d e o s t r e a m compression to 30Mbps
Single Clean Channel
Vi d e o s t r e a m compression to 30Mbps
10
30
4
12
High-fidelity urban village model
Multiplayer experience
Representation empowered by NVIDIA Cloud XR
Experiment result:
We invited 27 people to be participants in this experiment: Currently living in Baishizhou: 11 people. Previously lived there: 16 people.
Seek Target
Multiplayer experience
Digital Heritage
Interaction
UI design
Animation
Basing on the high-precision Baishizhou data collection and NVIDIA CloudXR technology for the transmission of high-precision content, the Baishizhou AR Sandbox shows excellent performance in the presentation model content, regardless of the richness or the accuracy of the model content. It exceeds the imagination of the viewer significantly. From the high-precision data collection of Baishizhou and the presentation of high-precision content, NVIDIA CloudXR technology has played a vital role in the workflow.
VR Offline Entertainment Experience Module Research & Development R&D from 0 to 1 [VR; Product experience design; Spatial Design] Location: Shenzhen, China Date: 12/2021-01/2022 Team Member: Xinyi Zhou, Liquan Liu Role： Responsible for product experience design; product production; supplier docking; product landing process; product iteration
From 0-1 design, production and construction of VR interactive game experience modules, to create a new social, high-quality and high-return offline entertainment experience. This product can set and control the number of modules according to the size and budget of the physical store.
Modelling
Rendering
Research & Development
Game modules
SALES MODEL | VR offline experience store
VR offline experience store
AR Cube A AR product to protect the lost cultural heritage Date: 10/2021-12/2021 Team Member: Xinyi Zhou, Xing Hao Role: Technology research, Innovation and technology planning, 70% XR content production As one of the oldest city in China, Beijing has the most complicated city water system, which has witnessed the development process of this historic city. Many culture meme, such as Hutong(name of traditional streets) and Obon(a traditional a traditional Buddhist and Taoist festival), are related to the water system. However, many people do not know these knowledge and cultural significance. This project aims to popularize the knowledge and history of water system of Beijing. Media technology can build multiple virtual worlds in parallel for different groups of interest, encouraging creation and recreation without the constraints of time, space, and social class. This project abstracted the characteristics of Shichaha, a manmade lake lies in the center of Beijing, and combined augmented reality with paper box in order to allow people experience the ancient activities related to water and arouse people's awareness of protecting the water environment.
Research & Modelling
AR Cube Product Single Object Tracking
PLAY MODE
Unity with Vuforia
Ancient Water Culture: Take Shishahai area as a example
Mutiple Object Tracking Scene Tracking (4Objects + Map)
Preservation
Dissemination Better
Product Sale Better
Biopolitics | Research on Context Beijing | Histroy | Watersystem
Yuan Dynasty [AD 1215] The city was rebuilt around the Bailiantan lake so that it is convenient for grains and materials from all over the country to the capital through water transportation.
Qing Dynasty [AD 1658]
Ming Dynasty [AD 1420]
In order to defend against enemies in the north, the Ming Dynasty moved the city walls south and rebuilt the moat on the original city.
During the Qing Dynasty, the city of Beijing followed the pattern of the Ming Dynasty and began to build gardens when the river was being treated.
Evolution of Reading Media Newspaper
E-Book
Video
1998: a dedicated e-book reading device appeared 1969: The Internet appeared in 1969
VR/AR
2007: Kindle
Metaverse
2020: Oculus quest 2 is released, making HMD more popular
2005: Youtube was founded
Timeline
2021: Immersive novel
Efficiency of Information Transfer
[Reading by AR] Physical Target
Temples Water conservancy facilities
Folk activities
Surrounding structures
Huitong Temple
Yinding Bridge
Fire Temple Dojo Obon Bathing Buddha Play on the ice Chunxu Shishahai Landscape
Making Process of Object Tracking and Cude Mode
Paper Book: 1st century BC Oracle: About 16th to 10th century BC
Guanghua Temple
Fire Temple
Landscape
Jin Dynasty [AD 1153] The governors of Jin Dynasty first made Beijing the political center, and the construction of palace gardens and the use of water became their first considerations.
Water landscape & culture
Lying in the north of the North China Plain, Beijing has abundant water resources brought by five river systems. Since 1153( Jin Dynasty), Beijing has always served as the political and cultural center of the northern regime and even the whole country. Thus culture about the water system is also thriving. The water system in the city of Beijing mainly provided water and landscape for the imperial city. At the same time, the flourishing of Buddhism led to the development of the water landscape into royal palaces, temples and suburban
Realm of Beijing Main river
Beijing Water Culture
Content Structure
Interaction
In many cases, augmented reality can effectively replace human imagination and place objects or devices in front of the learner’s eyes at the right moment in the training.
[Content in AR]
[Text]
[Image]
[Video]
[3DModel]
[3DAnimation]
AR to History | Intersection Method
AR Cube Product | Making Process Stone stele (recording the treatment of waters)
Huitong Temple
STEP1: MAKING
Underground sluice
Water beast
Surrounding environment
There is a large water gate under Huitong Te m p l e w i t h m o v a b l e wooden gates embedded in the stone troughs on both sides. The water level and flow rate of the entire Beijing waters are controlled. Therefore, it is called the throat of the capital waters.
X6
Underground Water Gate The production process of AR CUBE: abstract solid building facade x6 - combined into a cube
Huitong Temple
Guanghua Temple
Fire Temple Dojo
Map: Combination play mode of AR CUBE 1 Single Object Target
2 Mutiple Objects Target
STEP2: LINKING
Yinding Bridge
Fire Temple
Fire Temple Dojo
On June 23rd, all fire temples in the capital will hold grand avenues. Di’anmen Fire Temple is the largest of the fire temples in Beijing. The Taoist priests wore robes and vestments and held law enforcement instruments or musical instruments when doing the dojo.
Single Water-related Construction
Water-related Activity
3 Scene Target
unity Use the Vuforia platform in Unity to link virtual assets and physical objects and divide them into three levels of content for presentation: 1. Single object target 2. Multiple objects target 3. Scene target. To produce a better product experience for consumers and further understand the rich Shichahai water cultural content.
Obon
Play "AR Cube" on Map
Shicha waters culture： Obon
The Buddhist community held a grand Obon Festival on the Lunar New Year's Day on the 15th day of the lunar calendar. There were mass activities such as the lighting of river lanterns and burning of boats. Shicha Waters Culture The Shicha sea area was the terminal of the Beijing-Hangzhou Grand Canal in the Yuan Dynasty. It was the basis for the planning of the capital city of the Yuan Dynasty.
Product
Huge Folk Activity on Shisha Sea
Dissemination
STEP3: DISSEMINATION Package the designed AR Cube and Map into "AR CUBE" products for sale and dissemination. To publicize and promote the disappearing water culture of Beijing and let more people learn about and protect Beijing water culture through this exciting product.
AR APP | UI Design & Cube Play Mode
[Sanning Individual AR Cube]
This app has two main functions: 'Start' and 'About'. 'Start' is a camera mode UI, through which users could scan the cubes and the map. Then the 3D model of the sites including Huoshen Temple, Guanghua Temple, Yindian Bridge and Huitong Temple will be displayed. 'About' allows users to get the details of the sites directly. The combination of the cubes and the map will show different folk activities related to Shichahai.
Move cube
Open Collapsed Map
Scan object
Scan Map Scan 1 Cube
By scaning individual cube on the map, users can get detailed knowledge of the architecture and its components related to shichahai and water culture.
Scan 2 Cubes
The combination of two cubes and the map will show the users the relevant cultural activities about this two buildings as well as the landscape around Shichahai.
Adjust the Focus
Scan 4 Cubes + Map
If all of the four cubes and the map are scanned, the Obon Festival, the most celebrated festival during the history of Shichahai, will be displayed.
Manual Focus
[Sanning Multiple AR Cubes]
Prototype | Usage Scenarios
Interplay between the vir tual and physical heritage Using MR technology to protect architectural heritage. [Technologies: Hololens, Arduino] Location: Shenzhen, China Date: 11/2021-01/2022 Team Member: Xinyi Zhou, Fukai Chen, Zhouyi Li, Chaohe Lin Role： 80% Innovation & technology planning, 50% Virtual Content Production Cultural heritage contains multiple components, including tangible culture (buildings, monuments, etc.) and intangible culture (history, stories, music, etc.). The virtual and physical features together define the local identity and spirit of place. Immersive technology has much accelerated the digital transformation of heritage. Lots of researches in cultural heritage based on VR, AR, and MR have proved the feasibility. MR is defined as combining virtual elements (e.g., holograms or projections) with the real world. Compared with AR, MR can achieve a broader goal of enhancing our virtual and physical environment understanding.
t pu
In
ce
rfa
te
Hololens
Arduino In
In fe tera ed c Vi ba tiv su ck e al is at io n
We proposed a new method of interaction between the virtual and physical heritage in an MR application, aiming to break through the interplay limitations. This method is based on the integration of environmental sensors, MR technology, and cultural heritage. This integration provides an opportunity to utilize immersive and sensing technology to improve the participation and understanding of visitors.
Input Interface
Unity
Technical Test (MR devices include Hololens & Arduino)
Game Proposal Gesture interaction:
AR glass - Hololens: Display virtual contents and provide virtual interaction to users
Unclarity Low participation
Vapidity
Information platform - Unity: Connect virtual and reality and deliver information
Insufficient relationship of the interplay between virtual information and heritage artifacts
Sensing technology - Arduino： Offer physical interaction to track users' behaviours
Improve the participation and understanding of visitors to architectural heritages
Game platform:
A platform that provides promotions for MR games
Virtual heritage:
Kaiyuan Temple, Yanghe Building, Guanghui Temple, Changle Gate
MR technologies
Research
Solution
Prompt to reselect Choose wrong
Info notification x5
Physical interaction Base on on-site environment
RFID_RC522 x 5
Choose right Virtual heritages Game platform x 5 identifications Gesture interaction
Scale and move
Technical Process
RFID_RC522 x 5
Breadboard x 1
Arduino UNO x1
Finished
Game & Display Content displayed by Hololens
User Experience Experience process
Game procedure
Meta-Emo An virtual media with Pix2Pix GAN Date: 7/2020-10/2020 Team Member: Xinyi Zhou, Han Tu, Ziyu Han, Xinyi Yang Role: Conceptual Design 80%, Data mining 80%, Model training 50% Typically, people's reactions to art are considered subjective — For instance, Munch's The Scream might instill foreboding anxiety in you, but do absolutely nothing for your friend. However, artist Wassily Kandinsky feels that abstract art, what people would perhaps consider the most emotionally subjective form of art, is entirely objective. He suggests that the emotional objectivity of abstract art lies in the characteristics of the colors and their interactions with one another. Can the emotions expressed in pictures be recognized by computers? What characteristics of emotions in pictures can be well reflected by computers? How does the computer add emotional processing to the picture? This project aims to show that machine can recognize human emotions and express them through the image-to-image translation of hand-drawn sketches by conditional adversarial networks, so that the process of humancomputer interaction will treat emotions. People use art to communicate their emotions to each other, and in the same way art also heal people's emotions.
Dataset Mining & Classification WikiArt Emotions Dataset:
Classification:
There are 2800pieces of drawings collected from WikiArt dataset, which include 12
First, we split the data, using 85% of the data as the training set (and labeled
different genres such as Abstract, Impressionism, Pointillism, Post-Impressionism,
it as one of "Happy", "Angry", "Sad", "Neural" and "Fear"). The remaining 15% of the data is used as the test set.
Also, we conducted data mining on google image through tags such as "Painting", "Emotional", and "Western Painting", and finally we got a dataset of 3500 pieces.
WikiArt Dataset
Training Dataset - 85%
Data Mining
Test Dataset15%
01 | Tagging
02 | Classify ID: Test_Img_001
Fauvism, Classicism, Surrealism and Pop Art etc.
Method of Classification Sentiment scoring: ID: Test_Img_001 Anger rating:0.29 Happiness rating:0.57 Sadness rating:0.25 Neutral rating:0.35 Fear rating:0.19
03 | Identify
ID: Test_Img_001 Sentiment: Happiness
3500 Pieces Collected Dataset:
Sentiment Scoring Sentiment scoring:
Neutral
Happiness
ID: Test_Img_027 Anger rating:0.29 Happiness rating:0.57 Sadness rating:0.24 Neutral rating:0.35 Fear rating:0.18 Sentiment scoring:
Sadness
Fear
The purpose of data classification: classify various types of data sets according to their color and style, and finally classify them into five emotion categories: "Happy", "Angry", "Sad", "Neural" and Anger
"Fear".
Methods of RGB Color Extraction: Method1:
ID: Test_Img_025 Anger rating:0.32 Happiness rating:0.49 Sadness rating:0.24 Neutral rating:0.33 Fear rating:0.30 Sentiment scoring: ID: Test_Img_013 Anger rating:0.29 Happiness rating:0.35 Sadness rating:0.21 Neutral rating:0.26 Fear rating:0.46 Sentiment scoring:
Divide the three primary color dimensions (R, G, B) of the color of the picture into 4096 blocks if n=16, each block is recorded as a color block, and the rgb value of the midpoint of the color block represents the color of the color block . You can take and count the number of pixels in each color block of all the pixels of the picture, and then visualize the top 10 color blocks with the most pixels and use a pie chart to visualize them.
ID: Test_Img_0018 Anger rating:0.40 Happiness rating:0.39 Sadness rating:0.24 Neutral rating:0.32 Fear rating:0.29
Method2:
Sentiment scoring:
Each pixel of the picture, as a sample point, each sample point will fall in a three primary color dimension (R, G, B) space, according to the distance, use the method of clustering in machine learning, these The sample points are clustered into 10 categories, and all points in each category will have a centroid (or center, that is, the average value of the coordinates of each point), and then use the color corresponding to the coordinates of the centroid to color the sample points to obtain a three-dimensional space. A visualization of the distribution.
ID: Test_Img_042 Anger rating:0.29 Happiness rating:0.33 Sadness rating:0.41 Neutral rating:0.34 Fear rating:0.25
Color Cluster
3D distribution visualization
Pix2Pix Model Training & UI Design 3D Distribution Visualization Results
Pix2Pix Model Training
Emotion Express App
Each pixel of the picture, as a sample point, will fall in a three primary colour dimension (R, G, B) space, according to the distance, using the method of clustering in machine learning. These sample points are clustered into 10 categories. All topics in each category will have a centroid (or centre, that is, the average value of each point), and then use the colour corresponding to the coordinates of the centroid to colour the sample points to obtain 3D distribution visualization results.
Generate
Sketch Outline of Phone
Training Result of Classification: Sentiment Scoring Sentiment scoring:
Color Cluster
3D distribution visualization
ID: Test_Img_050 Anger rating:0.29 Happiness rating:0.34 Sadness rating:0.43 Neutral rating:0.35 Fear rating:0.26 Sentiment scoring: ID: Test_Img_0055 Anger rating:0.24 Happiness rating:0.39 Sadness rating:0.42 Neutral rating:0.39 Fear rating:0.19 Sentiment scoring: ID: Test_Img_081 Anger rating:0.32 Happiness rating:0.37 Sadness rating:0.27 Neutral rating:0.40 Fear rating:0.28 Sentiment scoring: ID: Test_Img_083 Anger rating:0.25 Happiness rating:0.47 Sadness rating:0.28 Neutral rating:0.50 Fear rating:0.15
Training Result Input
Output
Pix2pix Loss over time Target
Loss: Discriminator
Sentiment scoring: ID: Test_Img_085 Anger rating:0.25 Happiness rating:0.45 Sadness rating:0.25 Neutral rating:0.48 Fear rating:0.19
Loss: Generator
Loss: Generator_GAN
Sentiment scoring: ID: Test_Img_089 Anger rating:0.27 Happiness rating:0.39 Sadness rating:0.30 Neutral rating:0.44 Fear rating:0.23
Different Mood
Taida AR Exploration Enhance the fun and exploreability of a tour. [AR, App] Location: Tianjin, China Date: 03/2021-07/2021 Team Member: Xinyi Zhou, Rufeng Xiang, Mengqi Tu, Haida Sun Role: Innovation & technology planning, Content Production, UI Design This project is a large-scale commercial project, which includes the AR exploration experience of aircraft carrier park, popularization of aircraft carrier knowledge, and cloud tourism. My job was mainly responsible for the innovation and technology planning of each AR project and the design of UIUX [Unity, C#, 3ds Max, Adobe XR, App development]. It gave me a new understanding of innovation and technology from a business perspective.
Offline Armament AR experience | UIUX Design
Online Armament AR experience | UIUX Design
Live AR presentation | Offline Armament AR experience
Live AR presentation | Offline Armament AR experience
Profassional Work: Shenyang K11 Kaos VR Experience Space Design [VR Offline Experience]
Location: Shenyang, China Date: 12/2020 Role: Experience Design; Spatial Design
Exhibition: 2021 Asia Digital Exhibition Expo Responsible for product experience design; product display Location: Guangzhou, China Date: 05/2021 Role: Experience Design; Product display
Spatial Design: Design of YOMOV VR Experience Hall Space experience transformation, product display Location: Shenzhen, China Date: 03/2021 Role: Experience Design; Spatial Design