Portfolio_Xinyi Zhou

Portfolio

Real to Virtual

MAJOR PROJECT / THESIS XINYI ZHOU

Content

The traditional way of information transmission always has issues such as single interaction mode, limited transmission efficiency, and point-like way of transmission that because it has been restricted by two-dimensional interfaces. In the past 20 years, 2D interface personal computers (PCs) and mobile terminals have dominated information transmission. With the iteration of information technology, people access information, learn knowledge, and meet friends through more interactive, timely, and three-dimensional interfaces. The new interface allows human experience to move from indirect to direct, static to dynamic, and fragmented to continuous. Such an innovation expand users' abilities to interact and motivates interactive living status at any time.

01 Remenbering Urban Village

-The written Paper has been submitted to CAADRIA 2022 Conference and is under review (As 1st author)

The documentation, representation, and dissemination of urban villagesTake advantage of a new workflow based on NVIDIA CloudXR technology. [Photogrammetry, Panoramic video, AR, NVIDIA CloudXR technology]

02 "Treasure Hunting"

Build up two experience systems for architectural heritage [AR, App, Photogrammetry]

03 Meta-Emo

An virtual media with Pix2Pix GAN [Deep Learning, Pix2Pix GAN, App, VR]

04 AR Cube

New products to protect the lost cultural heritage [AR, App, Object Tracking] 05 Toward

a New Era: Virtual Gallery

- Including Pingshan Virtual Gallery & Virtual Gallery at Sansar Various ways to explore the definition of virtual gallery at new era [VR, Metaverse Platform]

Remembering Urban Village

Using CloudXR Technology as an Enhanced Alternative to Visualise Heritage Interactively

Location: Shenzhen, China

Date: 8/2020-4/2021

Team Member: Xinyi Zhou, Rufeng Xiang, Mengqi Tu, Haida Sun

Role：

80% Innovation & Technology Planner; 100% UI Design; 50% Project landing

Urban villages are strictly related to urban growth. It reflects the era characteristics and memory of a region in urban growth, which has significant value in heritage documentation and dissemination . Due to the continuous development of urbanization and the shortage of urban land, many urban villages will be replaced by more valuable functions. Therefore, how to better preserve the digitalization of urban villages and make more people understand the value of urban villages is particularly important. However, the existing technology still has shortcomings in disseminating digital heritage . For urban villages, usually a large-scale and complex environment, the hardware requirements will be very high for high-precision visualization. Most of the existing solutions use large hardware devices, such as the virtual sand table. Unlike hand-held devices, such devices are expensive and not portable, limiting better dissemination of such heritage. Due to the hardware limitation of hand-held devices, neither the display resolution nor the interaction effect is satisfying. Therefore, this project proposes a new workflow by NVIDIA CloudXR streaming technology to achieve high-precision visualization and a rich interactive experience on hand-held devices . Such heritage can be promoted and popularised more conveniently.

Written as paper Remembering Urban Village , accepted by CAADRIA 2022 Conference

Proposal

Urban villages have become a powerful booster for the urbanization process and provide affordable housing and jobs for the low-income migration for the rapid urbanism of China over the past years.

Baishizhou Urban Renewal Project

Methodology:

There are many methodologies for architectural heritage preservation. This project uses a methodology with documentation-computation-dissemination (DCD) workflow, focusing on technology to promote high accessibility, fidelity, and interactivity in disseminating digital heritage. This methodology proposes a new perspective of the preservation of digital architectural heritage.

Computational Process:

UI Design and Presentation:

NVIDIA ClouxXR technology significantly promotes the accessibility and fidelity of the content. Meanwhile, the Unity engine enables visualized information, amusing animation, sound effects, and UIUX design to be linked to the digital heritage to enhance the fun and interactivity of the content. When

Compared with using traditional localized high computing power equipment, NVIDIA CloudXR technology can remotely control server equipment for data processing. It is innovative digital heritage preservation and can replace localized high computing power equipment.

Design: A. Activates the device camera's Ray casting algorithm. When the user aligns the device camera at a specific area where will be highlighted.

Slide the screen upwards: the specified animation and sound effects emerge.

Swipe the screen up again. D. Based on the technical support of bi-directional audio in CloudXR, users can remotely have realtime communication and discussion.

Representation empowered by NVIDIA Cloud XR

Basing on the high-precision Baishizhou data collection and NVIDIA CloudXR technology for the transmission of high-precision content, the Baishizhou AR Sandbox shows excellent performance in the presentation model content, regardless of the richness or the accuracy of the model content. It exceeds the imagination of the viewer significantly. From the high-precision data collection of Baishizhou and the presentation of high-precision content, NVIDIA CloudXR technology has played a vital role in the workflow.

Experiment result: We invited 27 people to be participants in this experiment: Currently living in Baishizhou: 11 people. Previously lived there: 16 people.

Seek Target Multiplayer experience Interaction UI design Animation Digital Heritage

"Treasure Hunting"

Build up two experience systems for architectural heritage

- Take Zhou Family Courtyard as example

Location: Yongzhou, China

Date: 11/2020-03/2021

Individual Work

With the loss of the population and the intensification of aging, the idle and dilapidated indoor and outdoor spaces in the Zhou family Courtyard are increasing yearly.

Precious buildings and non-cultural heritage are gradually declining. Problems such as lack of protection, single transmission channels, insufficient funds for repairs, and lack of protection awareness among local residents have become more apparent.

The purpose of this project is to use the portable and exciting features of AR technology to disseminate and preserve cultural heritage and try to spread the cultural content of the Zhou Family Courtyard in the concept of "4D museums" locally and remotely so that more People participate in the protection of architectural heritage.

"Hunting Treasure" brings vivid digital heritage to visitors through lightweight and pervasive mobile devices , creating a new concept of "4D Museum". The boring and monotonous knowledge of ancient buildings is also better spread with this AR Game

Rewards by completing all levels of the game will promote tourists' desire to explore and compete in the game. The setting of two different experience systems enables more users to participate in the reflection and discussion of the heritage proservation

The final rewarded 360 panoramic video also leads the viewer to further understand the relationship between the three elements of doors, windows and walls in the indoor environment.

IN-SITE

STEP1: HUNTING

Getting hidden introduction by clicking different site icons on digital map

STEP2: GENARATION

| User Experience

STEP3: HINT

Using virtual button technology to show that how many pieces of the puzzle composed of this object

STEP4: PUZZLE

Use virtual button technology to show the u nique information of each structure, thus guiding users to solve puzzles successfully

STEP5: REWARDPANORAMA

Using camera and smartphone to catch almost 360 degrees' photos for building up 3D models

STEP1: HUNTING

Getting hidden introduction by clicking different site icons on digital map

STEP2: Experience

Click the "Treasure" position, and the corresponding target will be displayed. By observing the relationship between "treasure" and the surrounding environment, we have a preliminary understanding of

STEP3:

PUZZLE

Use virtual button technology to show the u nique information of each structure, thus guiding users to solve puzzles successfully

STEP4: Animation

Use the button to control whether the "treasure" shows an explosion animation, so as to better interact with people around you

Using camera and smartphone to catch almost 360 degrees' photos for building up 3D models

In

STEP5: REWARDPANORAMA

Using camera and smartphone to catch almost 360 degrees' photos for building up 3D models

On Off

The Zhou Family Courtyard is located at the source of the Xianshui River, a tributary of Xiaoshui River, Fujiaqiao Town, Lingling District, Yongzhou City.

Meta-Emo

An virtual media with Pix2Pix GAN

Date: 7/2020-10/2020

Team Member: Xinyi Zhou, Han Tu, Ziyu Han, Xinyi Yang

Role: Conceptual Design 80%, Data mining 80%, Model training 50%

Typically, people's reactions to art are considered subjective — For instance, Munch's The Scream might instill foreboding anxiety in you, but do absolutely nothing for your friend.

However, artist Wassily Kandinsky feels that abstract art, what people would perhaps consider the most emotionally subjective form of art, is entirely objective. He suggests that the emotional objectivity of abstract art lies in the characteristics of the colors and their interactions with one another.

Can the emotions expressed in pictures be recognized by computers? What characteristics of emotions in pictures can be well reflected by computers? How does the computer add emotional processing to the picture?

This project aims to show that machine can recognize human emotions and express them through the image-to-image translation of hand-drawn sketches by conditional adversarial networks, so that the process of humancomputer interaction will treat emotions. People use art to communicate their emotions to each other, and in the same way art also heal people's emotions.

WikiArt Emotions Dataset:

There are 2800pieces of drawings collected from WikiArt dataset, which include 12 different genres such as Abstract, Impressionism, Pointillism, Post-Impressionism, Fauvism, Classicism, Surrealism and Pop Art etc.

Also, we conducted data mining on google image through tags such as "Painting", "Emotional", and "Western Painting", and finally we got a dataset of 3500 pieces

Classification:

First, we split the data, using 85% of the data as the training set (and labeled it as one of "Happy", "Angry", "Sad", "Neural" and "Fear"). The remaining 15% of the data is used as the test set.

Method of Classification

01 | Tagging

02 | Classify

ID: Test_Img_001

Sentiment scoring:

ID: Test_Img_001

Anger rating:0.29 Happiness rating:0.57 Sadness rating:0.25 Neutral rating:0.35 Fear rating:0.19

03 | Identify

Anger Happiness Neutral Sadness Fear

Sentiment

Scoring Color Cluster 3D

Sentiment scoring:

ID: Test_Img_027 Anger rating:0.29 Happiness rating:0.57 Sadness rating:0.24 Neutral rating:0.35 Fear rating:0.18

Sentiment scoring:

The purpose of data classification: classify various types of data sets according to their color and style, and finally classify them into five emotion categories: "Happy", "Angry", "Sad", "Neural" and "Fear".

Methods of RGB Color Extraction:

Method1:

Divide the three primary color dimensions (R, G, B) of the color of the picture into 4096 blocks if n=16, each block is recorded as a color block, and the rgb value of the midpoint of the color block represents the color of the color block . You can take and count the number of pixels in each color block of all the pixels of the picture, and then visualize the top 10 color blocks with the most pixels and use a pie chart to visualize them.

Method2:

Each pixel of the picture, as a sample point, each sample point will fall in a three primary color dimension (R, G, B) space, according to the distance, use the method of clustering in machine learning, these The sample points are clustered into 10 categories, and all points in each category will have a centroid (or center, that is, the average value of the coordinates of each point), and then use the color corresponding to the coordinates of the centroid to color the sample points to obtain a three-dimensional space. A visualization of the distribution.

ID: Test_Img_025 Anger rating:0.32 Happiness rating:0.49 Sadness rating:0.24 Neutral rating:0.33 Fear rating:0.30

Sentiment scoring:

ID: Test_Img_013 Anger rating:0.29 Happiness rating:0.35 Sadness rating:0.21 Neutral rating:0.26 Fear rating:0.46

Sentiment scoring:

ID: Test_Img_0018 Anger rating:0.40 Happiness rating:0.39 Sadness rating:0.24 Neutral rating:0.32 Fear rating:0.29

Sentiment scoring:

ID: Test_Img_042

Anger rating:0.29 Happiness rating:0.33 Sadness rating:0.41 Neutral rating:0.34 Fear rating:0.25

Anger

Collected Dataset:

Distribution Visualization Results

Each pixel of the picture, as a sample point, will fall in a three primary colour dimension (R, G, B) space, according to the distance, using the method of clustering in machine learning. These sample points are clustered into 10 categories . All topics in each category will have a centroid (or centre, that is, the average value of each point), and then use the colour corresponding to the coordinates of the centroid to colour the sample points to obtain 3D distribution visualization results

Training Result of Classification:

Sentiment scoring:

ID: Test_Img_050 Anger rating:0.29 Happiness rating:0.34 Sadness rating:0.43 Neutral rating:0.35 Fear rating:0.26

Sentiment scoring:

ID: Test_Img_0055

Anger rating:0.24 Happiness rating:0.39 Sadness rating:0.42 Neutral rating:0.39 Fear rating:0.19

Sentiment scoring:

ID: Test_Img_081 Anger rating:0.32 Happiness rating:0.37 Sadness rating:0.27 Neutral rating:0.40 Fear rating:0.28

Sentiment scoring:

ID: Test_Img_083 Anger rating:0.25 Happiness rating:0.47 Sadness rating:0.28 Neutral rating:0.50 Fear rating:0.15

Sentiment scoring:

ID: Test_Img_085

Anger rating:0.25 Happiness rating:0.45 Sadness rating:0.25 Neutral rating:0.48 Fear rating:0.19

Sentiment scoring:

ID: Test_Img_089 Anger rating:0.27 Happiness rating:0.39 Sadness rating:0.30 Neutral rating:0.44 Fear rating:0.23

Pix2Pix Model Training

Emotion Express App (Demo)

Training Result

Pix2pix Loss over time

Sentiment

AR Cube

A AR product to protect the lost cultural heritage

Date: 10/2021-12/2021

Team Member: Xinyi Zhou, Xing Hao

Role: Technology research, Innovation and technology planning, 70% XR content production

As one of the oldest city in China, Beijing has the most complicated city water system, which has witnessed the development process of this historic city. Many culture meme, such as Hutong(name of traditional streets) and Obon(a traditional a traditional Buddhist and Taoist festival), are related to the water system. However, many people do not know these knowledge and cultural significance. This project aims to popularize the knowledge and history of water system of Beijing.

Media technology can build multiple virtual worlds in parallel for different groups of interest, encouraging creation and recreation without the constraints of time, space, and social class. This project abstracted the characteristics of Shichaha, a manmade lake lies in the center of Beijing, and combined augmented reality with paper box in order to allow people experience the ancient activities related to water and arouse people's awareness of protecting the water environment.

AR

Product

Huitong Temple

There is a large water gate under Huitong Temple with movable wooden gates embedded in the stone troughs on both sides. The water level and flow rate of the entire Beijing waters are controlled. Therefore, it is called the throat of the capital waters.

Underground Water Gate

Fire Temple Dojo

On June 23rd, all fire temples in the capital will hold grand avenues.

Di’anmen Fire Temple is the largest of the fire temples in Beijing. The Taoist priests wore robes and vestments and held law enforcement instruments or musical instruments when doing the dojo.

Obon

The Buddhist community held a grand Obon Festival on the Lunar New Year's Day on the 15th day of the lunar calendar. There were mass activities such as the lighting of river lanterns and burning of boats.

Shicha Waters Culture

The Shicha sea area was the terminal of the Beijing-Hangzhou Grand Canal in the Yuan Dynasty. It was the basis for the planning of the capital city of the Yuan Dynasty.

Making Process

STEP1: MAKING

Use the Vuforia platform in Unity to link virtual assets and physical objects and divide them into three levels of content for presentation: 1. Single object target 2. Multiple objects target 3. Scene target. To produce a better product experience for consumers and further understand the rich Shichahai water cultural content.

STEP3: DISSEMINATION

Package the designed AR Cube and Map into "AR CUBE" products for sale and dissemination. To publicize and promote the disappearing water culture of Beijing and let more people learn about and protect Beijing water culture through this exciting product.

This app has two main functions: 'Start' and 'About'. 'Start' is a camera mode UI, through which users could scan the cubes and the map. Then the 3D model of the sites including Huoshen Temple, Guanghua Temple, Yindian Bridge and Huitong Temple will be displayed. 'About' allows users to get the details of the sites directly.

The combination of the cubes and the map will show different folk activities related to Shichahai

Scan object

Scan 1 Cube

By scaning individual cube on the map, users can get detailed knowledge of the architecture and its components related to shichahai and water culture

Scan 2 Cubes

The combination of two cubes and the map will show the users the relevant cultural activities about this two buildings as well as the landscape around Shichahai.

Scan 4 Cubes + Map

If all of the four cubes and the map are scanned, the Obon Festival, the most celebrated festival during the history of Shichahai, will be displayed.

Toward a New Era: Virtual Gallery

New attempts with two virtual gallery including Pingshan Virtual Gallery & Virtual Gallery at Sansar

Location: Shenzhen, China

Date: 08.2020 - 12.2020 & 06/2021-08/2021

Team members & Role: Each part has a specific description

The form of Contemporary art can be diverse. The transformation of art in different spaces will give the audience a different experience and inspire them to think about the meaning of work from another perspective. Technology is one of the most important labels of Shenzhen. This young city has the best soil for new media art due to its advanced high-tech manufacturing abilities in this background. When the COVID-19 ravaged the world in the first half of 2020, the past life patterns of human beings were affected entirely and changed. This project discussed how traditional art could break through the limitation of space and region to continue the dialogue with the audience. Virtual reality technology and Metaverse platform may be the answer.

preservationAttracttovirtualexhibitions

Meta Quest (wireless streaming) uses positioning cameras to identify the environment and assist in positioning , which leads to the need for visually different designs on the walls around the exhibition hall so that the headset is easier to locate.

In-site photo of Virtual Gallery

Pingshan Virtual Gallery - Multiplayer experience

experience

The virtual art gallery exhibition hall built at Pingshan Art Museum is equipped with three Meta Quest 2 devices, which can support multiperson experience and is the physical entrance of the virtual art gallery.

Sansar Experience Center, for users to choose their favorite activities and participate

Enter the buffer gallery of the virtual art gallery to give users time to prepare

Specific exhibition space, taking the work "Meat" as an example

to Entrance Hall

The eight virtual spaces are all designed according to the same space experience system, including spaces with Original artwork space, Transfer space, View platform, and Main exhibition and other functions Exhibition hall entrance platform: users can choose their favorite exhibition to go

Pingshan Virtual Art Gallery Virtual Gallery Entrance Hall

Exhibition

Sansar Hall

Virtual Gallery Entrance Hall

Spatial Sequance | Virtual Gallery at Sansar

STEP3: Interaction STEP4: Communication

Production of Pingshan Virtual Gallery Technology Support | Wireless VR Streaming

2D Paintings 2D Videos

STEP2: 2D Artworks to 3D Virtual Experience Wireless VR streaming STEP1 | 2D Artworks: Eight Chinese artworks from DSL Collection In the context of the global pandemic of covid-19, collectors of DSL Collection hope to bring back the original collection of artworks (in the form of oil paintings, videos, etc.) to people in a new way.

Re-creation of eight independent physical works of art into the content of the virtual exhibition hall, creating a virtual art gallery through the production of new visual systems and interactive methods

Interaction:

Teleport：

Human-interface interaction: For example, in this experience, the user can hit a 3D object to make the object produce a unique movement trajectory

Audio: Text introduction voice, BGM, etc., rendering a better immersive experience

Text: Give the work a basic background introduction

3D Virtual Experience: Physical Exhibition: Avatar-AvatarInteraction Human-Avatar Interaction 6

4 7 5 3 8

1 2

The

virtual spaces are all designed according to the same space

experience system. The set experience sequance

Interactive Target of Entrance

Entering from the exhibition hall to each exhibition sets interactive animation content, prompting the user to enter a new exhibition

UI Design

Guidance system: including UI design of origital artwork, transfer space, main exhibition, viewing space

Single Exhibition

With eight recreated artwork experiences in a row, this section takes the shock of time as a example to explain what kind of interaction , text, sound, and animation have integrated in the space.

Teleport Space

The set experience sequance will be presented one by one with the narrative experience

Profassional On Going Work:

“Digital Animism”:

A web-side application for disseminating interactive educational content.

[Web/Mobile App - Educational Platform]

Location: Online, China Date: 07/2022-Now Team Member: Xinyi Zhou, Xiaogang Lian, Ziyan Zheng

Role: Innovative & technonical planning, Project management, Interactive deisgn & engineering

In this project, we want to achieve educational equality through a way of more accessible information transmission. In China, the gap in educational resources between large cities and villages is huge, and it is also hard to obtain high-quality educational contents through traditional educational assistance like book donation and teaching support. Therefore, based on the construction of this platform, we call on more talents with high educational backgrounds to research and develop interactive courses for “Digital Animism ”, which include diverse themes such as space exploration, environmental protection, and ancient Chinese architecture, hoping to gradually promote education equality with new technologies.

Link: https://v3d.net/dq7

Space Science - Mars

On going content...

Link: https://v3d.net/e1j

Link: https://v3d.net/b2j

Link: https://v3d.net/bit

Environmental Protection and Outdoor Construction

On going content...

https://v3d.net/dgh

On going content...

On going content

Reseach:

Interplay between the virtual and physical heritage

Using MR technology to protect architectural heritage. [Technologies: Hololens, Arduino]

Location: Shenzhen, China

Date: 11/2021-Present Team Member: Xinyi Zhou, Fukai Chen, Zhouyi Li, Chaohe Lin Role:

80% Innovation & technology planning, 50% Virtual Content Production

Cultural heritage contains multiple components, including tangible culture (buildings, monuments, etc.) and intangible culture (history, stories, music, etc.). The virtual and physical features together define the local identity and spirit of place. Immersive technology has much accelerated the digital transformation of heritage. Lots of researches in cultural heritage based on VR, AR, and MR have proved the feasibility. MR is defined as combining virtual elements (e.g., holograms or projections) with the real world. Compared with AR, MR can achieve a broader goal of enhancing our virtual and physical environment understanding.

We proposed a new method of interaction between the virtual and physical heritage in an MR application , aiming to break through the interplay limitations. This method is based on the integration of environmental sensors, MR technology, and cultural heritage . This integration provides an opportunity to utilize immersive and sensing technology to improve the participation and understanding of visitors.

InteractiveVisualisation feedback

Hololens

Arduino

InterfaceInput Input Interface Unity

Experiment:

Profassional Work:

Taida AR Exploration

Enhance the fun and exploreability of a tour. [AR, App]

Location: Tianjin, China

Date: 03/2021-07/2021

Team Member: Xinyi Zhou, Rufeng Xiang, Mengqi Tu, Haida Sun

Role: Innovation & technology planning, Content Production, UI Design

This project is a large-scale commercial project, which includes the AR exploration experience of aircraft carrier park, popularization of aircraft carrier knowledge, and cloud tourism. My job was mainly responsible for the innovation and technology planning of each AR project and the design of UIUX [Unity, C#, 3ds Max, Adobe XR, App development]. It gave me a new understanding of innovation and technology from a business perspective.

Acknowledgement: I would love to thank all the people who helped the portfolio production, including all colleagues of the Configreality, brother Zhang Xu, teammates: Xing, Ziyu, Han, Xinyi, Zhouyi...Thank you for your suggestions and criticism. Let the portfolio keep getting better and better. Thank you, my parents and friends, for your support and company, which keeps me moving forward and full of motivation.