THE CITY IS WHAT YOU WANT YOUR CITY TO BE -- YOUR WORLD, YOUR LIFE, YOUR CITY
YANG XINYI B.E ARCHITECTURE M.S URBAN DESIGN
AI DRIVEN SMART LIFE CITY | DESIGN | TECHNOLOGY There are a thousand Hamlets in a thousand readers’ eyes. Similarly, it is difficult to define people’s ideal happy life as it varies from person to person. Therefore, I want to spend my life creating solutions to gives people more freedom to self-define their own life while respect their own privacy. These devices would build on the spaces and services we already have, combing them with the best in research of urban environment, latest technology in artificial intelligence, and precise analysis of individual needs.
Personal Branding
Personal Home VR Tour
CONTENTS Personal Sketch Drawing
PERSONAL CITY IMAGE
PERSONAL AI DRIVEN URBAN NAVIGATION SYSTEM
Personal City Exploration System
PERSONAL EMOTION PAINTER
11
PERSONAL ARTIST COMPANION
18
CITY @ MY HOME
26
EXPERIMENTAL RESEARCH & DESIGN
33
AN AI DRAWING MACHINE OF SELF EMOTION EXPRESSING
AN ARTIST COMPANION ROBOT OF FACILITATING PERCEPTION
A REDIRECTED WALKING FRAMEWORK IN HOME SCALE VR Personal City Exploration View
2
RESPONDING URBAN CONTEXT WITH DIFFERENT SPATIAL STRATEGIES
Personal City Image
C
G
CITY IMAGE
V
STREET VIEW
PHYSICAL WORLD
I
VIDEO RECORDING
INDIVIDUAL CITY IMAGE
IMAGE SEGMENTATION COMPARISON
O
OBJECT DATA ANALYSIS
P
PREFERENCE
N
NAVIGATION SYSTEM
PERSONAL EXPERIENCE
SMART DEVICES
Main Topic: Personal AI-driven urban navigation system Tools: Google Street View API, MIT ADE20K dataset, Open-CV, Semantic Segmentation model, QGIS, Auto CAD, Adobe Suite Type: Individual Work Problem Statement: Existing navigation maps and route planning Apps provide recommendation routes based on the fastest routes or the main routes between two destinations. However, in real-world city wandering, people tend to explore the interesting points based on their own visual preference instead of walking through the fastest routes in the journey. People are more likely to lose the interest in their journey. Proposed Solution: Personal City Image is an offline plug-ins which is embedded in navigation systems or route planning apps. Instead of choosing the fastest routes for travellers, it can detect people’s visual preference and give recommendation routes so that it can bring the joy of travelling in the city. Together with AR mask, it can show more about the information and details about a specific places which people are visually interested.
Market Research
Defined Problem
The market for the navigation systems(Such as Google Maps) is growing at a significant rate in the emerging economies such as Asia-pacific, owing to a rise in the GDP and population in these regions, resulting in improvement in lifestyles, increased purchasing power of consumers, and development in road connectivity and infrastructure. In fact, people cannot go out without the help of navigation systems in an unfamiliar space, they need the navigation systems to tell them the particular positions, the travelling routes and so on.
We saw a tension between the want to customize their own journey and fixed navigation recommendation patterns of city travellers. We divided the relocate and narrow down the problems into six stages: Main issue and situation when city exploration, the walking preference of each individual, elements in walking preference, personal visual preference, ways of evaluation and the final route generation.
1 billion
200 million
Monthly Active Users
Situation When City Exploration
Places
Walking Preference
Elements of Walking Preference
Elements of Visual Preference
Evaluation
Route Generation
Besides the fastest route, people also consider other elements about a street, such as the greenery, safety and the sky coverage in an urban street.
Walking preference includes a lot of subjective elements and each people evaluate in different ways.
People make decisions based on what they see in the street.
Subjective assessments cannot be applied to general consideration.
The elements of the street is constantly changing because of the continuously movement
After we can correctly evaluate and give street score based on visual preference, how can we apply the data to the real situation?
There are many street elements and it is hard to find out their importance in making decision when city wandering.
Problem People not always want to choose the fastest route when they travel in an unfamiliar city.
Positioning
Way-finding
Navigation systems can help user quickly know the precise position at present and clearly show the situation of the surrounding. It contains maps, which may be displayed in human-readable format via text or in a graphical format.
Geo Information
These systems help people find their way from a starting point to an ending destination. To utilize the system, drivers need to provide the address of where they would like to go. Usually, it can provide the fastest route for users.
Most devices allow you to locate restaurants, gas stations, rest stops, hotels, and various attractions along the way. It is comforting to know that if you are running low on gas or need to find a room for the night.
Sometime fastest route can prevent people finding more interesting places in certain area.
Interview and Insights Two groups(Total 30 people) are set up for compare the difference between the people using experience and the people using GPS navigation toolsGoogle Maps) when exploring the streets in Central, Hong Kong. In the research, each person should draw a mental map which describes what they remember during the journey and then answer one questionnaire. “Matthew effect” happens in the group which people use navigation system: people are more likely to navigation “large” landmark and ignore small attractions and details of the city. In general, people remember different elements of the city and have different perspective when wandering in the city. 90% of people want to know more about the space which they are interested but get nothing through the navigation system. Higher Frequency
Difficult to capture every customers needs and preference about the street.
Even though elements of walking preference of each individual are known, it is very difficult to get individual information to analysis.
It is difficult to find equations which can represent the visual effect of each element in the city.
How to get people’s feedback when they finished wandering in a street?
WHO HOW WHY WHEN WHAT How might we help travellers personalize their own travel route based on their visual preference when they wander in the city?
Higher Frequency
Design Concept
District
Main Rd
Alley
Secondary Rd
Unclassified
Lower Frequency
Lower Frequency
Elements in Mental Map
Routes in Mental Map
Name: John Age: 45 Profession: Teacher Group: Group A
Name: Lucy Age: 23 Profession: Student Group: Group B
Name: Wilson Age: 31 Profession: Manager Group: Group B
“I can find the famous landmark very easily with the help of Google Maps and went directly to them.”
“ By asking the stranger about how to walk to certain point of the city, I knew very interesting things.”
“It is very interesting to walk through the city where I haven’t explored yet without the help of Tech.”
Higher Frequency
“Even though I used the Google Maps, I still cannot remember clearly about the place I went through.”
Edge
Lower Frequency
Name: Katie Age: 25 Profession: Student Group: Group A
Nodes
Higher Frequency
Lower Frequency
Interviewers travelled with Google Maps(Group A) Interviewers travelled without Google Maps(Group B) Samples of Mental Maps
Landmark
By using artificial intelligence, we can get proportions of each elements of the street in physical world. And by getting the individual focus points when they travel, we can compare these data with the physical world to help us identify what is people’s visual preference. TARGET USER
People want to tour the city without time limitation. They want to explore the city in the way they want and they want to know much about the places when they wander in the street.
DEFINITION
Personal City Image is an offline plug-ins which is embedded in navigation systems or route planning apps. Combined with the AR mask, people can see what they want to see in the city.
INFORMATION ARCHITECTURE
Home
Explore
Trips
Save
Trip 01 Routes Route Attractions Explore Result
Save Route
Account Profile Settings Notifications Help
Name
Log Out
Type
Privacy
Create Route
Introduction
Terms of Use
Route Recommendation
Others
About the App
Attractions
Product Feature 010110000010010 10010101010101 001100101000100 101010101010100 101010101010101 000000011011011 111110010101010 100101010100101 01010101010101 01001010101010
Real-time Way-finding
Vision Analysis
Street Score Evaluation
Real-time way finding, allows to automatically identify and track the location of people in real world. And it can provide recommendation route without any delay.
Vision analysis allows users to know better about their visual preference when walking in the street. It can also help system to get feedback about chosen street.
Street Score Evaluation allows users to review the situation of each street in different evaluation ways. Before start travelling, users can modify their choices.
GIS System
Data Mining
Path Recording
Open Street Map
Google Street View
Researcher: Random Walk
Central, HK
API Key
Go-pro+ GPS Recorded
osm. File
Load and Read File “ HK_Central.csv” “HK_Route.csv”
PgSql Processing
Calculate the frameto-frame transform
GIS : Points along GeomRequest URL
Points( Lat, Lng)
Manual Screening
HK_Central CSV. File
20,000 SVI (HK Central)
x,y,x Transform/ K- slope
Street View Database +
+
+
5110 pic
620 pic
103 pic
52 pic
30pic
23 pic
SVI (Personal Track)
SVI (Personal Focus Point)
SVI (Interest Point: Small)
SVI (Interest Point: Middle)
SVI (Interest Point: Large)
Input
Customized AR Mask
Respect Privacy
Let the user explore travel experiences in cities by providing general attraction information and details.
The Customizer allows you to preview changes and the style of AR mask before purchasing it. It allows you to show your own personalities through our products.
Sophisticated technologies operate offline in personal cellphone. Users don’t need to worry about their information which would be uploaded in the cloud.
Business & Marketing Strategy
4. Placement stage: At this stage, more detailed rules will be considered, such as the land use types of certain points.
Convolutional Encoder-Decoder Pooling Indices
People Courtyard / Plaza / Park /Plant Sculpture/ Bridge / Plant
Product Marketing Model
3. Rank stage: Based on the recalled position pool, generally a sorting will be done here;
Different Physical Feature
Hard scape/ Green scape People, Transportation/ Others
This offline navigation recommendation plug-ins will be embedded and partner with maps and travel software/ App. Through strategic partnering with up and coming brands, we have a precise and high-quality recommendation algorithms, which enables us to offer accurate, unique, and “preference-first” personalization experience on our platform.
2. Recall stage: Based on personalized information, follow a certain strategy to a position candidate set for route match, and recall locations that meet the threshold or conditions;
Calculate
Walking Route
No Exploration
Interest Point
Interest Point (Small)
Interest Point (Middle)
Interest Point (Large)
Image Analysis [1]
Partnership Model
1. Trigger stage: This is generally based on user behaviour data mining, using user pin to trigger the calculation of user-related personalized information, and use it for recommendation strategies;
Output
Video Analysis
Partnerships
There are two main business-side strategic innovations in Personal City Recommendation System supporting our user flow.
Referring to the marketing strategy of the advertising, every successful route recommendation is optimized and adjusted for the next route planning. It allows us to target users’ vision preference more and more precisely.
+
SVI (HK Central)
Image Segmentation Useful Information
Video Frame
Video Frame
Users (Travellers)
Sky Wall/ Fence/ Ceiling/Building Plant In this model, the landlord and the owner of the space can help to improve their vision elements and style for target customers.
Navigation Maps/ Travel Route Planning Apps Delivery Strategy
Location
Landlord/ Owner
Proportion
Image-ability
Enclosure
Recommendation System City Wandering
New Navigation Area
Personal Dataset
Street view Dataset
Smart AR Mask/ Navigation App
Sidewalk/ Road Skyscraper Signboard/ Street light
Human Scale
People Bicycle/ Mini-bike/ Car/ Bus Tree/ Street light
B Fastest Route
Complexity
A
Interest Point
B
Interest Point
×
A
√
Delivery Effect
Technical Architecture : AI - Navigation Navigation Recommendation System (Our Product)
Three computational models were been selected to complete the whole process for comparing differences between physical world and personal city images. There are three main parts in this research: 1. Getting data from Google street view to analysis the proportion of physical environment; 2. Getting data of individual focus points; 3. Make a comparison between these two parts. Note: [1] The street based assessment system was based on the below research: Measuring Urban Design: Metrics for Livable Places, Otto Clemente and Reid Ewing.
P .I. C | Research Process and Analysis Training Database —— City Street View Segmentation Model Measuring the human sense of place and quantifying the connections among the visual features of the built environment that impact the human sense of place have long been of interest to a wide variety of fields. Previous studies have relied on low-throughput surveys and limited data sources, which have difficulty in measuring the human perception of a large-scale urban region at flexible spatial resolutions. In this work, a data-driven machine learning approach is proposed to measure how people perceive a place in a large-scale urban region. A series of statistical analyses was conducted to determine the visual elements that may cause a place to be perceived as different perceptions. From the 15 object categories segmented from the street view images, various objects were identified in different proportions.[1]
Complexity
Feature Proportion
8%
Feature proportion can efficiently reflect the street visual situation in data. In this research, the street elements are divided into 15 categories and then evaluations (Complexity/ Enclosure/ Image-ability/ Human Scale can be analysis by calculating certain feature from the Google street view.
82
The complexity of a place depends on the variety of the physical environment, specifically the number and kinds of buildings, architectural diversity and ornamentation, landscape elements, street furniture, signs and human activity.
Image-ability
79
Image-ability plays to the innate human ability to see and remember patterns: a place whose elements are easily identifiable and grouped into an overall pattern: -Landmarks (singularity and location) - Sense of place: a characteristic visual theme
Enclosure
68%
In an urban setting, enclosure is formed by lining the street or plaza with unbroken building fronts of roughly equal height. The buildings become the “walls” of the outdoor room. The street and sidewalks become the “floor”; and if the buildings are roughly equal height, the sky projects as an invisible ceiling. The total width of the street, building to building, should not exceed the building heights in order to maintain a feeling of enclosure. Visual termination points also contribute to a sense of enclosure
Note: [1] The similar street segmentation model can be found the below research: Measuring human perceptions of a large-scale urban region using machine learning, FanZhang, BoleiZhou, LiuLiu, YuLiu,Helene H.Fung, HuiLin, Carlo Rattic
Human Scale
75%
of physical elements that match the size and proportions of humans and, equally important, correspond to the speed at which humans walk. Building details, pavement texture, street trees and street furniture are all physical elements contributing to human scale.
P .I. C | Research Process and Analysis Step 01: Navigate the city with Gopro Camera Recorded videos
Scene
AVI
Step 02: Cut the videos into pieces
AVI
IMG
Step 03: Calculate the frame-to-frame transform Vertical Horizontal Rotation IMG
Step 04: Analysis frame to frame transform data |K| <10 X 10<|K| <20 Y 20<|K| <30 Z 30<|K| <40 |K| >50
Frame to Frame Transform ( X , Y , Z) (Sample)
The essence of video jitter is that the image has small, random direction and high frequency motion. First, the direction of motion between image frames must be detected. Any object in the image usually contains unique features, but it is often composed of a large number of pixels. The corner point is a small set of points that can accurately describe the object. The corner detection algorithm can analyze the most obvious feature points of the image for object recognition and tracking.
Step 05: Select the useful data and tag them with GIS information
[1]
X Y Z
Lng/ Lat (114.1564 22.285156)
SVI
Strategy of Choosing the Focus Points Transform Slope -K (Sample) “K” represents the changing ratio (Fast or Slow) of object position. By analysing the vertical, horizontal and rotative changes, we can notice the degree of concern when city wandering. After the data was checked and analysed carefully, it shows that:
×
√
√
√
Stable
Horizontal Movement
Vertical Movement
Rotation Movement
Video frames did not change, which means people keep walking and look forward. Therefore, it is not the focus point.
When people pay attention to something up/downward, they change position of their heads. Therefore, it is the focus point.
When people pay attention to something left/ right, they change position of their heads. Therefore, it is the focus point.
When people pay attention to something in diagonal, they will change position of their heads. Therefore, it is the focus point.
Note: [1] The background image of Hong Kong view was drawn based on the below image website: https://www.pinterest.com/
|K| <10 Unaffected Point Video frames change little, because when people are walking, the camera vibrated. Therefore, it is not the focus point. It is the unaffected walking.
10<|K| <20 Focus Point
When people pay a little attention to some interesting spots, their head will move slightly. Therefore, it can represent the focus point of interviewer.
20<|K| <30 Focus Point
When people pay attention to some interesting spots, their head will move. Therefore, it can represent the focus point of interviewer.
30<|K| <40 Focus Point
When people pay attention highly to some interesting spots, their head will move intensely. Therefore, it can represent the focus point.
|K| >50 Unaffected Point
When people change their directions, the camera frames will change a lot. Therefore, it is not the focus points.
P .I. C | Research Result and Analysis
5110 IMGs Not Interested
Not Interested No Navigation Street
Interested Navigation Routes
Interested Foucs Points
620 IMGs
Navigation Route Feature Proportion Green Scape People/ Detail Transportation Hard Scape
103 IMGs
Evaluation
Researcher’s Focus Point
Maximum Value Average Value Score: 0.5/ 1
Imagination Human Scale Complexity Enclosure
Cluster 02 In this group, the interviewer was walking among the busiest street in Central. And there were most POI in this area. Moreover, the block scale here is smallest.
3 LEVELs
Researcher’s Focus Point
Cluster 03 In this group, Researchers are attracted by the people cluster in the River Bank. He watched people’s activities on grass. Interviewer was interested in people activities.
Proportion of Feature: Hard scape has dominated the most part of urban environment. Each features are average in this section.
Proportion of Feature: Hard scape has dominated the most part of urban environment. There are seldom transportation feature.
Proportion of Feature: People/ Details , Hard scape have dominated the most part of urban environment. Green scape has the highest score.
Evaluation: All four domains don’t show obvious bias. They got lowest score compared with two above.
Evaluation: All four domains don’t show obvious bias. They got average score compared with two above.
Evaluation: All four domains don’t show obvious bias. They got highest score compared with two above.
Little Interested Focus Points
Cluster 01 This is located inTai Kwun Centre for Heritage and Arts. The interviewer stayed a long time watching the exhibition. He was interested in art event.
Fairly Interested Focus Points
Most Interested Focus Points
Cluster 04 In this group, Researchers are attracted by the people cluster near the Central Garden. There was a Beauty Show and competitors were dancing on the stage. Interviewer was interested in people activities.
N
Research Street View
Focus Point (Large)
Navigation Route
Focus Point (Middle)
Focus Point
Focus Point (Small)
In this research, street views are divided into three groups: 1. Not interested Route (Which researcher did not choose to navigate); 2. Interested Route (Which research chose to navigate); 3. Interested Points (Which research paid attention to when navigation). And in last group, it is also divided into three groups: 1. Little Interested; 2. Fairly Interested; 3. Most Interested. And below, by analysing different feature proportion in these groups, we can make some conclusion about the perception preference of this researcher.
Proportion of Feature: Hard scape, People/ Details has dominated the most part of urban environment. Transportation get highest score in this section.
Proportion of Feature: Hard scape has dominated the most part of urban environment. Green scape get highest score in this section.
Evaluation: Enclosure gets high score. Complexity performs badly in this section.
Evaluation: Human Scale get high score. Image-ability/ Enclosure perform badly.
Proportion of Feature: People/ Details has dominated the most part of urban environment. Evaluation: Image-ability / Complexity/ Human Scale get high score. Interviewer seldom concerned about Enclosure.
P .I. C | Further Application Customized Appearance
People can design the mask in the way they like. Therefore, it is a way to show their own personality and increase the joy in city navigation.
Personal Preference
Micro Camera
This micro camera can record the journey when traveller visit the city. It can become an interesting memory of people’s life. More importantly, it can send information to the computer to analysis people’s preference in city navigation.
Physical World History Receive Store
Present Scene
Next Scene
Receive
Similar
……
Search
Analysis
Indicate
AR Information
Computer Database Reflect
History Segmentation Data Hardscape Greenscape People/ Transport Others
Search
Protection Mask
Machine Learning Model
Personal Preference Enclosure Human Scale Complexity Imageability
Image Segmentation Road Sidewalk Building Wall
Vegetation Terrain Sky Person
Fence License Plate Polegroup Traffic Light Traffic Sign
Rider Car Truck Bus Train
Search Route
Unlike the traditional AR glass, it has the protective mask so that people can protect themselves from dust and wind.
Information about the focus point and the route of the whole navigation process can be show in the AR glasses.
AR Glasses
GPS System
Lng 114.1564 Lat 22.2851 Ang 154.1164
AR glasses provide a way which people no need to look at the phone and can overlay the visual information on the physical environment.
Compare [1]
[1]
Navigation System Conceptual Model Principle
Navigation System Conceptual Model Structure
Based on the above research, there is a big deviation between the real physical world and personal preferences. Therefore, using this kind of deviation between people and the physical world, we can try to provide people with a better city navigation system that connects personal preferences with the real conditions of the physical world.
In this navigation system, besides the navigation app, people can equip with the AR mask to help them better city navigation. It has the customized appearance so people can design the mask in the way they like. Most importantly, in can provide information and guidance based on individual visual preference. Unlike the traditional navigation city, everyone has their unique routes.
Note: [1] The background image of the man portrait was drawn based on the below image website: https://www.pinterest.com/
P .I. C | Application Samples
B
F A E
C
D
[1]
[2]
A
D
B
F
C
G
Navigation System Recommendation Map Unlike the traditional navigation system, this recommendation map is based on individual preference which can be figured out on the above research. And people cannot just choose the fastest way from one destination to another, and they can choose the city view they like and get more interesting experience during the whole journey. Moreover, the potential POI will be show in the app so that people can click to get more knowledge about them.
Note: [1] The background image of the man/ lady portrait were drawn based on the below image website: https://www.pinterest.com/
Personal Emotion Painter FORM OF EXPRESSION
P
PAINTING
G
GENRE
CLASSIFICATION MODEL
I
INDIVIDUAL EMOTION
STYLE TRANSFER MODEL
D
DRAWINGS FACIAL RECOGNITION MODEL
HUMAN PERCEPTION
S
SELF-EXPRESS
P
PERSONALITY
PERSONAL IDENTITY
Main Topic: An AI drawing machine of self emotion expressing product for personal branding Tools: Data Scraping Code (Baidu Image/ Google Image), ResNet-50–Convolutional Network for Classification and Detection, Conditional Adversarial Networks for Pix2Pix Drawing, Facial-Emotion Recognition Model, Adobe Suite Type: Individual Work Problem Statement: In Maslow’s hierarchy of needs, every people have needs for self-esteem. People develop a concern with getting recognition, status, importance, and respect from others. Everyone expresses their emotions in different ways, some use words, some use actions, some use objects, some use art and so on. For many people, it is very difficult to express love or hate (the information they want to express)directly and precisely. Proposed Solution: Emotion machine painter is a series of daily appliance which can reflect people’s emotion in real-time using art-work style. By using artificial models such as style transfer, facial recognition model and machine learning classification model, daily personal belongings can change to different artistic style, to help people to set up his/her personal brand or easily self-expression. Messages can be more understandable for each other.
Public Research
Data Mining Baidu Image
Painting Websites
Flickr API Key
Group01
Tags = “Painting” “Chinese Paintings” “Modern Paintings” “Western Paintings” ...
Positive Negative
Group02
Division
Database
Request URL
21,500 Paintings
1,500 Paintings
Database 02
Cloud Uploaded
Video Frame
Camera Recording
Positive
Emotion Recognition Model Supervised
Scoring Connection Layer
Block
ConvS1
ConvTi
... ConvTj
... ConvN
... ConvNj
FC
Flatten
AVG POOL
ID BLOCK *2
CONV BLOCK
G
Original
Network
Paired Paintings
Completed
Classified Paintings
Selection System Angry
Fear
IS_Block DS_Block
...
ID BLOCK *5
D
Happy SS_Block
CONV BLOCK
ID BLOCK *3
CONV BLOCK
ID BLOCK *2
Classified Database
Negative
D
Generator
Digital Device
Discriminator
Pix2Pix Model
User Emotion
Facial Image
ReLU
Stage 01 Stage 02 Stage 03 Stage 04 Stage 05
User Sketch
User Sketch
CONV BLOCK
256*256Pix Paintings
Max POOL
CONV
Zero PAD
Classification Model (ResNet-50) Input
20,000 Paintings
256*256Pix Paintings
Batch Norm
Manual Screening
Angry Nocebo Mode
Neutral Sad
Happy Fear Neutral
Placebo Mode
Surprised
Sad Surprised
Technical Architecture : AI - Self branding
Training Database
Three computational models were been selected to complete the whole process for creating artworks for each individuals. There are three main parts in this research: 1. Classification Model provided the basic database (Artwork style - Emotion); 2. Facial Recognition Model ( How people feel in reality); 3. Gan Model (Change individual sketches into certain style artwork.
Dataset of 21,500 pieces of art (mostly paintings) has been downloaded from Internet. The pieces of art were selected and divided into different categories ( including Art Informel, Lyrical Abstraction, Pop Art, Colour Field Painting, High Renaissance, Realism ect.) All this artwork are tagged with certain labels, including Name, Artist, Time, Style and ID number.
P .E. P | Research and Data Collection Step 01: (Tagging Paintings) ID:001 Title: The Boulevard Montmartre on a Winter Morning Year: 1897 Artist: Camille Pissarro Style: Impressionism [1]
Title: Sail boats at Sunset Year: 1864 -1930 Artist: Ferdinand du Puigaudeau Style: Impressionism Happiness Rating: 0.7 Anger Rating: 0.05 Fear Rating: 0 Sadness Rating: 0 Surprised Rating: 0.2 Neutral Rating: 0.05
[1]
Title: Untitled Year: 1903 - 1970 Artist: Mark Rothko Style: Abstract Art Happiness Rating: 0 Anger Rating: 0.4 Fear Rating: 0.2 Sadness Rating: 0 .2 Surprised Rating: 0.2 Neutral Rating: 0
[1]
Title: The Scream Year: 1893 Artist: Edvard Munch Style: Expressionism Happiness Rating: 0 Anger Rating: 0.3 Fear Rating: 0.6 Sadness Rating: 0 Surprised Rating: 0.1 Neutral Rating: 0
Step 02: (Research Classification) Happiness Rating: 0.2 Anger Rating: 0 Fear Rating: 0 Sadness Rating: 0 Surprised Rating: 0 Neutral Rating: 0.8 Step 03: (Defined Painting - Emotion) (ID:001) Emotion: Neutral
[1]
Surrealism Abstract Expressionism Romanticism Northern Renaissance Early Renaissance Impressionism
Drawing ID
Art Informel Lyrical Abstraction Pop Art Colour Field Painting High Renaissance Realism
Year
Neo-Expressionism Post-Impressionism Cubism Minimalism Cubism, Expressionism Rococo
Expressionism Baroque Abstract Art Neoclassicism Neoclassicism,Romanticism Magic Realism
Happy
Note: [1] The famous paintings collected on the below website: http://commons.wikimedia.org/wiki/Category:Google_Art_Project
Angry
Fear
Sad
[1]
Title: The Old Guitarist Year: 1903 Artist: Pablo Picasso Style: Modern Art Happiness Rating: 0 Anger Rating: 0.1 Fear Rating: 0.2 Sadness Rating: 0 .7 Surprised Rating: 0 Neutral Rating: 0
[1]
Title: Les Demoiselles d’Avignon Year: 1907–1907 Artist: Pablo Picasso Style: Cubism Happiness Rating: 0 Anger Rating: 0.3 Fear Rating: 0.2 Sadness Rating: 0.1 Surprised Rating: 0.4 Neutral Rating: 0
[1]
Title: The Temptation of Christ Year: 1854 Artist: Ary Scheffer Style: Romanticism Happiness Rating: 0.3 Anger Rating: 0 Fear Rating: 0 Sadness Rating: 0 .05 Surprised Rating: 0.05 Neutral Rating: 0.6
Surprised
Neutral
P .E. P | Machine Learning Classification Result
Happy Angry Fear Sad Surprised Neutral
Rating:0.1 Rating:0.2 Rating:0.4 Rating:0.6 Rating:0.8 Rating:1.0
Surrealism Abstract Expressionism Romanticism Northern Renaissance Early Renaissance Impressionism
Art Informel Lyrical Abstraction Pop Art Colour Field Painting High Renaissance Realism
Neo-Expressionism Post-Impressionism Cubism Minimalism Cubism, Expressionism Rococo
Expressionism Baroque Abstract Art Neoclassicism Neoclassicism,Romanticism Magic Realism
Surrealism
Art Informel
Neo-Expressionism
Expressionism
Abstract Expressionism
Lyrical Abstraction
Post-Impressionism
Baroque
Romanticism
Pop Art
Cubism
Abstract Art
Northern Renaissance
Color Field Painting
Minimalism
Neoclassicism
Early Renaissance
High Renaissance
Cubism, Expressionism
Neoclassicism, Romanticism
Impressionism
Realism
Rococo
Magic Realism
Emotion Proportion in Each Drawing Style · Impressionism and Post-Impressionism mostly can give people the Emotion of Happiness. · Northern Renaissance and Early Renaissance largely give people the Emotion of Sadness. · Surrealism, Art Informel and Expressionism largely give people the Emotion of Fear. · Minimalism, Colour Field Painting, Abstract, Cubism and Pop Art give people the Emotion of Surprised. · Early Renaissance and Neoclassicism partly give people the Emotion of Anger.
P .E. P | Facial Recognition Sample and Analysis Step 01: (Defined Emotion Feature) Face Screen shot: 20* 6 Emotions Mouth Width/ Mouth Height:/ Brow Slope/ Brow Height/ Brow Width/ Eye Height Comparison: Ave/ Middle No. /Mode Step 02: (Facial Feature Analysis) Mouth Width: *** Mouth Height: *** Brow Slope: *** Brow Height: *** Brow Width: *** Eye Height: *** Step 03: (Defined Face - Emotion) (User: 001) Emotion: Happiness
Happiness
Mouth Height >= 0.03 Eye Height <= 0.056 Mouth Width >= 0.4
Anger
Mouth Height <= 0.03 Brow Slope<=-0.30 Brow Height <=0.13
Note: [1] The faces with emotion were collected on the below website: https://www.pinterest.com/
Fear
Mouth Height >= 0.05 Eye Height <= 0.056 Brow Slope >=-0.2
Sadness
Mouth Width <= 0.36 Mouth Height <=0.02 Brow Slope <= -0.40
Mouth Width: Happiness: 0.48(Related) Anger Rating: 0.30 (Related) Fear Rating: 0.34 （Unrelated) Sadness Rating: 0.34 (Related) Surprised Rating: 0.31 (Related) Neutral Rating: 0.35 (Related)
Brow Height: Happiness: 0.14（Unrelated) Anger Rating: 0.12 (Related) Fear Rating: 0.09（Unrelated) Sadness Rating: 0.12 （Unrelated) Surprised Rating: 0.13（Unrelated) Neutral Rating: 0.11（Unrelated)
Mouth Height: Happiness: 0.12 (Related) Anger Rating: 0.06（Unrelated) Fear Rating: 0.13 (Related) Sadness Rating: 0.01 (Related) Surprised Rating: 0.10 (Related) Neutral Rating: 0.01(Related)
Brow Width: Happiness: 0.40（Unrelated) Anger Rating: 0.38（Unrelated) Fear Rating: 0.36（Unrelated) Sadness Rating: 0.39 （Unrelated) Surprised Rating: 0.38（Unrelated) Neutral Rating: 0.39 (Unrelated)
Brow Slope: Happiness: 0.30（Unrelated) Anger Rating: -0.40 (Related) Fear Rating: -0.10 (Related) Sadness Rating: -0.50 (Related) Surprised Rating: -0.50（Unrelated) Neutral Rating: -0.30 (Related)
Eye Height: Happiness: 0.04 (Related) Anger Rating: 0.05（Unrelated) Fear Rating: 0.08 (Related) Sadness Rating: 0.04 （Unrelated) Surprised Rating: 0.07 (Related) Neutral Rating: 0.06（Unrelated)
Surprised
Mouth Height >= 0.03 Eye Height >=0.056 Mouth Width <= 0.35
Neutral
Mouth Height <= 0.03 Brow Slope>=-0.30 Mouth Width >= 0.33
P .E. P | Gan Training Process and Application
Anger
Batch_size: 1 Beta1: 0.5 Continue_train: False Crop_size: 256 Dataset_mode: aligned Direction: AtoB Display_env: main Left to Right: · epoch153_real_B Display_freq: 400 · epoch153_real_A Display_id: 1 · epoch153_fake_B Display_ncols: 4 Display_port: 8097 Display_winsize: 256 ID:899 Epoch: latest Title: Venice, The Epoch_count: 1 Pink Cloud Gan_mode: vanilla Artist: Paul Signac Init_gain: 0.02 Style：Pointillism Init_type: normal Year：1909 Input_nc: 3 IsTrain: True Left to Right: [Default: None] · epoch171_real_B Lambda_L1: 100.0 · epoch171_real_A Load_iter: 0 · epoch171_fake_B Load_size: 286 ID:056 The Lagoon of Saint Mark, Venice Artist: Paul Signac Style：Pointillism Year：1905
Fear
Gan Training
However, sometimes people might have difficulties in expressing themselves to others, which leads to the unhealthy relationship with people who care about them. In this design, people just need to upload the sketches they drew can it can be turned into an artwork which can reflect people’s emotion.
Surprised
Step 03: (Pix2Pix Model Training)
Neutral
PS
For all relationships, in order to increase mutual intimacy, we will take the initiative to express some of our emotions, attitudes, concepts, and feelings to each other. People on the platform will chat, but the people at the dinner table still must talk about life. Because through this way of sharing information with others, we can get the other’s response. When each other feels the other’s understanding, belief, identification, and concern, the relationship will deepen.
Happiness
Step 02: (Training Pairs Prepared)
Application: An Artwork Which People Can Use to Self-express
Sadness
Step 01: (Set Up Dataset01) Emotion：Happiness Artist: Paul Signac Style: Impressionism - Pointillism Painting Number: 1000 Pic
Dataset 01_Pix2Pix_Loss_Over_Time
STEP 01:
Sketches uploaded Note: [1] The faces with emotion were collected on the below website: https://www.pinterest.com/
STEP 02:
Face Recognition
STEP 03:
Style Chosen
STEP 04:
Apply Objects
P .E. P | Further Application
Jewellery
Decoration
Clothes
Furniture
Digital Device
Wallpaper
Home Appliances
Invitation Card
Daily Supplies
Note: [1] The background images were collected on the below website and redesigned by the author: https://www.pinterest.com/
Personal Artist Robot
S
SKETCH DATASET
D
DRAWING STAGE
S
I
SKETCH BEHAVIOUR
INDIVIDUAL PERCEPTION
S
SENSORY PROCESSING
VISUAL DEVELOPMENT
PHYSICAL DEVELOPMENT
E
EVALUATION SYSTEM
Main Topic: An artist companion robot of facilitating children’s perception of graphics and drawing behaviour Tools: Sketch-RNN Model, Google Speech API, Google Natural Language API, Quickdraw Dataset, Tensorflow object detection API, Adobe Suite Type: Individual Work Problem Statement: Children follow rules and principles when they are in traditional drawing lessons. They are taught to draw a certain object under the guidance of teachers. Actually, it is a devastating destruction of children’s creativity, imagination, and future abilities. Children might lose the interest in drawing in the process. Therefore, new ways should be found out to help children to improve their perception while maintain the interest in drawing. Proposed Solution: Personal artist companion is a smart robot which aim at augmenting children’s perception in their early stage. It based on the study of children drawing habits in different ages and let children freely draw based on their cognition. It doesn’t aim at improving the drawing methodologies, but aim at improving different skills and abilities such as mathematics and language. Moreover, it has cute and attractive appearance.
Increasing Children’s Perception by DrawingAi
Children
Machine
Object
Children
Species/ Groups
Order
Machine
Machine
How to classify objects? Improve children’s power of choosing a similar attribute
Geometric Combination
Position
Machine
Machine
How can geometries create an object? Form children’s Complete understanding of certain object
Children
<Mirror>
How to change the structure of object? Enhance children’s ability of imagining different visual perspectives
Machine
Children
Machine
Children
Machine
Children
<Three Flowers>
Children
<A frog is in front of a flower>
<Elephant>
Transform
Children
<Marine Animal>
What can this feature represent? Enlarge children’s knowledge of different kinds of objects Machine
Machine
How to put objects in logical way? Help children have better understanding between each details of an object Children
Children
Scale
How to measure object? Enhance children’s ability to measure objects’ size
Machine
Children <In the evening, the monkey is watching at the squirrel eating strawberry.>
<Flower> Form - Vocabulary
What is the relationship between object and vocabulary? Improve children’s language skill
Scene - Sentence
How to depict scene in language? Improve children’s ability of observing scene and vocabulary organization
Note: [1] The face of the little girl was collected on the below website and redesigned as a part of the image: https://www.pinterest.com/
How to put objects in logical way? Improve children’s ability of solving mathematics problems Machine
Children
Scene
How to depict the background of scene? Enhance children’s ability of imaging Machine
Children
<In the evening, the monkey is watching at the squirrel eating strawberry.>
<Hot>
<Twice Larger>
What are relationships between objects? Help children developing their fine motor skills
Calculation
Feeling - Object
Sentence - Context
Machine
Machine
What is the object Constancy? Improve capacity to understand that “out of sight” doesn’t mean “gone.”
Children
< A pig >
Complement
How to describe an object in comprehensive way? Enhace children’s memory of objects
How to organize objects based on description? Improve children’s organization ability
Colour
Children
What are the colors of objects? Help children’s identify the each colours of each object
Sketch Database 354 Species of Sketch
Data Processing Each Sketch is classified and tagged with different property
Data Manipulation
50 million Sketch
User Input
Attribute
Propotion
Position Type Complexity
Context Feeling Domain
S
S
S
S
User Input
S
S
S
Z
S
S
Z
h
Z
S
S
y
T
h
Z
S
y
T
h
Decoder RNN
h
y
T
Decoder RNN
h
y
T
Decoder RNN
y
T
Decoder RNN
h
S
Decoder RNN
Backward Encoder RNN
Backward Encoder RNN
Backward Encoder RNN
h
S h
S Forward Encoder RNN
h
h
S Forward Encoder RNN
Random Sketch
h
h
S
Forward Encoder RNN
S
Forward Encoder RNN
S
h
Forward Encoder RNN
h
Backward Encoder RNN
Backward Encoder RNN
Generative Vector Image Model
Z
S
Object Detection System P*Q
M*N
Feature Map
3*3
Sketch Based on Requirement (Object)
Reshape
1*1
Softmax
Reshape
Proposal
Softmax
1*1
User Input
Sketch Evaluation System
Sketch Based on Requirement (Picture)
Sun/ Nature
(3,3)/ Small
Sky/Background
Hot/ Sunny
Attribute/ Type
Coordination/Area
Position/Context
Feeling
Text2Image Generation Model (A) Text Encoder
(B) Image Encoder
(C) Recurrent Module
(D) Attention Modules
(E) Object Prediction
(F) Attribute Prediction
User Input (G) Foreground Embedding
…
Input Sentence
Phone Recorded
Canvas
Recurrent Hidden State
Language Context
Application Children Learning App
＋ Artist Robot Companion
bbox_pred
ROIPooling
Scribble Stage (18months - 3 years old) Pre-Schematic Stage (3-5 years old) Schematic Stage (5-8 years old) Preteen Stage (9-11 years old)
Object One-hot
Location Attribute Map Maps
Foreground Patch
<<Sun>,<Tree>,<Flower>> Voice Recognition
Sketch - Perception Relationship
Visual Attention Visual Discrimination Visual Memory Visual Spatial Relation Ships Visual Sequential-Memory Visual Figure Ground Visual Form Constancy Visual Closure
Geometry Drawing Object Recognition Image Description Imagination
Technical Architecture : AI - Education
Training Database
In this research, there are four main parts : 1. Reclassified the data so that I can execute a call in each computational model. 2.User interface which people can draw on the screen of phones or I-pads. 3. Sketch-RNN generation model and text2Image generation model (which are both the open source on the Internet). 4. Evaluation model which can give feedback of the drawing performance.
The training dataset is from the Google Quick-draw and it is a collection of 50 million drawings across 345 categories. The drawings were captured as timestamped vectors, tagged with meta-data including what the player was asked to draw and in which country the player was located.
P .A. R | Data Classification, Analysis and Visualization Raw Data from Google Quick-draw Dataset [
[ // First stroke [x0, x1, x2, x3, ...], [y0, y1, y2, y3, ...], [t0, t1, t2, t3, ...] ], [ // Second stroke [x0, x1, x2, x3, ...], [y0, y1, y2, y3, ...], [t0, t1, t2, t3, ...] ], ... // Additional strokes ]
Step 01: Calculate the drawing stroke number/ total drawing time (with pause time)/ drawing time
Drawing Time
Step 02: Evaluate drawing complexity based on the above three elements Step 03: Rank the complexity based on type
By carefully observing the structure of the dataset, we can figure out the average stroke number/ drawing time/ drawing total time. In this research, I concluded that if people draw with more strokes and draw in a longer time, this pattern is difficult for them to draw.
Data Reclassification
In this research, the Quick-draw dataset will be used to evaluate whether children drawings are correct. Therefore, before setting up the computational model, each sketch will be reclassified into different categories. After children draw, the system can give the feedback whether it is right or wrong. Moreover, the dataset will also be used to generate images and scenes.
Drawing Complexity (Ranked by Type) 01 Abstract (8)
02 Natural (25)
Drawing - Context 01 Not-defined (19)
03 Food (29)
02 Background (22)
Drawing - Position 01 02 Water Ground (19) (98) Drawing - Proportion 01 Large (39)
04 Object (173)
03 Sky (4) 02 Middle (47)
Drawing - Feeling Relationship 01 02 Mood Weather (2) (8)
05 Vehicle (24)
06 Animal (51)
03 Forefront (304) 04 Air (21) 03 Small (251)
05 Not-defined (203)
Vehicle Abstract Animal Body part Location Food Nature Object
04 Not-defined (8)
03 Not-defined (335)
Drawing - Colour 12 Not-defined (272)
Drawing Stroke Drawing Complexity Result When the dataset is used in improving children’s perception, it is very important to figure out whether they understand concepts of each objects. Therefore, it is useful to rank the difficulties and complexity of each objects so that children can select what they understand and start to sketch. In this model, I analysed the drawing stroke, the drawing time and the drawing total time (including pause time) to evaluate the drawing difficulties.
P .A. R | Data Classification, Analysis and Visualization
Sort By Type
Sort By Context
Sort By Position
(Vehicle/ Object/ Food/ Location/ Nature/ Animal/ Abstract/ Body)
(Forefront/ Background/ Not-defined)
(Water/ Ground/ Sky/ Air/ Not-defined)
Sort By Propotion
Sort By Object - Feeling Relationship
Sort By Colour
(Large/ Middle/ Small/ Not-defined)
(Weather/ Mood/ Not-defined)
(Red/ Blue/ Black/ Purple/ Green/ Yellow/ Pink/ Brown/ White/ Grey)
P .A. R | Increasing Children’s Perception by DrawingAi Stage 01: Drawing as listening narrative story
Stage 02: Drawing as mission challenging
At the early stage of children, they do not have the ability to draw something. Therefore, in this stage, children can sketch whatever they want and then the machine will finish the drawing. Moreover, the machine can tell what it draws in an interesting way.
After children have the ability to draw something, the machine will invite children to complete several tasks (including different visual skiils). And the machine can adjust the difficulties based on children's performance.
Training Samples
Training Samples
Stroke 01
Sheep
Bird
Pig
Cat
Angle
Top
Bottom
Left
Right
On
Over
Stroke 02
Bird
Bee
Bus
Rabit
Basket
Number (Three)
Couple
Twice
Half
Dozen
One Third
Stroke 03
Cactus
Crab
Monkey
Spider
Skull
Rotate
Mirror
Array
Scale
Projection
Similar
Stroke 04
Bridge
Toothbrush
Pinapple
Duck
Light-house
Hot
Cold
Cloudy
Windy
Happy
Scared
System
Audio
Human- Computer Interation Process Interface
Children Input
System
Machine Generation
System
Machine Chosen
Human- Computer Interation Process Audio
Audio
Monkey
Monkey Story
Explanation
StoryTelling
Note: [1] The faces of the little girls were collected on the below website and redesigned as a part of the image: https://www.pinterest.com/
Audio
Interface
(Apple) Correct
Rotate the apple "A c t i o n" - - " O b j e c t " Machine Question
System
Children Input
Evaluation 01
(Action) Correct Evaluation 02
Correct Machine Output
P .A. R | Increasing Children’s Perception by DrawingAi Stage 03: Drawing as story expresser
Stage 04: Drawing as depicting scene
When children get older, they begin to learn how to express themsleves in language. Therefore, in this stage, the machine would provide multiple scene and evaluation children's performance when they describe the picture.
The last but not least, when children has the overall abilities of visual perception, the machine will ask them to draw a picture based on there descroption. And the machine will evaluate children's performance at last.
Training Samples
Training Samples
01 There is a house next to the grass and a tree. It is a c l o u d y d a y.
05 Three sheep are running in the rain.
01 A rabit is watching at the carrot u n d e r t h e s k y.
( Tu p l e s ) rabit carrot sky/cloud under/below
02 The elephant is standing by the kangaroo under the sun.
06 It is a cold d a y. I b u i l t a snowman just n o w.
02 There is a bottle of wine and a glass on the table.
( Tu p l e s ) wine glass table on
06 We need to brush our teeth with toothpaste and toothbrush
( Tu p l e s ) teeth/tooth tooth-brush tootth-paste
03 The croccodile is running after the fish under the boat i n t h e w a t e r.
07 The monkey is watching at the banana on the grass.
03 A cow is eating grass under the sun.
( Tu p l e s ) cow grass/flower sun/sunny/hot under/below
07 If we eat too many burgers, we will get sick and drink h o t w a t e r.
( Tu p l e s ) water/cup hamburger sick/ill/ syringe
04 The bus stops at bus station and it cannot catch up with the plane.
08 It is a sunny d a y. A c o u p l e o f b e e s i s fl y ing over four fl o w e r s .
04 The girl is watching at the clock and waiting for a phone.
( Tu p l e s ) clock/time gril/woman/ lady Phone/call
08 I worked with my computer and my pencil under the lamp late at night.
( Tu p l e s ) lamp computer/ desktop book/paper pen/pencil
Human- Computer Interation Process System
The bus stops at bus station and it cannot catch up with the plane. Children Input
System
Machine Generation
System
Machine Draw
( Tu p l e s ) birthday cake shirt
05 It is my birthd a y. M y m o t h er gave me a cake and a new shirt.
Human- Computer Interation Process Audio
Audio
A bus stops at bus station and a plane i s i n s k y.
Correct
Childen Input
Evaluation
Note: [1] The faces of the little girls were collected on the below website and redesigned as a part of the image: https://www.pinterest.com/
Audio
Interface
A cow is eating grass under the sun. Machine Question
Children Input
System
System
( Tu p l e s ) cow grass/flower sun/sunny/hot
( Tu p l e s ) under/below
Evaluation 01
Evaluation 02
Audio Correct Machine Output
P .A. R | Further Application Drawing Ai based on children drawing habit development [1] Stage 01
Stage 02
Stage 03
An artist companion
Stage 04
Besides turning this model into software, it is more interesting to change it into an artist robot to company children growing up.
End-effector
12 Months
Develop motor coordination Scribbles
2 Years-old
Separate lines and circles Predict direction
4 Years-old
Draw hands and feet Use language to describe
5.5 Years-old
Start using the base line Basic 2D plane world
This end-effector can carry different types of pens and can draw together with children.
Camera
Cameras can detect what children have drawn and then make response to children’s action.
18 Months
Begin traversing movement With the elbow as the axis
3 Years-old
Freely draw circle/line Give meaning and stories
4.5 Years-old
Draw various forms Lining up like a catalogue
6 Years-old
Abstract and pattern things Show detail drawing abilities
Micro Computer
20 Months
Develop continuous circles From large to small circles
3.5 Years-old
Head portrait with dots /lines Have stories in drawings
5 Years-old
Draw division of area Begin methodical expression
7-8 Years-old
Draw from different angles Draw people with actions
Conceptual App model for drawing Ai
Step 01 Welcome Page
Step 02 Stage Chosen Page
Step 03 Drawing Page
Soft Robotic
Inspired by Chinese Traditional Lantern, the robot-body can be designed like the lantern. In this way, the robot can not only have the cute appearance but also can rotate 360°
This micro computer can process children’s drawing and make feedback to children immediatly.
Step 04 Extension Page
Note: [1] The image of the drawings from children were collected on the below website and redesigned as a part of the image: https://www.youshibaodian.com/a/a77281a406da488aad875fd7d815875b.html
[2] Note: [2] The image of conceptual soft mechanic system (folding behaviour) referred to the below website: https://www.pinterest.com/
City @ Home VIRTUAL WORLD
REAL WORLD
N
I
NAVIGATION EXPERIENCE
INDIVIDUAL MOVEMENT
V
VIRTUAL REALITY
T
H
TRAVEL ROUTE
HOME LAYOUT
R
REDIRECTED WALKING
P
PERCEPTION ILLUSION
UNLIMITED WALKING IN LIMITED SPACE
Main Topic: A redirected walking framework to assist people explore cities in home scale VR Tools: Data Scraping Code, Flickr API, Unity 3D, Auto CAD, Adobe Suite , Sketch Up, Reactive Algorithms (Steer-To-Center Algorithm and Steer-To-Orbit Algorithm), Predictive Algorithm( Zig-zag Technique) Type: Individual Work Problem Statement: Due to COVID-19, people have to stay home for a long time and are not allowed to go outside for outdoor activities. Moreover, travelling aboard is time-consuming and requires a lot of reparations. Therefore, there is an increasing need in virtual tour. Lots of VR programs about VR museum tours or city tours appear in the market but none of them can really give people walking experience like in the actual world. Proposed Solution: City @ My Home is a home VR walking framework which help people to have urban experience by using redirected walking algorithms at home. By analysing walking patterns at home, this walking framework can help to solve out correspondent relationships between certain space at home and exact GPS location in urban street. Besides, people are not only allowed to travel but also get more knowledge during virtual tours.
Oculus released the Oculus Rift S and a standalone headset, the Oculus Quest. These headsets utilized inside-out tracking compared to external outside-in tracking seen in previous generations of headsets.
The 2000s were a period of relative public and investment indifference to commercially available VR technologies.
In 2020, Oculus released the Oculus Quest 2. Some new features include a sharper screen, reduced price, and increased performance. Facebook now requires user to log in with a Facebook account in order to use the new headset.
Google launched Street View, showing more and more panoramic views of the world, such as roads, buildings and rural areas.
Landmark Identification
Data Mining API Key Load and Read File “ Hong Kong””30km Area” “22.373648, 114.117801”
Landmark Ranking Local or Tourists
Request URL Downloading CSV Dataset
Route Identification
1,000,000 Message (Tags/ User/ Date)
Home layout Analysis Foundation of Oculus Rift
Route Classification
Popularity of road Ranking
1. Tourism Attraction 2. Eating & Drinking 3. Others
1. The user set of the images assigned to one landmark 2.The numbers of images taken by one user 3.landmark numbers
Development Ratio in VR Industry (Past Data and Future Prediction)
Route Generation
Education Industry Engineering Industry Game Industry Healthcare Industry Live Event Industry Real Estate Industry Retail Industry Travel Industry Video Entertainment
2002
2004
2006
Hotspot Detection 1. Choose the point by over ten users. 2. Choose the landmarks in the cluster 3. Most frequent tags with the names of candidate landmarks.
Data Cleaning
Flickr
Landmark Usability
HTC shipped its first units of the HTC Vive SteamVR headse
2008
2010
2012
2014
2016
Certain Road Landmark Points Calculation
2018
2020
2022
Stand and hand movement
Road Assessments
Assessment Road 01+ Road 2+ Road 03 ……
Landmark Point & Length
Total Route Accumulation among different Road
Redirected Strategy
Correspondence Principle Living Room - Circular
More and more VR products appear in every market and VR has becoming an important part in people’s daily entertainment. Each kind of industry wants to find the opportunity for combing VR and their traditional products. But on the above diagram, people tend to use VR in their ordinary days, which means that VR should be more portable and give more freedom in usage because of personal use.
Move in a limited room with cameras at the corner of the room
+
2024
Development of VR Application
Sitting and watching
Road between Two Landmark
Tourist Urban Street Building Building
15%
30%
Table Building Building
Corridor - Straight Walking Move without limitation in the house
Building
Building
Interior Space Room
Bedroom - L Shape Walking
Room
Room
Table
Rotation Gains
Curvature Gains
Arrangement Gains
Virtual Route
Practical Route
(22.373648, 114.117801)
(Room A)
Representative Object Object Object
Table Table
Zone
Zone
Kitchen - Z Shape Walking
Zone
Translation Gains
Home VR Tour
Cultural Zone Zone
Bed
75%
9%
Object
Degree of Freedom of Motion Using VR Equipment
Technical Architecture : AI - Visual Tour
Comparing with the traditional equipment, in this research, I want to further explore whether there are potential to get rid of the traditional boundary of VR and people can freely explore cities or play VR games in their home houses. Using the walking behaviour to indicate the virtural scene we can create.
In this research, volunteered Geographic Information(Using Flickr in this case) is used to identify travel journey in each city (Take Hong Kong as an example). Moreover, I analysis the route journey in a simple house layout plan ( Take One -bedroom Apartment as an example). At last, by using the redirected walking technique, I apply the travel route in room layout.
C .A . H | Home Redirected Walking Vision and Strategy
Using Home Waking Pattern to create City Exploration Experience
Redirected Walking Analysis in Different Scale (Unity Simulation)
The major issue of this research is that how to apply make a transformation between the physical data and virtual data using redirected algorithm. Not each strategy is suitable to certain type of urban experience.
THIRD PERSON VIEW
THIRD PERSON VIEW Physical Path
PERSONAL VIEW
VIRTUAL TOP VIEW
PERSONAL VIEW
VIRTUAL TOP VIEW
Human in City (>1KM WALK)
Human in City (>1KM WALK)
City Scale has huge difference with the apartment scale, people have to redirect many times using any algorithms, therefore, fail to apply redirected walking in this situation.
Different way should be found out to help people explore -- Using the transit point similar to many VR games. Certain points in reality can help people transfer suddenly to another space.
Virtual Path Virtual Path Physical Path
Physical Path THIRD PERSON VIEW
Popular Urban Street
Famous Architecture
Interior Space
Cultural Object
Exploring the urban street is the main way of knowing a city. It is the key issue this research should achieve.
Hotspots and place of Interest can be visited by tourists like the traditional travel process.
Unlike the traditional VR tour, people just sit and visit, people are encourage to explore the interior space in building.
When entering the building, tourists can get the chance to further study about the historical artwork in this city.
Conceptional Strategy of Redirected Walking [1] Based on the existing redirected walking research, I summarize the following strategies which I can use in this research. These methods can benefit: 1. Visually enlarge the limited space. As we all know, the space in an apartment is small and they are the same level of scale of urban street. 2. Unlimited walking by walking circularly. Using the perception of illusion, we can walk without the limitation of length.
THIRD PERSON VIEW
PERSONAL VIEW
VIRTUAL TOP VIEW
PERSONAL VIEW
VIRTUAL TOP VIEW
Human in Urban Street (100M WALK)
Human in Architecture (~10-50M WALK)
With around 6 rounds walking in the circle, people can finish walk in 100 meters. (Using the Steer-To-Orbit Algorithm).If the room size can larger than 5 meters in size, the longer people can walk in a circle.
With certain curvature gains and Rotation gain, people can walk at a right angle and each line length depends on the limitation room size (Apartment’s length is mostly 2- 5m).(Using the Zig-zag Algorithm)
Virtual Path
THIRD PERSON VIEW
PERSONAL VIEW
Arrangement Gains
Translation Gains
Curvature Gains
Rotation Gains
75% larger Compared to Original Scale
26% Upscale 14% Downscale
Arc radius >= 20 Meters 5m*5m Smallest Area
49% Amplification 20% Dampening
Physical Path THIRD PERSON VIEW
Physical Path
VIRTUAL TOP VIEW
Virtual Path
PERSONAL VIEW
VIRTUAL TOP VIEW
Human in Room (~3-5M WALK)
Human with Artwork (<1M WALK)
Space with 3-5 meters in length can help people to walk like the way when they visit in the room. Way people arrange the zone(Room) determine the actual path in reality.
Only space with 1 meters in length can help people to walk like the way when they visit the artworks. The way people arrange the artwork determine the actual path in reality.
Note: [1] The summary of redirected walking are based on the below research: M. Azmandian, T. Grechkin, M. Bolas, E. Suma. The Redirected Walking Toolkit: A Unified Development and Deployment Platform for Exploring Large Virtual Environments. Everyday VR Workshop, IEEE VR 2016
C .A . H | Road-based Travel Route Generation Using Geo-tagged Images [1] What Represent a City ? Using Hong Kong As an Example N
Landmark Classification
Landmark Ranking
Sai Kung District
3,050 Photos 98 Users 50 Travellers 48 Local People Tags: Natural/ Park/ Bird Sight-seeing/ Sea/ Forest/ Hill
View/
Yau Tsim Mong District
6,900 Photos 203 Users 156 Travellers 47 Local People Tags: Shopping/ Harbour City/ Shopping/ Vectorial Harbour/ Hong Kong Art Museum/ K11 Shopping Mall
Sham Shui Po District
6,001 Photos 150 Users 101 Travellers 49 Local People Tags: Night/ Shopping/ Food/ Beauty/ Old District/ Traditional Street
Not all flickr photos can be used to count the popularity of landmarks and roads. The photos from tourists can represent the tour situation. A tourist stays short time in a whole year, thus the taken dates of her images should be usually within the same month or at most two successive months.
Central and Western District
7,034 Photos 199 Users 100 Travellers 99 Local People Tags: Tramway Station/ Night/ Old District/ Food/ The University of Hong Kong/ Sea/ Sunset
To rank the landmarks, we need to quantify the tourism popularity of the landmark based on the geotagged images. There are two kinds of tourists with different knowledge of the landmark to certain space. Therefore, we should take into account whether the photo can represent the popularity of the place.
Road Ranking
Overall Ranking (Landmark+Road)
First, we should assign each flickr photos to the nearest road which can represent the location of the tourist. Similar to the landmark ranking, each photos should be evaulated to define whether they can represent the popularity of the road.
Both the popularity of the landmark and the road should be considered. After we seclect the important landmark which we use to create the destinations, we should evaluate each road between these two landmarks.
Eastern District
Tourist Photo
Resident Photo
In this research, more than 900,000 photos are downloaded from Flickr to study what impresses travellers when they travel to Hong Kong. Based on the data, we can further find out an appropriate travel route in our home VR application.
Top Landmark Identification
Flickr Images
4,300 Photos 103 Users 66 Travellers 37 Local People Tags: Causeway Bay/ Vectorial Park/ Shopping/ Night/ Food
Wan Chai District
9,000 Photos 200 Users 121 Travellers 71 Local People Tags: Central/Lan Kuai Fung/ Art/ Food/ Tai Kuan/ PMQ/ Food/ Bank
Road Popularity Calculation
Route Recommendation
Image Assignment
Landmark A
Landmark Identification Candidate Landmark Landmark Ranking Top Landmarks
Route (A,B) Road Popularity
Landmark B
Route in VR Route Landmark Interior Object
Note: [1] The mathematics calculation of popularity of landmarks and roads are based on the below research: Sun, Y., et al. Road-based travel recommendation using geo-tagged images. Computers, Environment and Urban Systems
C .A . H | Road-based Travel Route Generation Using Geo-tagged Images Flickr photos are used to identify what people like in the process of travelling. Therefore, the favourite objects and cultural stuffs can be displayed in the Home VR tour. In this way, people can have more in-depth knowledge about a city.
Although there are some photos in the eastern part of Hong Kong Island, they were taken by local people to record their life. Therefore, the eastern part of Hong Kong Island will not become a part of the tour. Tourist like to go shopping and have food in this area. Therefore, the typical photos of this area are related with food or goods.
It is the most representative location of the whole city. It contains the largest amount of photos in this area. People can identify Hong Kong with any building in this area. Therefore, in this design, this area should be included.
Hong Kong Ferris wheel Bank of China HCBC Building
People like to explore the small coffee shops and the bars in this area. Cafe shop Bar and the restaurant in this area A large amount of photos were taken in this area as a historical and art tours space. PMQ - Staunton Tai Kwun - Centre for Heritage and Arts
Western Landing Port Victoria Peak
People like the sunset and the sea view in this area. Sai Wan Swimming Shed Western Pcwa Landing Port.
Map of AOI and Route
Elevation
Representative Flickr Photos
Based on the previous research, I can find out that the most popular tour area is in Hong Kong island. Therefore, in the following research, I use Hong Kong island as a small sample to explore the potential of Home VR city tour. Area of Interest (AOI) are drawn on the above diagram and five tour sessions are divided based on their type (Commercial Tour/ Cultural Tour/ Seashore Tour, etc.)
C .A . H | Room Movement Pattern Analysis and Redirected Walking Strategy Home Waking Pattern Analysis
Step 05: Transit to Next Street
Step 04: Object Observation
Please Enter
Kitchen - Z Shape Walking
Representative Object
Table
Object Object
Table Object
Table
In order to create the feeling of seeing exhibtion, people would slight change their direction (20 degree) when they turn the direction in the ketchen. In this way, it is like continuing watching artworks in traditonal way. When they stop, they can go to the transit points in the
Toilet
Bus Station
Room
Bus
Step 03: Interior Zone Movement
This create a transition point to travel to another street and explore other urban streets.
Please Enter
Step 01: Urban Street Movement
Please Enter
Bedroom - L Shape
Cultural Zone Zone
Bed
Zone
Zone
Door of Interior Space
Zone
Certain Artwork
In this movement, people remain the same walking pattern as the physical world but have different images about the space. Step 02: Building Movement Famous Building
Please Enter
Living Room - Circular
Tourist Urban Street Building Building
Building
Building Building
Building
Corridor - Straight Walking
Interior Space
Table
Living Room
Corridor
Bedroom
Kitchen
Toilet
Circular
Straight
“U” Shape
“Z” Shape
Still
~100meters Walk Straight
~10 meters Corridor Walking
~4meters Room Walking
~2 Steps Walking/ Watching
Stand Still Waiting
Urban Street
Building
Zone
Object
Transit
5m*5m circular movement make sure people can have infinite walking in the urban street. Detail(Scale) should be further explored.
Physical Movement Path Visual Eyesight Area Eyesight of Certain Points Point Which Transit to Next Step Point Which People Focus Point
Room
When people change the direction(180 Degree), the VR change 90 degree so that people can walk like in corridor with mant rooms.
Home VR Journey (Virtual Redirected Walking) Based on the existing researches on redirected walking, I want to explore the potential of using different space at home to create different VR experience and people can use VR to explore the city in different scale. 1. The behaviour while people walk along the urban street is the most different from the pattern which people walk in inner space. As people cannot notice the rotation when they are walking, in this research, people will walk in the living room (5m *5m) to create infinite walking paths. 2. Slightly change the rotation angle which people might ignore during the journey. 3. Follow the physical exploration steps to create the virtural visit tour.
C .A . H | Apply City Data to Home Scale VR Scene (Sample) In this research, as the diagram shows, people spend most of their time in the living room and walk in circular, because many points of Interest are located in the urban street. But in the Hong Kong Convention and Exhibition Centre, people spend most of their time visiting the artworks. Therefore, they spend most of their time in kitchen walk in the “Z” way in this session. Recommendation Streets Tourist Buildings Representative Local Object Zone Tour Transit Point
Western District Public Cargo Working Area Central and Western District (Culture Area) Central ( Commercial Area) Central ( Historical Building Area) Hong Kong Convention and Exhibition Centre Eastern District (Commercial Area)
Sample of Redirected Route (5km) The sample shows the whole home VR journey in Hong Kong Island. Point of Interests (the points which people would visit) are shown in the there dimensional model on the right. 1. Each cluster is separated because the routes from one cluster to another is not the one which people really want to explore. In this way, people can decrease the time when exploration. 2. The numbers of Points of Interest has no relationship with the size of the cluster. For example, the Central and Western District cluster is small but there are many points which people can explore. The Eastern District cluster is large just because the Victoria Park covers a large area. 3. The Hong Kong Convention and Exhibition Centre is a simple building but there are a lot of historical artwork which people can explore in this building. Thus, it is a separated session.
POI AOI
LAT
22.28920523 22.29017521 22.29047273 22.29037899 22.28395475 22.28411218 22.28403678 22.28396698 22.28392826 22.28390992 22.28385184 22.28379376 22.28325372 22.28314368 22.28323538 22.28345140 22.28339943 22.28335562 22.28316202 22.28297046 22.28281354 22.28281354 22.28283188 22.28302446 22.28259345 22.28267394 22.28282067 22.28227859 22.28221491 22.28241258 22.28222204 22.28201621 22.28183688 22.28170238 22.28139465 22.28158825 22.28091167 22.28180631 22.28187967 22.28214868 22.28239934 22.28261128 22.28282526 22.28302089 22.28050918 22.28064572 22.28090250 22.28103700 22.28128766 22.28094733 22.28100032 22.28139567 22.28164022 22.28196425 22.28227604 22.28273661 22.28411014 22.28490898 22.28344987 22.28448918 22.28483969 22.28585047 22.28508831 22.28588715 22.28615207 22.28570374 22.28504755 22.28297708 22.28214562 22.28163207 22.28164430 22.27929661 22.28029520 22.28029927 22.28093918 22.28143235 22.28163614 22.28172174 22.28397971 22.28324200 22.28336020 22.28285481 22.28284258 22.27780076 22.27837138 22.27885030 22.27906021 22.27925789 22.27948818 22.27944742 22.27914580 22.27858129 22.27881769 22.27963287 22.27963083 22.28034003 22.28053567 22.28067833 22.28088008 22.28172785 22.28216192 22.28257765 22.28297504 22.28366996
LNG
114.1329458 114.1386631 114.1426582 114.1454420 114.1502453 114.1502508 114.1504711 114.1507970 114.1512110 114.1515943 114.1518552 114.1520523 114.1522704 114.1521272 114.1520204 114.1519312 114.1515998 114.1525677 114.1527009 114.1528254 114.1529025 114.1528188 114.1525424 114.1530137 114.1530500 114.1531932 114.1533451 114.1533517 114.1527114 114.1529129 114.1534294 114.1538753 114.1538181 114.1539590 114.1541881 114.1539965 114.1537520 114.1545030 114.1545449 114.1548069 114.1549787 114.1551571 114.1553201 114.1554963 114.1555205 114.1557253 114.1555778 114.1554941 114.1558266 114.1561768 114.1563618 114.1569322 114.1572604 114.1577405 114.1580664 114.1584034 114.1586060 114.1578660 114.1584738 114.1596279 114.1593416 114.1577295 114.1599362 114.1604692 114.1611695 114.1621209 114.1621870 114.1659662 114.1660147 114.1655742 114.1658385 114.1620461 114.1614514 114.1609361 114.1609229 114.1596367 114.1590200 114.1584430 114.1729829 114.1735071 114.1733309 114.1727275 114.1722518 114.1837612 114.1856993 114.1849593 114.1846047 114.1843449 114.1841400 114.1839242 114.1835564 114.1841621 114.1847677 114.1833582 114.1842986 114.1852809 114.1858469 114.1862807 114.1875118 114.1867630 114.1872630 114.1879413 114.1885734 114.1896613
Experimental Research & Design
E
ENVIRONMENT
C
CONTEXT
T
TARGET
RELATED FIELD
U
URBAN PLANNING
R
RESEARCH
G
GROUP PATTERN
A
ARCHITECTURE
I
INDIVIDUAL BEHAVIOUR
Main Topic: Urban Planning, Urban Design and Architecture Design Tools: Sketch-up, Rhino, Grasshopper, Adobe Suite, Auto CAD, GIS Type: Individual Work/ Collaborative Work Abstract: Spatial Design is a complex system. It involves human physiology, psychology, behaviour and the built environment, social and cultural customs, laws and regulations, and other interdisciplinary disciplines. Below shows how I did research to figure out phenomenon of human behaviour, how I organize and reallocate resources in large area, and how I deal with different architecture by using different strategies to in response to specific built context. 2.5 years working experience in architecture and urban planning industry helps me to master the basic patterns of interactions between people and environment. The smart living projects are all based on the actual problems which I found in the traditional industry and rigorous investigation which I did in the real world practice.
E | Research
The Impact of Hi-technology Navigation Systems on Three-dimensional Urban Design space in Hong Kong
Supply & Demand of Roadside Parking in Hong Kong Using GIS-Based Method
Starting with the introduction of relationships between human perception and the built environment, the research builds a link between virtual aspects and physical aspects of built environment. Based on analysis of Fahad Al-Harigi model, the research discusses technology usage in forming the city image and finds out what changes are in the five elements raised from it .
Each vehicle needs a parking space at its origin and destination, thus parking spaces play a vital role of in transportation system. I collected data of illegal on-street parking during 8.30-10.30 am and 5.00-7.00 pm of weekdays and weekends separately, and try to find out the effects brought by the unbalanced supply and demand of legal parking lots.
(School Dissertation - Distinction)
(Considering the Cases of Illegal Roadside Parking in Kennedy Town)
E | Urban Planning & Urban Design
Conceptual Urban Design for Shenzhen Special Cooperation Zone Central Area
Conceptual Urban Redevelopment for Le Vert de Maisons Zone, Paris
In this project, we create a blue & green system based on mountain - river fractal mode. The units are divided according to the water catchment line, , low impact infrastructure are deployed among each units. Confirm the main storm water system corridor and set water purify equipment along the corridor. The corridor also guarantees the biodiversity in the river, and create unique urban landscape as well.
The site is located in the countryside of Paris next to Seine River. Instead of incresing GFA, I proposed to change the old industry zone into a new technology hub. Using TOD model, the site can connect education clusters next to it, and public activities happened more frequently along the riverside. Moreover, three sites next to it can strengthen the competitive strength. More open space and public area were provided in the site.
(International Consultation Competition - Third Prize)
(School work / Individual- Best studio design)
E | Healthcare Facilities
Architecture Design for Hospital Complex Development, Shijiazhuang City
Design & Construction of community health centre cum Social Welfare Facilities
The Zhaohua Hospital Project includes a general hospital, rehabilitation hospital, O&G hospital, paediatric hospital, confinement centre, science research building, staff canteen and accommodation building. In this project, we tried to break out the strict principle of hospital organization and make it “organic”. Therefore, it provides large atrium, setback landscaping and gardens. Moreover, the back of house was connected in the basement level.
The site at Pak Wo Road is located in the North District of Hong Kong, part of HK’s new town developed from older generations. The proposed NDCHC is a reach out institution to provide long term, step-down and out-reach clinical services to the residents living in the area and other targeted group. In the project, vertical zoning of the building is clean and distinctive, throughout the ten levels of building, all functional areas occupy one or less than one complete floor plate separately.
(Schematic Design stage, Design Development stage, Construction Documents stage, Under Construction）
(Under Construction)
E | Cultural Facilities
Architecture Design for Science Museum & Archive, YiChang City, Hubei Province
Architecture Design for opera, Feidong City, Hefei Province
Yichang has been built on mountains since ancient times, and most of the memories of Bashu culture are related to the mountains and rivers in Xiling. In the archives and science museums design, I focused on how to express the Shan-shui culture of the region. The Science Museum demonstrates freedom, agility and the future. The entire design has a hollow courtyard where people can walk up the spiral escalator. The mountain-like archives respect the closed nature of archives’ collections.
In this design, I wanted to provide a “Stage” for local people in Fei-dong city. Fei-dong is an old city with thousands of cultures and traditions. The opera create “green slope” so that people can climb to the roof of the building to enjoy the beautiful scenery next to the opera. Moreover, water pool, large city plaza and sunken courtyards were also created to invite different activities here.
(International Consultation Competition - First Prize)
(International Consultation Competition - Third Prize)
E | Elderly Care Home & Welfare Centre
Redevelopment of Lam Tin Complex of Hongkong Society for Rehabilitation
Proposed Development for the Elderly at Clear Water Bay, Hong Kong
The project aimed at providing basic design direction for the government and helped client to get approval of building a rehabilitation centre. It calmly comprises: 1. Cared and attention home for a capacity of elderly for 400 persons with day care centre; 2. Hostel for moderately and severely mentally handicapped persons; 3. Integrated vocational rehabilitation services centre; 4. special child care centre. It focused on how to make clear transportation flow between different users.
The project focused on creating suitable living environment for elderly people. Loneliness and difficulties in participation in social activities are two main issues in old life. Clear hierarchy of living space is designed to guarantee both privacy and publicity. One-bed room and two-bed room maintain private space. Outdoor garden and living rooms in each floor provide space for group chatting and playing. Health centres at the bottom of the building provide social communication for more people.
(Technical Feasibility Study)
(Technical Feasibility Study)
E | Public Facilities
Architectural Redevelopment for Library in SCUT, Guangzhou
Architecture Design for Residential complex, Hong Kong
The site is surrounded by noisy highway so I want to design a single box to response to the complex situation. A large plate which contained both reading and storage space were created in every floor and people can have visual connection. The assistance function would be put in the back of the building and the hall would be separated from the main building. Inner gardens can provide additional lighting for readers.
The school work related with “living” and it led me to face the dilemma that all city have: large scale of city expanding, explosive speed of population, scarcity of resources, pollution, bad standard of rental house and so on. Finally, I chose Hong Kong, a typical city of modernization, to provide a high-density residential skyscraper which provide a movable living units in cities. A framework was set up to arrange these movable boxes.
(School work, Individual, Rookies’ Award for Architectural Students, China, top 100)
(School work - Individual)
E | Small-scale Facilities & Multimedia Display Design
Architectural Redevelopment for Canteen in SCAU, Guangzhou
Multimedia Design for Advertising Planning Program for Guangzhou T2 Airport
In this project, I tried to create a “silent” architecture. The site is next to a beautiful lake in the college. Therefore, there are no solid wall facing the lake side. People can open all the window next to the lake and enjoy the scenery of the whole school. And the building just have two floors, which made it into a low profile and it would not block the view of people on the other side of the lake.
The multimedia bran display area is located in the commercial corridor area of the airport after security check. It is a largescale brand display space where passengers can take advantage of multi-functional, multimedia, and multiple interactive experiences before boarding. The L-shaped adjustable interactive LED display screen was introduced into the exhibition space in a gentle way, changing the original founder’s regular space experience.
(School work - Individual)
(International Consultation Competition - First Prize)