Portfolio_2024_Mingyang Sun

MINGYANG SUN Selected Works 2019-2023

CONTENTS

01

URBAN ECHOES AI-Guided Exploration of Istanbul’s Cityscape Similarities

02

COGNITIVE LANDSCAPES Exploring Emotional Responses in VR-EEG Driven Art Therapy

03

RECITYGEN Interactive and Generative Participatory Urban Design Tool

04

SECTIONAL SAMPLING Controlled Generation of High-Resolution Architecture Design

05

SPATIAL SYMMETRY Algorithmic Exploration of Calabi-Yau Forms

06

INCLUSIVE AND CONNECTED LANDSCAPES For Self-Constructed Settlements in Quito, Ecuador

07

OTHER WORKS 3D Scan, Speculative Design, GIS Analysis, Community Engagement

2

URBAN ECHOES AI-Guided Exploration of Istanbul’s Cityscape Similarities Independent Research | Suzhou, China | Summer, 2023 Tutor: Han Tu My Role: 80% Concept Design, 30% Machine Learning, 90% Drawing Team: Yaluo Wang, Lingkai Wang, Shiqi Dai Confronting the challenge of understanding urban complexity, the Urban Echos project in Istanbul set out to reveal the historical connections of the city with Athens, Rome, Paris, and Bursa. The goal is to discern similarities across these cities by examining their urban characteristics in depth. This comprehensive analysis involves three key techniques: satellite images are processed using U-Net for segmentation, revealing urban textures like buildings, streets, green spaces, and water bodies. Street views are dissected using DPT model, segmenting images into elements such as buildings, streets, sidewalks, sky, and greenery. Lastly, the ResNet model is used to classify architectural styles, identifying 52 styles. These components are compared with similar elements in the other cities, using standardized scores to measure similarities.

This analysis shows that certain urban elements in Istanbul closely resemble those in the other cities. The Urban Echoes project provides insights into the complex structure of urban environments and enhances the understanding of residents and urban designers. By using machine learning and data analysis, the project sheds light on the detailed nature of urban features.

3

CONCEPT: HISTORY & SIMILARITY Conceptual Process

Urban Feature

Dataset

Model

Step 1

Step 2

Step 3

Satellite Urban Texture

Facade Architectural Style

Street View Street Features

Satellite images of 5 cities’ historical districts divided to 300m grids

1000 Google Street View images from 5 selected streets in each city

5000 Google Street View images of each city's historical districts

U-Net Image Classification

ResNet Architectural Style Classification

DPT Image Segmentation

Street

Categories

Green

Istanbul:

Istanbul:

Percentage of each satellite

Similarity Score

Sky

Road

Green

Others

Sidewalk

Building 18 Architectural Styles

Water

Building

Istanbul:

Percentage of each facade

Percentage of each street view

90%

13%

...

20%

80%

75%

...

10%

90%

62%

...

10%

90%

90%

70%

95%

95%

99%

60%

80%

80%

65%

50%

20%

Average percentage of each city

Paris

Bursa

Rome

Athens

Average percentage of each city

Paris

Bursa

Rome

Athens

Average percentage of each city

Paris

Bursa

Rome

Athens

100

69

Calcualte Standard Deviation of Each Category & Remapping Average Score 94

97

95

95

85

100

52

78

77

85

4

URBAN TEXTURE RESEARCH: SATELLITE Classification Method Satellite Classification Model: U-Net & Siamese Network

Data Mining: Scraping Satellite from Google Map

Istanbul

Athens - 3*3 km Sample Areas

3

32

Satellite Classification with U-Net

32

64

32

Average Similarity Results

Similarity learning with Siamese Network

32

Athens

Rome Crop into 10x10 grids as training dataset

64

Rome Istanbul every individual grid

64

8x8

512

16x16

32x32

256 256

16x16

Paris

64x64

128 128

32x32

64x64

128x128

128

128x128

64

Bursa

94.10

256x256

256x256

3 km

512

1024

Average

96.83 Similarity score for each grid

256 128

Bursa

Differencing layer

94.92

512 256

Conv 3x3 ReLu Copy MaxPool 2x2 Up-conv Pool 2x2 Conv 1x1

512

1024

Paris Each category of Athens, Rome, Bursa, Paris

Conv Network

Encoding

94.96

Classification Analysis

Athens

Istanbul

Building

Street

Green

Aerial Images of Other Cities Reflecting Back to Istanbul’s Map

Water

57 16

12

19

Rome

Bursa 0.9444

13 N/A

Bursa

0.9409

Rome 0.9508

60 31

Rome

Rome 0.9472

Bursa 0.9415

Rome 0.9488

53 27

8

12 Bursa

Rome

0.9411

0.9683

65 26

9 N/A Athens 0.9428

Paris

81

8

6

Paris 0.9215

5 Sun, Y., Bi, F., Gao, Y., Chen, L., & Feng, S. (2022). A multi-attention unet for semantic segmentation in remote sensing images. Symmetry, 14(5), 906. doi:10.3390/ Koch, G.R. (2015). Siamese Neural Networks for One-Shot Image Recognition. ICML deep learning workshop, vol. 2.

5

ARCHITECTURAL STYLE RESEARCH: FACADE Prediction Method ResNet Models Tryout (ResNet 34, 50, 101)

Data Mining: Scraping Facades Views from Google Map & Data Augmentation

CONV_1

Athens

CONV_3

ResNet 34

De-colorized

3*3, 64 *3 3*3, 64

Rome

ResNet 50

X 18 Styles

Bursa

CONV_3

1*1, 64 3*3, 64 *3 1*1, 256

Rotation Data Augmentation

CONV_4

CONV_1

1*1, 256 3*3, 256 *6 1*1, 1024

1*1, 128 3*3, 128 *4 1*1, 512

1*1, 64 3*3, 64 *3 1*1, 256

Crop

CONV_4

3*3, MaxPool, Stride 2

ResNet 101

7*7, 64 Stride 2

Line Romanesque Architecture ...

1*1, 512 3*3, 512 *3 1*1, 2048

FLOPS Average Poll, 1000-d fc, 3.8 x 10

83.43

9

CONV_2 CONV_3

Paris

CONV_5

Convolution +ReLU

...

1*1, 256 3*3, 256 *23 1*1, 1024

1*1, 128 3*3, 128 *4 1*1, 512

Max pooling

CONV_5

FLOPS

1*1, 512 3*3, 512 *3 1*1, 2048

Average Poll, 1000-d fc, 7.6 x 10

Fully connected+ReLU

72.46

9

Architecture

De-texturized

Georgian Architecture

Heading 2 = 90+a

9

CONV_2

7*7, 64 Stride 2

Heading 1 = 90-a

Average Poll, 1000-d fc, 3.6 x 10

70.81

Style of Image

Byzantine Architecture

Field of view = 90 m

3*3, 512 *3 3*3, 512

FLOPS

CONV_1

N

50

CONV_5

3*3, 256 *6 3*3, 256

3*3, 128 *4 3*3, 128

3*3, MaxPool, Stride 2

Scrape facades: Turgut Ozal Millet Cd Road

Art Deco Architecture

CONV_4

3*3, MaxPool, Stride 2

3 km

Confusion Matrix

Accuracy (%)

CONV_2

7*7, 64 Stride 2

Istanbul

Predication Accuracy

153

0

0

3

0

0

0

0

1

0

0

0

0

0

0

0

0

0

0

110

1

0

0

0

0

0

2

0

0

2

0

0

0

0

2

0

0

11

72

0

0

0

0

0

0

0

0

3

0

0

2

0

3

1

5

0

0

115

0

2

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

110

4

0

4

2

1

0

0

2

0

0

1

0

1

0

0

0

0

5

141

1

0

2

1

0

1

0

0

0

0

2

0

0

0

0

0

1

1

111

0

2

1

0

1

0

1

2

1

1

0

0

0

0

0

4

3

0

85

1

0

0

0

0

0

0

0

0

0

0

0

0

0

3

8

5

0

105

0

6

0

0

0

2

0

0

0

1

0

0

0

1

0

3

0

0

82

0

1

0

0

0

0

0

0

0

0

2

0

3

1

0

0

3

0

61

0

0

0

0

0

0

3

0

1

2

0

1

0

0

0

0

0

0

140

0

0

2

0

7

0

0

0

0

0

1

0

0

3

0

0

0

0

82

0

0

0

0

0

0

1

2

0

1

2

4

0

1

1

1

3

0

80

7

0

0

0

0

2

1

0

0

1

2

0

2

0

0

5

0

2

105

0

2

0

0

0

0

0

0

0

0

0

0

0

0

1

0

0

0

107

0

0

0

1

0

0

1

0

0

0

5

0

2

2

0

0

1

0

120

0

0

0

0

0

12

0

0

16

0

0

3

0

1

0

0

1

0

88

Predicted Style

Softmax

Similarity Analysis Palladian Architecture

Art Deco Architecture

Art Nouveau Architecture

American Craftsman

Romanesque Architecture

Chicago School Architecture

Athens

Byzantine Architecture

Rome Bursa Paris

Bursa

Paris

Paris

Athens

Bursa

Paris

Achaemenid architecture American craftsman style

Bursa

Art Deco architecture Art Nouveau architecture

Pa

ris Istanbul

Istanbul

Istanbul

Istanbul

Istanbul

Istanbul

Bauhaus architecture Beaux-Arts architecture

Istanbul

Byzantine architecture Chicago school architecture Colonial architecture Deconstructivism Edwardian architecture Georgian architecture International style Novelty architecture Palladian architecture Postmodern architecture

Bu

rsa

Ro

me

Romanesque architecture Tudor Revival architecture

Istanbul

Istanbul

Istanbul

Rome

Bursa

Athens Athens

Georgian Architecture

Achaemenid Architecture

Bursa

Paris

Rome

Paris

Tudor Revival Architecture Xu, Z., Tao, D., Zhang, Y., Wu, J., & Tsoi, A. C. (2014). Architectural Style Classification using Multinomial latent logistic regression. Computer Vision – ECCV 2014, 600–615. doi:10.1007/978-3-319-10590-1_39

6

STREET FEATURES RESEARCH: STREET VIEW Segmentation Method Data Mining: Scraping Google Street View Images

Istanbul

Image Segmentation Training Model: DPT Model

Athens: 3*3 km Sample Areas Downloaded 4860 Google street view images

3 km

Input

Fusion

Reassemble 32

Fusion

Transformer

Heading = 180-a

Resample

Residual Conv Unit

0.45

0.09 0.06 0.23 0.03 0.14

0.49

0.10 0.09 0.20 0.050.08

+

Residual Conv Unit

Paris

Project

Concatenate

Embed

0.10 0.12 0.20

Rome

Resample 0.5

Paris

0.12 0.07

0.39

Bursa

Project

2300 images

Field of view = 90

Green Road Sidewalk Others

Athens

Output Read

Bursa a

Reassemble 32

Building Sky

Transformer

2100 images

m

Fusion

Transformer

Rome

50

Reassemble 32 Transformer

1830 images

N

Average Segmentation Results

0.37

0.11 0.10 0.26

0.38

0.14

0.05 0.11

Head Istanbul

3300 images

Reassemble

Tokens

Fusion

0.08 0.08 0.20 0.02

Similarity Analysis 100

0.07 29 0.12 23 0.20 16

50

0.14 15 0.03 12 0.23 13

x 2100

0.06 93 0.09 41 0.45 30

34

Rome 0.08 28 0.05 13 0.20 16

x 2300 Average

0.09 100 0.10 45 0.49 18

64

Bursa 0.11 19

Building

Average

0.10 100 0.11 63 0.37 100

62

Sky

Green

Road 93

...

Building

85

41

30

13 Sky

Green

Road 100

80 ...

100

45 17 Building

Sky

Green

Road

100

0.05 13 0.26 10

x 3300

49

45

16

Athens

Average

...

Remapping

0.12 49 0.10 45 0.39 100

Calculate with each point of street view in Istanbul

x 1830 Average

77

69

63

...

50 11

Paris

Building

Sky

Green

Road

Ranftl, R., Bochkovskiy, A., & Koltun, V. (2021). Vision transformers for dense prediction. In Proceedings of the IEEE/CVF international conference on computer vision (pp. 1217912188).

7

SIMILARITY RESULT ANALYSIS

Athens - Street View Similarity

Rome - Street View Similarity

Bursa - Street View Similarity

Paris - Street View Similarity

Istanbul - Street View Similarity

Athens - Architectural Style Similarity

Rome - Architectural Style Similarity

Bursa - Architectural Style Similarity

Paris - Architectural Style Similarity

Istanbul - Architectural Style Similarity

Athens - Satellite Similarity

Rome - Satellite Similarity

Bursa - Satellite Similarity

Paris - Satellite Similarity

Istanbul - Satellite Similarity 8

TOP SIMILARITY SCORES MAPPING Bursa 94.1

Rome 94.6

35

Rome 94.5

Rome 94.4

Athens 93.0

Focus Area for Similarity Assessment Rome 94.5

Similarity Score Satellite

Architectural Style

Street View Athens

Rome Rome 94.3

Rome 94.2

Bursa

Paris Legend: Score Scale for Each Category

Bursa 94.1

Paris 93.6

Satellite: 0

100

0

100

Architectural Style:

Street View:

Highlighted Grid for Scores >93 9

COGNITIVE LANDSCAPES Exploring Emotional Responses in VR-EEG Driven Art therapy Independent Research | Suzhou, China | Summer, 2023 Tutor: Tu Han My Role: 100% VR Design, 40% User Test, Group Data Analysis Team: Hansen, Zihe Wang, Qingyun Liu This project addresses the escalating mental health crisis by integrating Electroencephalogram (EEG) technology with Virtual Reality (VR) in art therapy. Traditional art therapy faces challenges in medium selection and measuring progress, which are addressed by our VR-EEG platform. This immersive environment enhances therapeutic engagement and facilitates emotional expression.

The VR-based art therapy procedure unfolds in three phases: planning, creation, and sharing, each crafted to elicit specific emotional responses. Various VR scenarios, encompassing diverse contexts, textures, and models, were developed to explore their impact on participants’ perceptions and emotions. In this project, interactions of 12 participants within VR settings were tracked using EEG technology. The focus was on examining shifts in alpha, beta, and theta brainwaves, which are critical indicators of emotion states. Integration of EEG allows for real-time brain activity monitoring, offering deep insights into the emotional effects of the therapy. The project also includes a sharing platform, enabling participants to share art creations, give feedback, and communicate feelings with others.

10

RESEARCH BACKGROUND Emotion Problems

Research Process

Global mental health needs are surging due to social disparities, pandemic impacts, and modern stressors, with 40% of American adults facing mental challenges and over 275 million people globally suffering from anxiety or depression.

Creating

Planning

Style

Context

Ancient Painting

Natural

Model

Move

Rotate

Sharing

Delete

Review

Share

Review and Browse shared share your own scenes published scene created by others Realistic

Anxiety

Distraction

Comment

Comment and communicate with friends

Urban

Loneliness

Art Therapy Challenges

Scanning & Photogrammetry

Art therapy, a form of psychotherapy that primarily employs various forms of art as a means of communication, is well-regarded for its efficacy in facilitating the articulation and therapeutic processing of multifaceted emotional states. Nonetheless, it encounters specific challenges:

VR

EEG Monitor

Review

3 Reference Sensors

2 Forehead Sensors Alpha: 8Hz-12Hz

Online Sharing Platform

2 SmartSense Conductive Rubber Ear Sensors

Type Here ...

Share

α Rising Trend Relaxed

Beta: 12Hz-40Hz

β

Steady Focused

Media Selection

Emotion Measurement

Emotion Expression

Hard to select mediums aligning with emotional needs

Hard to measure emotional changes

Need guidance to express emotions

VR-EEG Platform

Comment Comments

Theta: 4-8Hz

θ Rising Trend

Type Here ...

Communicative

Test Environment

This research integrates EEG with VR art therapy, aiming to fill existing gaps in the traditional art creation process. EEG's non-invasive brain activity monitoring offers real-time emotional insights, offering a novel dimension to art therapy.

EEG Headset

VR Headset VR View Recording

Testee 1 Planning

Creating

Sharing

Personalized choices in VR Based on Individual Preferences

EEG’s monitoring to understand emotional changes

VR social platform for sharing and emotion expressions

Testees exhibit higher negative affect than positive on Positive and Negative Affect Schedule (PANAS) [1] [1] Roemer, A., & Medvedev, O. N. (2023). Positive and Negative Affect Schedule (PANAS). In Handbook of Assessment in Mindfulness Research (pp. 1-11). Cham: Springer International Publishing.

11

CONTEXT AND MODEL COLLECTION Study Process

Nature + Urban

Context Categorization

Urban

Scanning and Photogrammetry

Nature

Scanned from Panmen Scenic Area

Aerial Photography DJI Mini 3 Pro

Context Collection

Metashape 24.86 Hectares a4: Urban

a3: Nature + Urban Typical Episode Extraction

8 Models Pavilion

Pan City Gate

a4: Urban

a3: Nature + Urban

Ruiguang Tower

Sirui Hall

Landscape

Sirui Hall

Bridge

Model Collection

5000 Photos

Ruiguang Tower

Furniture

Ancient Building Identification

a2: Nature

Pan City Gate

Pavilion

Bridge

a2: Nature

Furniture

Landscape

Sculpture Point Cloud

b1: Scanning and Model Generation Model Wireframe

b2: Model Stylization

Model Confidence

Combination

Realistic Combination

a1b1: Empty Context + Realistic Model

a2b1: Nature Context + Realistic Model

a3b1: Nature & Urban Context + Realistic Model

a4b2: Urban Context + Stylized Model

Model Texture Ancient Painting Combination

a1b2: Empty Context + Stylized Model

a2b2: Nature Context + Stylized Model

a3b2: Nature & Urban Context + Stylized Model

a4b1: Urban Context + Realistic Model

12

DESIGN STUDY Planning

Creating

Sharing

Research Question 1: What kind of brainwave dominate during the initial planning process?

Research Question 2: What style-context pairs best enhance relaxation, focus or communicative state[1]?

Research Question 3: How do the emotions change During different states of the creating process?

Research Question 4: Does the process of reviewing, selecting favorite scenes, and sharing contribute to therapeutic outcomes?

Collect EEG data on wandering and planning in blank scene.

Calculate the normalized values per scene, identify top 3 effective scenes, and select a3b2 as the sample.

Contrast brainwave data with screen recordings to assess brainwaves during creating operations.

Collect EEG data during reviewing and favorite scene-choosing process.

Summarize and categorize changes in brainwaves corresponding to different operations.

Large objects manipulation

Compare the average value of α, β and θ of 30s resting state, integrating questionnaires.

Emotional States Scale Relaxed

-3 to 3

Difference in mean scores preand post- test

Communicative -3 to 3

Objects selection Compare the average value of α, β and θ of resting state, and integrate questionnaire data.

Top 3 scenes: α wave

Top 3 scenes: β wave

Focused

-3 to 3

Top 3 scenes: θ wave

Figures and landscape elements placement

Research Question 5: Do testees experience positive emotion changes after the entire design process?

Planning 1:

Planning 2:

Planning in Mind Planning for Style and Context Selection

Creating:

Sharing:

Wander, place, move, rotate, delete, select

Reviewing, favorite scene choosing, sharing

α β θ

Wandering and Planning in the blank scene for 1.5 min.

[1] Fischer, N.L., Peres, R., Fiorani, M., 2018. Frontal Alpha Asymmetry and Theta Oscillations Associated With Information Sharing Intention. Front. Behav. Neurosci. 12.

13

PLANNING Planning 1

Planning 2

Grow rates of average values :

Choose mood-enhancing combinations:

Judging from the results, most people will increase the average beta wave after the planning1 stage, and it can be seen that they will be more focused.

+106.43%

0.6 0.4 0.2

1.2

<35%

a1b0

47%

+219.99%

0.6 0.4 0.2

1.0

1.0 0.8

0.0 1.4 1.2

0.0 1.4 1.2 1.0

+223.75%

0.8 0.6 0.4

44%

0.0

Rest 30s

Planning 60s

Rest 30s

+118.66%

0.6 0.4

0.0 1.4

a1b0’

46%

1.2

0.8

0.8

43%

0.6 0.4

ment; a3b2 with 68% normalization is most effective. a2b1

60%

52% Rest 30s

Planning 60s

Rest 30s

a1b2

58%

a1b2

58%

57%

0.6 0.4

49% Rest 30s

Planning 60s

Rest 30s

1.2

50%

0.2 0.0 1.4

a2b0

42%

1.0

Rest 30s

Planning 60s

a3b2

Rest 30s

0.0

52%

0.6

1.4

0.4

1.2 1.0

+141.12%

0.8 0.6 0.4

1.4

0.2 0.0

1.2

1.4

1.0

1.2

Rest 30s

Planning 60s

Rest 30s

+6.21%

0.6

59%

59%

0.6

0.0

1.2

1.4

1.0

+8.19%

0.8 0.6

1.2

Rest 30s

Planning 60s

Rest 30s

0.2 0.0

0.8

1.4

0.6

1.2 1.0

-58.69%

0.6

a3b0

62%

0.4 0.2

1.4

0.0

1.2

1.4

38%

68%

43%

49%

a2b1 48%

0.2

1.2

Rest 30s

Planning 60s

Rest 30s

a3b1

65%

58%

0.8

37%

0.6

a1b1

0.4 51% Rest 30s

Planning 60s

Rest 30s

1.4

a3b0’

52%

51%

0.2 0.0

1.2

Rest 30s

Planning 60s

61%

Rest 30s

a3b2

52%

1.0

1.0

1.2

a2b2

1.0

0.2 0.0

a3b2

Rest 30s

0.6

0.0

0.4

0.8

Planning 60s

0.8

1.4

1.0

0.4

1.2

Rest 30s

0.4 56%

0.2

1.4

48%

0.2

1.0

0.2 0.0

65%

0.6

0.0

0.4

0.4

52%

60%

0.8

1.4

a2b0’

0.8

1.0 0.8

a2b1

0.4 41%

0.2 0.0

1.2 1.0

0.8

0.2

Testee 9

Communicative

61%

0.2

1.0

1.4

1.2 1.0

Testee 8

Compare and select top 3 scenes for mood enhance-

Focused

1.4

0.8

Testee 7

Relaxed

47%

0.6

1.0

0.0

a1b1

1.6 1.2 0.8 0.4 0.0

0.4

48%

0.2

0.2

1.2

0.8

0.2

0.8

Testee 6

35~40%

0.4

1.4

1.0

Testee 5

40~45%

0.6

0.0

Testee 4

and a3b1 less positive than blanks, others more effective.

45~50%

1.2

1.0

Testee 3

50~55%

Relaxed

1.2

0.8

Testee 2

55~60%

Focused

Testee 1

Analyze EEG and questionnaire data from 16 scenes; a2b2

60~65%

1.4

Selecting top 3 effective scenes of each emotion:

Questionnaires:

>65%

1.4

1.4

Normalized values’ extent:

1.0

0.6 0.4

48%

0.6 0.4

0.2 0.0

Testee 10

0.8

0.0

1.2 1.0

+87.26%

0.8 0.6

1.4 1.2

55% Rest 30s

Planning 60s

Rest 30s

0.0

Testee 11

a4b0

35%

1.0

-36.93%

0.4

0.0 1.4

1.2

1.2 1.0

-87.47%

0.8 0.6 0.4

Planning 60s

Rest 30s

Rest

55 s

2 min

50 s

45 s

40 s

35 s

30 s

25 s

20 s

15 s

5s

Planning

10 s

55 s

1 min

50 s

45 s

40 s

35 s

30 s

25 s

20 s

5s

10 s

15 s

Rest

a4b1

51%

59%

0.6

a4b0’

0.0

47%

1.2 1.0

0.8

0.8

61%

64%

0.2

1.0

Rest 30s

Planning 60s

Rest 30s

a4b2

51%

41%

0.6 0.4

52%

0.2 0.0

64%

Rest 30s

0.8

1.4

0.4

0.0

0 min

Rest 30s

0.6

0.2

Beta average data

51%

0.2

1.4

1.2

Planning 60s

0.4

0.4

0.0

0.2

Beta original trend

52%

0.6

0.6

56% Rest 30s

1.0

0.8

1.2

0.8

Testee 12

0.0

1.0

1.4

a4b1

0.2

1.4

0.4 0.2

68%

0.6 0.4

0.2

1.4

0.8

Rest 30s

Planning 60s

Rest 30s

a3b2

56%

a1b1

52%

48%

0.2 0.0

Communicative

+6.92%

0.8

Rest 30s

Planning 60s

Rest 30s

14

CREATING: SPACE CONSTRUCTION Chose a3b2 as the sample scene for creating: All α, β and θ waves show a gradual upward trend in fluctuations, with β waves increasing the most, meaning that the users become more and more focused.

15

g

VR-EEG ART THERAPY PLATFORM Select from Context Collection Planning Anxiety

Choose Creating Mode Creating

Distraction

Planning

Comment & Browse Others’ Posts

Sharing

Loneliness

Creating

Sharing

Planning

Select from Model Collection Planning

Creating

Sharing

Place, Edit, Delete Models Creating

Architecture

Planning

Tree

Creating

Bridge

Furniture

Building Part: Temple

Sharing

Planning

Plan in Blank Scene Planning

Creating

Sharing

Share Your Creation Creating

Cancel

Planning

Creating

Sharing

Planning

Creating

Share

Sharing

Planning

Creating

Sharing 16

RECITYGEN Interactive and Generative Participatory Urban Design Tool Independent Research | Beijing, China | Spring, 2023 Tutor: Runjia Tian My Role: 100% App Development, 80% Data Analysis, Group User Test Team: Di Mo, Chengxiu Yin RECITYGEN is a tool aiming at enhancing urban design through participatory methods.

Its primary goal is to enable collaboration between residents and urban planners, ensuring that community needs and preferences are incorporated into urban development. The tool harnesses latent diffusion models and the Segment Anything Model (SAM) to create an accessible platform for public input in urban design. Users interact with RECITYGEN by uploading street view images to its online interface and using SAM to mark areas for improvement. A key feature is the tool’s AI-driven visualization: users provide text descriptions of desired changes, and RECITYGEN uses a latent diffusion model to quickly generate a visual preview of these changes.

The tool, first tested in a Beijing urban renewal project, received positive feedback.

This suggests it is effective in aligning urban design with the expectations of the public.

Future plans include broadening its application, improving user interaction, and encouraging more community participation in the urban design process.

17

RESEARCH BACKGROUND Residents Facing Communication Barriers I've got tons of ideas to improve our neighborhood, but there's just no easy way to share what we're thinking.

Traditional Process Traditional Process

Proposed Process Proposed Process

Public

Public

Agree. I see so many areas needing improvement, like the old community center. But there's a huge gap between our wishes, and what the urban planners understand.

Initial Concerns

Conceptual Process

User Interface

Technical Process

Locate Sites

Pin Sites

Baidu mapping API

Select Sites

Take Site Photos

Upload Site Photos

Identify Concerns

Add Points to Set Mask

Segement Anything Model

Generate Images

Input Text Prompts

Latent Diffusion Model

Evaluate/ Regenerate

Evaluate/ Rate Results

Upload Data to Cloud

Interview with Designers

Comment and Take Survey

Data Analysis

Initial Concerns

Data Collection

Designers Struggling with Information Gathering

Concerns Indentification Quick, cost-effective input is hard with our methods. We're always constrained by resources and time.

Concerns Indentification Workshops Public Forums Stakeholder Engagement

RECITYGEN APP

Online Surveys Interviews This area has potential, but figuring out what the community really wants? It's like we're guessing half the time without their input.

Feedback and Refinement Image Generation

Collaborative Communication Gap This draft plan is not quite what we want.

If only there was a way to show you my ideas directly, with images or something tangible, to make my points clearer.

Evaluation/ Regeneration

I understand the challenge. We need a more engaging way to collaborate and refine our approach with real-time feedback.

Initial Design Concepts

Designer

Concerns Indentification

Designer 18

CONCEPTUAL PROCESS Step1: Locate Sites

Step2: Select Photos

Step3: Identify Concerns

Step4: Generate Images from Texts

Step5: Evaluate and Regenerate

Step6: Share with Designers

Accessible green space

+

Active edge with planters

...

TECHNICAL PROCESS Output Image

Prompt Encoder

Points

Image Encoder

...

Lightweight Mask Decoder

Image Embeddings

Input Image

Mask

512 X 512

1 - Mask

Valid Mask

Step3: Segment Anything Model

VAE Decoder Outside Mask

Inside Mask

Step4: Latent Space

VAE Encoder

... z0

z1

z2

zt Forward Diffusion

Lantent Vector

3X 512 X 512

zt-1 Denoising UNET

Noisy Lantent Vector

z0 Lantent Vector

...

Prompt Encoder

Prompt Segment Everything

t-1

Mask Catogaries Architecture

Structure

Results A towering, sleek skyscraper with a glass facade ...

8

A contemporary steel footbridge with ...

Pavement

A winding garden path made of patterned stone ...

Green

A vibrant community garden street pocket park ...

5 4 2

19

INFORMATION ARCHITECTURE

USER INTERFACE

20

RESULT ANALYSIS Pilot Test Mapping

Score

0

Mask

Prompt

Generated Results Summary

Top Rated Results

A residential complex with a glass facades ...

6

6

3

4

A sleek, straight concrete pavement , minimalist garden ...

4

4

6

8

A tranquil community park elements of a Japanese garden ...

7

2

5

8

A circular cobblestone path in a garden, smooth pebbles ...

6

8

6

7

A tall, artistic residential tower with a unique geometric facade ...

4

5

6

4

A sleek, futuristic pedestrian bridge with glass walkways ...

1

3

5

3

A circular cobblestone path in a serene Zen garden...

4

9

8

6

A street pocket park with colorful flowers and tall grasses ...

4

5

5

8

An office building with active store fronts, modern style buildings ...

4

6

4

2

A traditional Chinese style store front accessible from the path ...

8

6

6

7

A cozy urban oasis pocket park, peddle stone pavement ...

6

8

6

4

A modern park in an urban setting, minimalist landscape ...

2

4

5

4

5 10

0 5 10

0 5 10

0 5 10

0 5 10

0 5 10

0 5 10

0 5 10

0 5 10

0 5 10

0 5 10

0 5 10

21

SECTIONAL SAMPLING Controlled Generation of High-Resolution Architecture Design Independent Research | Boston, MA | Summer, 2023 Tutor: Runjia Tian My Role: 30% Data Collection, 50% Model Training, 50% Writing Team: Zheng Fang, Xuechen Li This paper addresses the challenge of algorithmically generating 3D architectural designs. Conventional rule-based methods and GAN-based 3D generative models offer limited extrapolation and control and are limited to generating low-resolutional results. Our method utilizes fine-tuned latent diffusion models, specifically Stable Diffusion, applied to an architectural dataset, to produce detailed architectural sections.

These are then synthesized into 3D models using latent interpolation.

The pipeline starts with creating sectional poché datasets for various building typologies—houses, museums, and offices. The fine-tuned Stable Diffusion models, enhanced through LoRA, generate new sections, conditioned on section profiles via ControlNet. This ensures fluid transitions between sections. The generated sections are reconstructed into 3D models using the Marching Cubes algorithm within the Grasshopper. Our research contributes significantly to AI-generated architectural content, providing designers with advanced tools for creating high-quality, controlled models during the initial stages of a project.

22

CONCEPTUAL FRAMEWORK Data Collection

Step1: Precedent Dataset Rhino Model x 9

2D Sections Generation

Step2: LoRA Model Training Serial Sections x 60

Step3: ControlNet Conditioned Key Sections x 10

3D Construction

Step4: Diffusion 2D Inference Interpolation 30 x 10

Step5: Sampled Point Cloud Point Cloud x 3

Step6: Mesh Reconstruction Generated Model x 3

Data Collection House

Museum

Office

Motherhouse

Poli House

House In Leiria

Yale Center for British Art

Ningbo Museum

Museum of Image and Sound

UC Innovation Center

Netherlands Institute

Larkin Building

Independent Architecture

Pezo von Ellrichshausen

Aires Mateus

Louis I. Kahn

Wang Shu

Diller Scofidio + Renfro

Alejandro Aravena

Neutelings Riedijk

Frank Lloyd Wright

23

TECHNICAL PROCESS

24

RESEARCH BACKGROUND Profile: Mother House

Monochrome, Greyscale, Archihtectural Section, House in Leiria, Single Family Home No Humans

Prompt: Single Family

+

LoRA: House in Leiria

Profile: Yale Center for British Art

Monochrome, Greyscale, Archihtectural Section, Museum of Image and Sound, No Humans

Prompt: Museum

+

LoRA: Museum of Image and Sound

Profile: UC Innovation Center

Monochrome, Greyscale, Archihtectural Section, Larkin Building, No Humans, White Background

Prompt: Office

+

LoRA: Larkin Building

25

3D CONSTRUCTION RESULTS House

Museum

Office

The algorithm captures single family house design, reflecting wall thickness and slab artic-

For museums, the method mirrors unique spatial distribution, showcasing its ability to

In office design, the algorithm mirrors typical office building scale, proportion, and struc-

ulation, essential in residential architecture.

reproduce the complex composition of public cultural spaces.

tural integrity, highlighting its potential in designing functional office spaces.

X1

X2

X3

X1

X2

X3

X1

X2

X3

Y1

Y2

Y3

Y1

Y2

Y3

Y1

Y2

Y3

Y1

Z1

Z3

Z2

X2 X2

X1

Y1

Z1

Z3

Z2

X1

Z1

Y1

X1 Y2

Y2

X2

Y3

Z2

X3

Y3

X2

Z3 Y1 Y2 Y3

X3 Z3 Z3 Z2 Z1

Z2 Z3 Z2

Z1

Z1

26

SPATIAL SYMMETRY Algorithmic Exploration of Calabi-Yau Forms Arch 743 Seminar | University of Pennsylvania | Fall 2020 Tutor: Ezio Blasetti My Role: Group Research, 100% Algorithm, 100% Generative Design Team: Bohao Sun, Xianlong Deng This project merges the disciplines of architecture, geometry, and computational science to utilize generative systems in transforming Calabi-Yau manifolds, which are pivotal in algebraic geometry and superstring theory, into innovative three-dimensional architectural models. The process begins with conformal mapping to maintain the local angles of Calabi-Yau manifolds, thereby streamlining the complexity of their geometrical attributes. Subsequently, a 4D conformal Calabi-Yau manifold is mapped and subjected to stereographic projection to achieve a 3D visual representation. The final step involves the 3D reconstruction of point clouds into a mesh format.

Emphasizing algorithmic design principles, the project adopts an iterative, experimental approach to leverage computational capabilities for creating diverse spatial structures. This method underscores the importance of spatial dimensions in architectural design and highlights the role of advanced technology in visualizing and interpreting intricate spatial forms. The project aims not just to explore theoretical concepts but also to offer practical applications in architectural design, pushing the boundaries of how we understand and utilize space in the built environment.

27

CONCEPTUAL FRAMEWORK

N=2

Conformal Mapping

Top

Right

Axon

4D Projection

3D Representation

f(z)=Z

f(z)=1/z N=3

f(z)=z^2/2

N=4

f(z)=√2z

f(z)=sin z

N=5 f(z)=cos z

f(z)=e^z

28

CONFORMAL MAPPING

HYPERDIMENSIONAL TRANSFORMATION

By changing the functions of x and y values of points, a catalog of possible transformed forms of 5-dimension calabi-yau is created.

5-Dimension Manifold

f(z)=1/z

Conformal Manifold

f(z)=z^2/2

f(z)=√2z

#Function to create a 3D parametric plot of a conformal shape def ParametricPlot3D(step=0.1): maxB = math.pi/2 + step for k1 in range(n): for k2 in range(n): for a in qrange(-1,1,step): for b in qrange(0,maxB,step): z1= Complex_z1(a,b,n,k1) z2= Complex_z2(a,b,n,k2) x_new = z1.real y_new = z2.real #Calculate the new z coordinate by combining the imaginary parts of z1 and z2 with a rotation angle Alpha. z_new = math.sin(Alpha)* z1.imag + math.sin(Alpha) * z2.imag points.append(rhp3(x_new,y_new,z_new)) return points

f(z)=sin z

f(z)=cos z

Map a 4D Calabi-Yau manifold, project into 3D, rotate across xy, xz, xw, yz, yw, zw planes, and catalog resulting geometries.

Conformal Manifold

0o

f(z)=e^z

f(z)=1/z

XY

f(z)=z^2/2

XZ

f(z)=√2z

XW

f(z)=sin z

YZ

f(z)=cos z

YW

f(z)=e^z

ZW

4D Manifold Projection

30o

60o

def rotate(self, XY, XZ, XW, YZ, YW, ZW): vec = [self.x,self.y,self.z,self.w] angles = [XY, XZ, XW, YZ, YW, ZW] for i in range( (angles)): angle = m.radians(angles[i]) c = m.cos(angle) s = m.sin(angle) rotationMatrix = [ [[c,s,0,0] ,[-s,c,0,0] ,[0,0,1,0] ,[0,0,0,1] ], [[c,0,-s,0] ,[0,1,0,0] ,[s,0,c,0] ,[0,0,0,1] ], [[c,0,0,s] ,[0,1,0,0] ,[0,0,1,0] ,[-s,0,0,c] ], [[1,0,0,0] ,[0,c,s,0] ,[0,-s,c,0],[0,0,0,1] ], [[1,0,0,0] ,[0,c,0,-s] ,[0,0,1,0] ,[0,s,0,c] ], [[1,0,0,0] ,[0,1,0,0] ,[0,0,c,-s],[0,0,s,c] ] ] matrix = rotationMatrix[i] vec = transformVec(vec, matrix) return HyperVec(vec[0],vec[1],vec[2],vec[3])

90o

120o

150o

29

3D REPRESENTATION Selected Geometries Matrix f(z)=sin z

f(z)=√2z

f(z)=z^2/2

8-dimension calabi-yau manifold by conformal transformation of sin(z) and e^z:

N=3 Top

Right

Axon

N=4 Top

Right

Axon

N=5 Top

Right

Axon

30

INCLUSIVE AND CONNECTED LANDSCAPES For Self-Constructed Settlements in Quito, Ecuador Larp 702 studio | University of Pennsylvania | Spring 2021 Tutor: David Gouverneur My Role: Group Research, 80% Urban Design, 100% Drawing Team: Shiqi Ming, Siying Xu Over a billion people around the globe live in unplanned of self-constructed settlements frequently at peri-urban locations, excluded from the benefits of city life, and occupying high-risk sites. Latin America is still considered the continent with the highest social inequalities, manifested in the disconnection between the formal and the informal areas. These areas are characterized by environmental/health problems, the lack of communal services, infrastructure and public spaces, poor accessibility, stigmatized by the formal city, and presenting low income and high levels of violence.

This project posits that landscape-driven approaches are powerful tools to address the aforementioned concerns, fostering social inclusion, and breaking down physical and cultural barriers, connecting the settlements to their natural systems, linking formal areas to the informal ones, improving physical and performative relations among the informal settlements, and establishing productive local, urban, and metropolitan networks. Supported by Quito municipality, a composite of informal enclaves, including Comité del Pueblo, La Bota, and Puertas del Sol, located on rather flat land in the north-east of Quito, Ecuador was selected as the site.

31

BACKGROUND Environmental Challenges

Social Conditions

In Quito, Ecuador, neighborhoods face environmental degradation due to deep ravines and biodiversity loss. Ravines, rich in habitats, are

Highly consolidated informal settlements are isolated from the formal city and each other, lacking community facilities and public spaces.

neglected and inaccessible. Some housing is built on environmentally unstable, high-risk terrain.

Limited connectivity and congested commercial corridors exacerbate these challenges, contributing to a high poverty rates.

Steep Slope

Surface Runoff

Degraded Habitat

Unstable Structures

Site Landslide Risk Map

Isolation from the City

Disconnection

Lack of Amenities

Economic Challenges

Site Connection Map

PUERTAS DEL SOL

PUERTAS DEL SOL

Land Risk Proportion

1 in 24 person under poverty level

LA BO TA

LA BO TA

CO MI TE

CO MI TE

DE LP UE BL O

EL CARMEN

DE LP UE BL O

EL CARMEN

Site Boundary

Disconnection

Risky Structures

Road Connection

Risky Area

Educational

Landslide Events

Commercial

Ravines

Ravines

32

DESIGN STRATEGIES 1. Reduce Risks & Address Environmental Concerns

Site Plan

1. Tech School

Relocation: Densified Community

2. Agriculture School 3. Church 4. Farmer's Market

Natural Preservation

5. Park Library 6. Ecology Exhibition & Education

10

7. Recycling Center 8. School

Street Green

9. Water Management Facility

9

10. Community Center 11. New Campus Urban Agriculture

2. Enhance Accessibility & Mobility

Proposed Metro Bridge

Central Corridor

4 2

5

3

7 6

1 Gandola Station 8

3. Celebrate the Community

Community Open Space 11 District Park

Commercial Street Campus

0

75

150

300m

33

RISK MANAGEMENT STRATEGY A Green Armature: As a Tool to Reduce Risks and Address Environmental Concerns

Agriculture Runoff Habitat Restoration

ation

Reloc

re

icultu

n tio

n1

n Agr

Urba

Sec

3

tio

Natu

c Se

s

ral Re

Section 1

Trail

ion torat

Community Raingarden Agroforestry

on

i ect

S

Section 2

ral Natu

n

ratio

Resto

k

n Par

ventio

Inter

re

icultu

n Agr

Urba

2

Terrace Agriculture

Bio-retaining Wall

Trail

Section 3

ion

ificat

Dens

k

& Par

Road

34

DIGITAL PRESERVATION: 3D Scanning of Xi’an’s Dongyue Temple Arch Studio | Southeast University | Fall 2018 This project entailed a detailed 3D scan of Xi'an's Dongyue Temple, using Faro technology. Over three days, our ten-student team captured

The Dongyue Temple, located near Xi'an's Ming City Wall, was originally established in 1116 during the Song Dynasty and underwent several

the temple's architectural intricacies, aiming to digitally preserve and study this historic site for future conservation.

renovations and expansions during the Ming and Qing Dynasties.

100m

Elevation: Specifies the color of points based on heights.

50m

Dou gong

Cha shou

Liang jia

Colored murals

Middle Hall

0m

Rear Hall

100%

Intensity:

Main Hall

Specifies the color of points based on the laser return intensity.

50%

Shanmen (Gate)

0%

Elevation: Specifies the color of points based on surface orientation.

35 35

WATER MINER: Speculative Urban Design on Johannesburg 2050 LARP 701 studio | University of Pennsylvania | Fall 2020 This studio project employs multimedia to imagine Johannesburg 2050, where extreme weather drive residents to an underground city within former mines. This speculative design emphasizes ecological restoration for sustainable living in challenging environments.

Storyboard

Water level

Camera height

Water level

Camera height

Water level

Camera height

Water level

Camera height

Water level

Camera height

36 36

DYNAMIC EDGES: Conservation and Development of New York State LARP 601 studio | University of Pennsylvania | Fall 2020 This research project reenvisions the borders of New York State as dynamic, functional areas. Utilizing GIS data analysis, it proposes strategies

Key approaches include expanding conservation easements to enhance activities and broadening

that balance conservation efforts with development needs.

protected areas for ecological restoration, aiming to transform borders into sustainable spaces.

Conflicts

Existing Edges

Dynamic Edges

The boundaries of conservation zones

Identify approaches for managing

and areas of high population density

boundaries based on their ecological

suggest a possible conflict.

and economic significance.

Expand activities Expand protected area

Strategies

Albany Pine Bush Preserve

Greenport Conservation Area

Phoenicia Wild Forest

Highland Lakes State Park

Blue Mountain Reservation Park GIS Data Source: McGarigal K; Compton BW; Plunkett EB; DeLuca WV; Grand J; Ene E; Jackson SD. 2018. A landscape index of ecological integrity to inform landscape conservation. Landscape Ecology 33:1029-1048.

37

UNIVERSITY OF NEW MEXICO INTERGRATED CAMPUS PLAN Online Survey / On Site Open Houses for Design Reflection As a part of Sasaki Urban Design Team, I developed the UNM’s 2050 campus plan focusing on community engagement. Utilizing the Comap

This extensive engagement helped identify welcoming spaces, areas of safety concern, and accessibility issues on campus. The analysis of this

platform, we gathered 435 online responses and over 1,300 comments, plus feedback from 500+ participants in Albuquerque open houses.

rich community input was pivotal in shaping our design framework.

CoMap Platform Result Analysis for Online Survey

38

Perception - Design - Technology

39