Apply Computer Vision and Machine Learning to Measure Human Perceived Street Environments by Waishan Qiu

Apply Computer Vision and Machine Learning to Measure Human Perceived Street Environments

Our Company Big Data Analytics & Informatics

Novel Tech ArtiďŹ cial Intelligence

Expertise on cities Design & development

A-STRA.ai is a data and technology driven platform where decisions on urban development can be informed and made explicitly. We have provided strategic advisory services for governments in China, North America and Middle East. We understand the importance of right strategies to support the long-term economic, social and environmental development of cities. We leverage our skills in strategic consulting, big data, artiďŹ cial intelligence and machine learning to help our clients create urban strategies and transform them into actionable plans.

Our Service

Action

Better data supports better decision. We leverage our superior skills in big data, artiﬁcial intelligence (AI) and spatial analysis

Insights are actionable in economic, institutional and physical dimensions.

Data

●

Urban & Community Sustainability

●

Economic Development & Impact

●

Urban Policy

●

Real Estate and Financial Feasibility

●

Transportation

Insight We provide strategic insights to governments, companies

Our Team Our founding members got superior academic training from top universities including Harvard, MIT, Cornell, UCL as well as professional experience in consultancy, design and engineering industries including BCG, SASAKI, Senseable City Lab and Media Lab.

Co-founder

Xiaokai Huang

is a doctoral candidate in urban and real estate studies at Harvard University. He worked at Boston Consulting Group as a consultant. Before that, he worked as a project manager at a boutique consulting ďŹ rm, with a focus on the Chinese market. In this role, he collaborated with municipal governments and real estate developers in China for their development strategies. He also worked with the Development Research Centre of State Council, P.R.China. He began his career as an urban planner with Sasaki Associates in Boston.

Co-founder

Waishan Qiu

is a Ph.D. student in Regional Science at Cornell. He obtains master of city planning degree at MIT and is an experienced spatial data scientist and digital designer. He was a former researcher at the Media Lab, Senseable City.Lab, LCAU and Harvard Kennedy School working on various geospatial analysis and urban computing projects.

Team Member Spatial Analysis | Xiaojiang

is a postdoctoral fellow at MIT Senseable City Lab, and is an expert in data-driven geospatial analysis. His work has been featured in popular media outlets like the Wall Street Journal, Wired, The Guardian and Forbes.

ArtiďŹ cial Intelligence | Bill

Cai

Obtains S.M. in Computational Engineering from MIT and specializes in deep learning and computer vision. As a data scientist in One Concern, he builds deep learning pipelines for city-wide analysis. He previously interned in Thumbtack and Arcstone.

Transportation | Jintai

Jintai is a Ph.D. student in the Interdepartmental Doctoral Program in Transportation at MIT. His current research works with Transit Lab and the Urban Mobility Lab focus on 1) demand prediction in response to the introduction of ride-hailing services provided by autonomous vehicles, and 2) user preference and segmentation.

io n

3 al

ud i

eâ&#x20AC;&#x2122;s

fe m

ve h

jo b

ar ke t

nd s

Ar ab

hi na ira

an gh ai

et h ra bi

er da

ili

rA m

sy s

or ta tio

ib ili

ac ce

Au to n

Tr an sp

t, Un ite

ric

on om

to w

io n

ab i,

ta E te w valu at a er tio de fro n ve nt of in re lo a Bo l pm e st st on at en ,U ed t SA ev el op m en tf or

ho sm u en Zh t ej of ia 24 ng s ,C m hi all na ci ti

na b

ta i

ity ,A

an n

st ra te

ar b

-c

lo w

im ba pa n a ct nd A s re Ta sse iz s g

2 Sa

ar ch

1 Sm de ar ve t c lo ity Re pm a ge ne en nd ra s t t

Project Portfolio

Ongoing corporations

Partners

For a livable, sustainable, and competitive city The purpose of the project is to identify factors in the built environment with positive or adverse effects to stress levels in humans. Of special interest are factors like building configurations, e.g. clusters of skyscrapers, traffic flows, materials, vegetation, humidity, temperature etc.

Ongoing corporations Research Framework and Expected Outcome

Living in urban dwellings

Factors in the built environment building configurations, e.g. clusters of skyscrapers, traffic flows, materials, vegetation, humidity, temperature etc.

Correlation

Pollutions in terms of air, noise, and light

Serious stress-related diseases

Depressions, diabetes II, autistic disorders from exposure in children’s early years For older people Cardio-, cancers, and respiratory diseases

Outcomes A set of fact-based design codes

Pudong Project Highlights

Segment more than 100,000 street view images (SVI) from Baidu API

Apply 10 machine learning and 2 computer vision algorithms in the study

Run across 6 leading innovation cities in the globe

We propose 3 key actionable urban development metrics

0 | Overall Framework

1 | the development of the Street Space Assessment System 6 dimensional scores and 35 features segmented from Street View Image Evaluation System Establishment

1 Safety Score Algorithm + 4 dimensions

Evaluation Indicators Selection

2 | Database Build-up with 100K+ Street View Images across Pudong District 100K+ Street View Images Machine Learning

Experts Evaluation Street Score Prediction Feature Segmentation

Computer Vision

3 | Sampling for Training Models

3.1 Randomly sample 5000 images across Shanghai region as the training data

3.2 Download street view images with consistent angles and requirements Point of view is located in the middle of the road and parrelled to the road direction, control by a list of variables including [‘fov’,’pitch’,’heading’]

Sample code for Baidu Street View Image API: http://api.map.baidu.com/panorama/v2?ak=pYm2gy8gN6ODmgOelTS2BrwdpIM3GGr D&width=600&height=300&location=121.501361,31.287838&fov=150&pitch=0&headi ng=300 Sample code for Google Street View Image API is also similar

Training Model Highlights

1000

sample images across Shanghai as the training Data

aspects of human perceptions about street environments

street level features segmented from street view image

4 | Training Data Through Public Participation Visitors can directly click on the photo they prefer. To acquire training data of peopleâ&#x20AC;&#x2122;s preferences on street view images, we developed an online questionnaire platform on which the visitors can select the images they directly. These preferences are then translated to scores with Microsoft TrueSkill Algorithm.

A screenshot of our web page interface

1. Select the type of space this image represents: () Urban () Outskirt () Waterfront () Village () Industry park

Click on the image to visit the site 2. Click on the image which you think has better feeling of enclosure

5 | Sample Evaluation (Training Data Summary) Q1. Imageability

Q2. Enclosure

Q3. Human Scale

Urban

High

Waterfront

Low

Q4. Complexity

Q5. Total Score

Outskirt

High

Rural

Low

Industrial Park

6 | Model Training Step 1 Computer Vision: Semantic Segmentation (PSPNET)

6 | Model Training Step 1 Computer Vision: Instance Segmentation (MASK RCNN)

6 | Model Training Step 2 Machine Learning with ClassiďŹ cation and Regression Algorithms

Linear kernel SVM

Support Vector Machine

Predict Labels

Polynomial kernel SVM RBF kernel SVM Dual SVM

Urban/rural/outskirt/... Number of neighborhoods

Machine Learning

K-nearest neighbors

Classification

Predict Scores

Predict scores for unseen street view images based on the training data Regression

Euclidean/Manha ttan distance Weighted distance

Overall Score Decision Tree Regression Enclosure Score Linear Regression Human Scale Score Richness Score

Multivariable Regression

Polynomial Regression

Best Fit Model Base on RMSE and R square

6 | Model Training Results

6 | Result Geospatial Visualization

by features: pedestrian count

by features: green view index

by features: sky ratio

6 | Result Geospatial Visualization

6 | Result Geospatial Visualization Dimension Correlation Analysis

Robust Check with External Data (overlay Q4. Complexity map with Dazhongdianping app data)

7 | Global City Benchmark Overall Score Evaluation across 5 other Global Innovative Cities with Shanghai

Kendall Square Cambridge, USA

Knowledge Quarter, London,UK

Wall Street, Manhattan, USA

Downtown, San Francisco, USA

South Lake Union, Seattle, USA

Zhangjiang Park Shanghai, China

7 | Global City Benchmark Overall Score Evaluation across 5 other Global Innovative Cities

街道质量得分的核密度曲线 | Kernel Density Estimation Shanghai

San Francisco

街道质量评价地图 | Street Overall Score Mapping

London

Seattle

Cambridge

Manhattan

7 | Global City Benchmark Street Assessment Results of the 6 global cities based on Street View Image Segmentation and Machine Learning

Kernel Density Estimation (KDE)

Q3.Human Scale Score

Q5.Overall Score Kernel Density Estimation (KDE)

(1 is urban and 2 is rural)

Q4.Diversity Score

Q2.Enclosure Score

Kernel Density Estimation (KDE)

Q1. Average score of the IdentiďŹ cation for Urban/ Rural cityscapes

7 | Global City Scores Breakdowns Look at the score breakdowns in the 5 dimensions of perceptions of â&#x20AC;&#x153;enclosure, humanity, diversity, safety and overall scoresâ&#x20AC;?, Shanghai receives the lowest scores in all, and particularly in Enclosure and Diversity breakdowns as the gaps between Shanghai and other cities in terms of these 2 dimensions are the greatest.

7 | Global City Street Features Breakdowns Look at the feature breakdowns in the more than 20 street features ranging from road, building, sky, tree to sidewalks, Shanghai receives the relatively bad performance in sky ratio,building ratio, road ratio, street walls, fences. Those features particularly aďŹ&#x20AC;ect Enclosure and Diversity perceptions. Surprisingly, the greenery or tree canopy ratio in Shanghai is very high.

7 | Global City Street Features Breakdowns Look at the feature breakdowns on Diversity score, we can see that Shanghai receive the worst Diversity Score as a result of high ratio of street walls and fences.

7 | Global City Street TraďŹ&#x192;c Breakdowns We can even get some glimpse at the traďŹ&#x192;c amount on the road from street view images.

7 | Global City Street Slow TraďŹ&#x192;c Breakdowns We can even get some glimpse at the slow traďŹ&#x192;c amount on the road from street view images.

7 | Global City Urban Form Comparison

7 | Global City Urban Form Metrics Comparison

7 | Global City Urban Development Metrics Comparison

7 | Global City Urban Amenities Comparison

Three Key Actionable Metrics

5-6 FAR

40-50

5-7

/sqkm

/1000 PPL

Block Density

Third Place Density

8| Implications for Zoning & Urban Design Guidelines Learn from the evaluation results, urban designers now are enabled with more human-centered and scalable approaches. The photo is a example: a student generated a urban renewal scheme in a design workshop based on our methods.

9 | Other Deliverables

Click on the image to visit the site

Unlocking NYCâ&#x20AC;&#x2122;s Potential

Equitable Development

Assets Investigation and Management

Zoning and Urban Planning Guidelines

Urban Environment Assessments

Equitable Development Online platform to collect and learn a diverse group of peopleâ&#x20AC;&#x2122;s collective insights and perceptions about street and public spaces

Assets Investigation and Management Learn to predict vacant houses and properties, predict property values, potential commercial activities

Zoning and Urban Planning Guidelines Provide more human-centered and evidence-based approaches to support planning and design decisions

Urban Environment Assessments Scalable and eďŹ&#x192;cient tool to evaluate the performance of built environment most related to public spaces (streets, parks, plazas)

THANK YOU