Driving innovation - First car to be driven by AI-powered smartphone ‘RoadReader’ – a project designed to push the boundaries of Huawei’s object recognition technology, putting the learning capabilities, speed and power of its AI-powered devices to the test by piloting a driverless car. V2 -MOVING VEHICLE IMAGE RECOGNITION
YOUR CAR IS READY, ARE YOU? PRESS DRIVE AND ENJOY THE RIDE.
PLEASE PLACE GEAR INTO DRIVE
V2 -MOVING VEHICLE IMAGE RECOGNITION
OBJECT DETECTION SYSTEM ACTIVE INITIALIZING AUTONOMOUS CONTROLS
INITIALISING OBJECT RECOGNITION
How it works: Step by step
D
GET STARTED
A
INITIALISING OBJECT RECOGNITION
User clicks GET STARTED on the home screen
SCANNING FOR OBSTACLES
A
B
C
The app starts
User is instructed to place the car into gear
DRIVE
The carTHROTTLE appears to run SETthrough some initialisation TO 17% processes
The user clicks DRIVE and the car drives at 5mph down the 100m road
Object recognition:
C User is instructed to place the car into gear OBJECT DETECTION SYSTEMDETECTION ACTIVE OBJECT
INTO DRIVE
A A
User clicks GET STARTED on the home User clicks GETscreen STARTED on the home screen
SCANNING GETOBSTACLES STARTED FOR SCANNING
Camera detects object and passes this data into an HDMI converter, then turns this data into a UVC protocol which is directly read by the Mate 10 Pro, through its USB-C
FOR OBSTACLES
A
User clicks GET STARTED on the home screen
EMERGENCY STOP
B F B
EMERGENCY STOP
DOG HAStoBEEN User A is instructed place the Objects are displayed DETECTED car into gear User iswhen instructed to place the detected car into gear
EMERGENCY STOP
EMERGENCY STOP
D
EMERGENCY STOP
P
LEARNING COMPLETE YOUR CAR IS READY, ARE YOU? EMERGENCY PRESS DRIVE STOP AND ENJOY THE RIDE.
D H D
A CAT HAS The car appears to BEEN run through 2nd DETECTED object detected INITIALIZING some initialisation The car appears to processes run through someAUTONOMOUS initialisation processes CONTROLS
EI E
A BIRD HAS BEEN The user clicks DRIVE and the 3rdDETECTED object detected car at 5mph down the Thedrives user clicks DRIVE and the 100m road down the car drives at 5mph
DRIVE 100m road
LEARNING ACOMPLETE CAT HAS BEEN LEARNING DETECTED STAY COMPLETE SEATEDI AND PLACE
A DOG HAS BEEN DETECTED
G
H
GEAR INTO PARK 3rd object detected STAY SEATED AND PLACE GEAR INTO PARK
Objects are displayed when detected
C
D
E
User is instructed to place the car into gear
The car appears to run through some initialisation processes
The user clicks DRIVE and the car drives at 5mph down the 100m road WHICH MANOEUVRE SHOULD BE ACTIVATED WHEN DRIVING
COLLECTING RECOGNISED A CAT HASOBJECTS BEEN DETECTED A CAT HAS BEEN DETECTED
2nd object detected
WHICH MANOEUVRE A BIRD BEEN SHOULD BEHAS ACTIVATED DETECTED WHEN DRIVING A BIRD HAS BEEN DETECTED A DOG HAS BEEN DETECTED SWERVE LEFT
SCANNING
EMERGENCY STOP CONTINUE EMERGENCY STOP
F F appears A radar style animation
G Gdisplayed Objects K are
when are detected Objects displayed Back at the start line, the user gets backwhen in thedetected car and clicks continue on the app A DOG HAS BEEN DETECTED
EMERGENCY STOP EMERGENCY STOP
H LH
COLLECTING RECOGNISED OBJECTS 2nd object detected 2nd object detected The app collects the detected objects A CAT HAS BEEN DETECTED
BRAKE
EMERGENCY STOP EMERGENCY STOP
I MI
WHICH MANOEUVRE SHOULD BE ACTIVATED 3rd object detected WHEN DRIVING 3rd object detected Tthe user is asked to select a manoeuvre for the first object A BIRD HAS BEEN DETECTED SWERVE LEFT
BRAKE
SWERVE RIGHT
SWERVE LEFT
A DOG HAS BEEN
Data is sent to AI chip which identifies object. The Mate 10 Pro’s NPU (Neural Processing Unit) uses a complex object-recognition algorithm (Tensor Flow), the NPU interprets the image data and matches the object from its model library of 1000s of images F
A radar style animation appears to scan ahead, and the user can stop at any time
YOUR SELECTED EMERGENCY STOP MANOEUVRES
G
Objects are displayed when detected COLLECTING RECOGNISED OBJECTS COLLECTING RECOGNISED OBJECTS
K K
continue on the app
INTO DRIVE DETECTED
MANOEUVRES
PLL
The app collects the A summary of the manoeuvres detected objectsthe The app collects is briefly displayed COLLECTING detected objects RECOGNISED OBJECTS
I
WHEN DRIVING THROTTLE SET TO 17%
BRAKE
SWERVE RIGHT SWERVE
M Q M
Tthe user is asked to User is instructed to place the select auser manoeuvre Tthe askedfor to car intoisgear the afirst object for WHICH MANOEUVRE select manoeuvre SHOULD the BE firstACTIVATED object WHEN DRIVING
D
PLEASE PLACE GEAR INTOPLACE DRIVE GEAR PLEASE
SCANNING FOR OBSTACLES BRAKE
SWERVE RIGHT
D D
PREPARING TO SWERVE LEFT
PLEASE PLACE GEAR INTO DRIVE
EMERGENCY STOP DOG DETECTED AHEAD
Q U Q
User is instructed to place the The user is informed when an car into gear User object is instructed to place the is detected car into gear PREPARING TO SWERVE LEFT
D
SWERVE LEFT
BRAKE BRAKE
M
SYSTEM ACTIVE Tthe user is asked to INITIALIZING select a manoeuvre for AUTONOMOUS INITIALIZING the first object CONTROLS AUTONOMOUS SCANNING CONTROLS FOR OBSTACLES THROTTLE SET TO 17% SET MANOEUVRE THROTTLE TO 17% ACTIVATED
EMERGENCY STOP
OBJECT DETECTION SYSTEM ACTIVE
R VR
The car appears to run through The manoeuvre is activated some initialisation INITIALIZING The car appears to processes run through MANOEUVRE some AUTONOMOUS initialisation processes CONTROLS ACTIVATED THROTTLE SET TO 17%
J NJ
WHICH MANOEUVRE SHOULD BE ACTIVATED When learning is complete, the WHEN DRIVING journalist is collected from thethe car When learning complete, user isisasked to and is Tthe walked back to the start line journalist from selectisa collected manoeuvre forthe car and is walked backobject to the start line the second
SWERVE LEFT
P BRAKE
SWERVE RIGHT
A BIRD HAS BEEN YOUR CAR IS READY, DETECTED A BIRD HAS BEEN ARE YOU? DETECTED PRESS DRIVE AND ENJOY THE RIDE. Tthe user is asked to select a manoeuvre for the second object
J
U
5. Smart Manoeuvring
PLEASE PLEASE STAY SEATED
V
P
Q
R
A summary of the manoeuvres DOG DETECTED is briefly displayed DOGAHEAD DETECTED AHEAD
User is instructed to place the car into gear
The car appears to run through some initialisation processes
PREPARING TO SWERVE LEFT PREPARING TO SWERVE LEFT
The Mate 10 Pro sends the car’s robotic system a series of commands (reduce throttle, apply the brakes, turn the steering wheel etc… The commands are sent from the phone via WiFi using UDP (User Datagram Protocol)
MANOEUVRE MANOEUVRE ACTIVATED SCANNING ACTIVATED FOR OBSTACLES
EMERGENCY STOP EMERGENCY STOP
EMERGENCY STOP EMERGENCY STOP
U U
The manoeuvre is activated
The user is informed when an object is detected The user informed when an DOGis DETECTED
V V
The manoeuvre is activated
object is detected AHEAD
PREPARING TO SWERVE LEFT
EMERGENCY STOP
P P
PLEASE STAY SEATED AND PLACE GEAR INTO PARK
W W
Keep calm - we’re in control :) The journalist steps out of :) Keep calm - we’re in control the car. The journalist steps out of
SWERVE
A CAT HAS BEEN RIGHT DETECTED OBJECT DETECTION SYSTEM ACTIVE
SWERVE LEFT
P
EMERGENCY STOP
U
V
W
The user is informed when an object is detected
The manoeuvre is activated
Keep calm - we’re in control :) The journalist steps out of the car.
Processing power:
SWERVE RIGHT
The Kirin 970 chip has a NPU (Neural Processing Unit) which acts like its brain, giving the car the unique ability to understand and learn about objects and hazards on the road and decide on and carry out the appropriate manoeuvre.
WHICH MANOEUVRE SHOULD BE ACTIVATED WHEN DRIVING Tthe user is asked to select a manoeuvre for the second object
O
SWERVE LEFT
BRAKE
SWERVE RIGHT
SCANNING FOR OBSTACLES
O
Tthe user is asked to select a manoeuvre for the second object
BRAKE
EMERGENCY STOP
INITIALIZING AUTONOMOUS CONTROLS
DRIVE Tthe user is asked to And finally the user can select manoeuvre for Ttheauser is asked to choose when to ‘go’ the second object for WHICH select aMANOEUVRE manoeuvre SHOULD BE ACTIVATED the second object WHEN DRIVING
N RN
SCANNING FOR OBSTACLES
SWERVE
A BIRD HAS BEEN RIGHT DETECTED
Tthe user is asked to The car appears to run through THROTTLE SET for select auser manoeuvre is asked to someTthe initialisation processes TO 17% the second object for WHICH MANOEUVRE select a manoeuvre SHOULD BE ACTIVATED the second object WHEN DRIVING
O S O
T A radar style animation appears to scan ahead, and the user can stop at any time
EMERGENCY STOP SWERVE LEFT
BRAKE
SWERVE RIGHT
ARE YOU? YOUR CAR IS READY, PRESS DRIVE AND STAY SEATED ARE YOU? ENJOY RIDE. PRESS THE DRIVE AND AND PLACE GEAR The carENJOY appearsTHE to run through RIDE. INTO PARK some initialisation processes
R
N Tthe user is asked to select a manoeuvre for the second object
DRIVE DRIVE
P
PLEASE STAY SEATED AND PLACE GEAR INTO PARK YOUR CAR IS READY, ARE YOU? PRESS DRIVE AND ENJOY THE RIDE.
S W S
And finally the user can Keep calm - we’re in control :) choose when to ‘go’can And finally the user The journalist steps out of choose thewhen car. to ‘go’
P
DRIVE
WHEN THE CAR WHEN THE CAR IS BACKW AT THE IS BACK AT THE THE STARTLINE,
Keep calm - we’re in control :) STARTLINE, The journalist steps THE out of JOURNALIST the car. JOURNALIST
CAN GET BACK SHAVE CAN GET BACK IN TO And finally the user can IN TOwhen HAVE ANOTHER RUN. choose to ‘go’ ANOTHER RUN. THE APP WILL THE APP WILL RESTART AT RESTART AT
Q Q
WHEN THE CAR IS BACK AT THE STARTLINE, THE JOURNALIST CAN GET BACK IN TO HAVE ANOTHER RUN.
the car.
MANOEUVRE ACTIVATED
BRAKE
DRIVE
YOUR CAR IS READY, ARE YOU? PRESS DRIVE AND SWERVE BRAKE RIGHT ENJOY THE RIDE.
SWERVE LEFT
BRAKE
SWERVE RIGHT
SCANNING
FORSCANNING OBSTACLES WHEN THE CAR FOR OBSTACLES SAT THE IS BACK And finally the user can STARTLINE, THE choose when to ‘go’ JOURNALIST CAN GET O BACK IN TO HAVE Tthe user is asked to select a manoeuvre for ANOTHER RUN. the second object
THE APP WILL RESTART AT
Q
T A radar style animation appears to scan ahead, and the user can stop at any time
Intelligent learning:
THE APP WILL RESTART AT
Q
WHEN THE CAR IS BACK AT THE STARTLINE, THE SCANNING JOURNALIST FOR OBSTACLES EMERGENCY STOP CAN GET BACK EMERGENCY STOP IN TO HAVE ANOTHER RUN.
T T
THE WILL A radar styleAPP animation appears toA scan ahead, and the user can radar style animation appears RESTART AT at any to scanstop ahead, andtime the user can stop at any time
Q
EMERGENCY STOP
STAY SEATED AND PLACE GEAR INTO PARK The manoeuvre is activated AND PLACE GEAR INTO PARK
WHICH MANOEUVRE SHOULD BE ACTIVATED WHEN DRIVING
SWERVE LEFT
SWERVE LEFT
FOR OBSTACLES
The user is informed when an object is detected
J
When learning is complete, the journalist is collected from the car and is walked back to the start line WHICH MANOEUVRE SHOULD ACTIVATED WHICH BE MANOEUVRE WHENBE DRIVING SHOULD ACTIVATED
SCANNING FORSCANNING OBSTACLES EMERGENCY STOP
A BIRD HAS BEEN DETECTED
A BIRD HAS BEEN DETECTED
LEARNING COMPLETE
Huawei developed the Mate 10 Pro’s built-in object recognition mode, teaching the device to distinguish between specific objects and implement different avoidance manoeuvres.
When learning is complete, the journalist is collected from the car and is walked back to the start line
WHEN DRIVING
SWERVE RIGHT
PLEASE
Q User is instructed to place the car into gear OBJECT DETECTION SYSTEMDETECTION ACTIVE OBJECT
INTO DRIVE
SWERVE LEFT
YOUR CAR IS READY,
P
A summary of the manoeuvres is briefly displayed
L
P P
BRAKE
A DOG HAS BEEN RIGHT DETECTED
The app collects the detected objects DOG DETECTED AHEAD
is briefly displayed
D
BRAKE
SWERVE RIGHT
N
Tthe user is asked to select a manoeuvre for the first object OBJECT DETECTION SYSTEM 3rd objectACTIVE detected INITIALIZING WHICH MANOEUVRE AUTONOMOUS SHOULD BE ACTIVATED WHICH MANOEUVRE CONTROLS WHENBE DRIVING SHOULD ACTIVATED
PLEASE PLACE GEAR INTO DRIVE SWERVE LEFT
K
A summary of the manoeuvres is briefly A summary of displayed the manoeuvres
M
H
2nd object detected
SWERVE LEFT
Back at the start line, the user gets back in the car and clicks continue on the app
YOUR SELECTED MANOEUVRES
DETECTED
L
The app collects the detected objects
WHICH MANOEUVRE SHOULD ACTIVATED WHICH BE MANOEUVRE WHENBE DRIVING SHOULD ACTIVATED
SWERVE LEFT
YOUR SELECTED CONTINUE MANOEUVRES YOUR SELECTED
DETECTED EMERGENCY STOP A CAT HAS BEEN
WHEN DRIVING YOUR SELECTED MANOEUVRES
CONTINUE CONTINUE
Back at the start line, the user gets back in the and clicks Back at the startcar line, the user on the app clicks getscontinue back in the car and
A CAT HAS BEEN
DETECTED PLEASE PLACE GEAR EMERGENCY STOP A DOG HAS BEEN
K
Back at the start line, the user gets back in the car and clicks continue on the app
P
STAY SEATED AND PLACE GEAR INTO PARK
CONTINUE
EMERGENCY STOP
P P
A CAT HAS BEEN DETECTED
SWERVE RIGHT
2K
J When learning is complete, the journalist is collected from the car and is walked back to the start line
EMERGENCY STOP
B
DETECTED
STAY SEATED AND PLACE GEAR INTO PARK
THROTTLE SET TO 17%
EMERGENCY STOP
The app starts
FOR OBSTACLES EMERGENCY STOP EMERGENCY STOP
toA scan and the user can radarahead, style animation appears at any to scanstop ahead, andtime the user can stop at any time
A BIRD HAS BEEN DETECTED
TO 17%
A radar style animation appears to scan ahead, and the user can stop at any time
A DOG HAS BEEN DETECTED A DOG HAS BEEN
LEARNING COMPLETE E
STAY SEATED AND PLACE The user clicks DRIVE and the GEAR INTO PARK car drives at 5mph down the 100m road
DRIVE DRIVE
OBJECT DETECTION SYSTEM ACTIVE
C G C
The app starts A radar style animation appears The app to scan ahead, andstarts the user can INITIALISING stop at any time OBJECT RECOGNITION
F
D
CONTROLS A CAT HAS BEEN THROTTLE SET DETECTED TO 17% SET THROTTLE
A DOG HAS BEEN DETECTED
SCANNING FOR OBSTACLES
PLEASE PLACE GEAR INTO DRIVE
ARE YOU? YOUR CAR IS READY, PRESS DRIVE AND ARE YOU? ENJOY RIDE. PRESS THE DRIVE AND The carENJOY appearsTHE to run through RIDE. some initialisation processes
SYSTEM ACTIVE INITIALIZING AUTONOMOUS INITIALIZING CONTROLS AUTONOMOUS
INITIALISING OBJECT RECOGNITION INITIALISING OBJECT RECOGNITION
GET STARTED GET STARTED
What makes this unique?
E
YOUR CAR IS READY,
B
The app starts
Camera scans road ahead and streams its image data wirelessly using a Teradek Bolt 3000 V2 -MOVING VEHICLE IMAGE RECOGNITION D D
The Mate 10 Pro then initiates the appropriate avoidance manoeuvre for the car to make
INITIALIZING AUTONOMOUS CONTROLS
D
D
PLEASE PLACE GEAR INTOPLACE DRIVE GEAR PLEASE
User clicks GET STARTED on the home screen
4. Selecting Avoidance Manoeuvre
2K
GET STARTED
1. Scanning
3. Object Recognition
YOUR CAR IS READY, ARE YOU? PRESS DRIVE AND ENJOY THE RIDE.
OBJECT DETECTION SYSTEM ACTIVE
V2 -MOVING VEHICLE IMAGE RECOGNITION V2 -MOVING VEHICLE IMAGE RECOGNITION
2. Detection
DRIVE
THROTTLE SET TO 17%
PLEASE PLACE GEAR INTO DRIVE
EMERGENCY STOP
T A radar style animation appears to scan ahead, and the user can stop at any time
2K
If the obstacle matched one of the 1000 entries in the current object recognition ‘model’ the phone uses, then the Mate 10 would “know” what the object was. If the object isn’t currently listed, then the phone has the ability to be trained how to identify it via deep machine learning.