1
3
Copyright © 2017 By Yitian Ma First edition
CREATE THE THINGS YOU WISH EXISTED.
TABLE OF CONTENTS 8 10
FOREWORD HISTORY OF AUGMENTED REALITY
14 16 18 26 28 30 32 36 38 40 48 58 64 68 72 74
THE PAST WHAT IS AUGMENTED REALITY (AR)? WHERE DID AR COME FROM? HOW DOES AR WORK?
THE PRESENT REVAMPING CONTENT WITH AR AR TREATMENT REDUCES PHANTOM PAIN AR IN EDUCATION ENTERTAINMENT APPLICATIONS IN REALITY TECHNOLOGY
THE FUTURE INTRODUCTION THE FUTURE DEVELOPMENT OF AR THE FUTURE OF HUMAN INTERACTION CHANGING LIFE FOR THE BETTER PEERING INTO THE FUTURE OF AR GAMES USING AR FOR INTERIOR DESIGN AND PRODUCT DESIGN MAKEUP BRANDS ARE TESTING AR TO DRIVE CONVERSIONS
CONCLUSION CREDITS
Future of Augmented Reality 8
FOREWORD
Computer graphics have become much more sophisticated since then, and game graphics are pushing the barriers of photorealism. Now, researchers and engineers are pulling graphics out of your television screen or computer display and integrating them into real-world environments. This new technology, called augmented reality, blurs the line between what’s real and what’s computergenerated by enhancing what we see, hear, feel and smell. On the spectrum between virtual reality, which creates immersive, computergenerated environments, and the real world, augmented reality is closer to the real world. Augmented reality adds graphics, sounds, haptic feedback and smell to the natural world as it exists. Both video games and cell phones are driving the development of augmented reality. Everyone from
9
tourists, to soldiers, to someone looking for the closest subway stop can now benefit from the ability to place computergenerated graphics in their field of vision. The basic idea of augmented reality is to superimpose graphics, audio and other sensory enhancements over a real-world environment in real time. Sounds pretty simple. Besides, haven’t television networks been doing that with graphics for decades? However, augmented reality is more advanced than any technology you’ve seen in television broadcasts, although some new TV effects come close, such as RACEf/x and the super-imposed first down line on televised U.S. football games, both created by Sportvision. But these systems display graphics for only one point of view. Nextgeneration augmented-reality systems will display graphics for each viewer’s perspective.
Future of Augmented Reality 10
HISTORY OF AUGMENTED REALITY Myron Krueger builds an “artificial reality” laboratory called the “Videoplace”. The “ Videoplace” combined projectors with video cameras that emitted on-screen silhouettes, surrounding users in an interactive environment.
A prototype dubbed the Sensorama is built along with five short films for it that engage multiple senses(sight, smell, sound, and touch). The mechanical device still functions today.
1970
1960
1968
Ivan Sutherland is credited with creating the first head-mounted display system. The system was suspended from the ceiling. The computergenerated graphics the users saw were simple wireframe drawings.
199
1980
1974
1962
1982 Thomas Furness at the US Air Force’s Armstrong Medical Research Laboratories developed in 1982 the Visually Coupled Airborne Systems Simulator – an advanced flight simulator. The fighter pilot wore a HMD that augmented the outthe-window view by the graphics describing targeting or optimal flight path information.
11
The term “Augmented Reality” was coined by Boeing researcher Tom Caudell.
90
AR is first used for entertainment purposes when Julie Martin creates what is believed to be the first Augmented Reality Theater production, “ Dancing in Cyberspace.”
Naval researchers begin work on Battlefield Augmented Reality System (BARS), the original model of early wearable systems for soldiers.
ARToolkit brings AR to Web browers.
Wearable AR makes headlines, mostly thanks to Google Glass. Other companies, like Epson, have also developed their own smart glasses, Startup company Innovega is taking smart one step further, introducing AR contact lenses.
1994
1999
2009
2014
2004 A group of German researchers bought see-through AR to cell phone.
2010
Louis Rosenberg develops Virtual Fixtures, one of the first functioning AR systems, for the Air Force. This allowed the military to work in remote areas.
2000
1992
2013 Car manufacturers use AR to replace vehicle service manuals. The Volkswagen MARTA app (Mobile Augmented Reality Technical Assistance), for example, provides detailed information for service technicians, while the Audi AR app uses the iPhone camera to provide details about a vehicle’s features. Toshiba introduces a 3D augmented reality hybrid theater planning application for medical procedures.
FUTURE
Future of Augmented Reality 12
13
THE PAST
CHANGING THE EYES WHICH SEE REALITY
Future of Augmented Reality 14
WHAT IS AUGMENTED REALITY (AR)?
Augmented reality in GPS app Augmented reality in heads-up display Augmented reality in sports
15
Augmented reality sounds horribly theoretical and abstract, but it’s actually very simple: real life plus topical, relevant, background information equals something massively more useful. A few quick examples will make the idea clearer: You’re watching a tennis grand-slam on TV and there’s a controversial call from one of the line judges. Was that serve in or out? The TV station runs an instant replay with a computer animation showing the exact trajectory of the ball and where it landed—just outside the line. Then a little table comes up on the screen showing how many serves have been in or out for each player and how the figures have changed over the course of the match. You’re walking the streets of London, England and you suddenly come across an amazing bit of architecture. What is this fantastic building? Who was the architect? Is that really titanium? You’re dying to find out more, but the building is closed and there’s no information about it at all. So you hold your cellphone up and take a quick photo. The phone uses its built-in GPS (satellite navigation) system to figure out roughly where you are, then quickly searches Google Images to find similar photos taken in the same neighborhood. In a couple of seconds, it’s identified the building and brought up a Wikipedia page telling you all about it.
You’re a fighter pilot flying over a warzone with anti-aircraft fire shooting up at you. You really have to concentrate and looking down at all the gauges on your instrument panel is a distraction you can do without. Fortunately, you’re wearing what’s called a heads-up display (HUD), a set of goggles with built-in, miniaturized computers that automatically project instrument readings so they “float” in front of your eyes. You can find out everything you need to know without taking your eyes off the sky. You’re driving down the freeway sometime in 2020 and you start to feel hungry. Wink your right eye twice and a computer display overlays your windshield with a list of eating places in nearby towns. Wink your eye to select the one that looks most promising and your sat-nav system reads out directions for how to get there. You can email an order in advance so it’ll be ready when you arrive. You can see that augmented reality is actually a mixture of real life and virtual reality, somewhere in between the two, so it’s often referred to as mixed reality. The key point is that the extra information it gives you is highly topical and relevant to what you want to do or know in a certain place and time.
Future of Augmented Reality 16
Head Mounted Display de Ivan Sutherland
WHERE DID AR COME FROM?
17
The first augmented reality head-mounted display system was suspended from a ceiling, and the viewer experienced computer-fed graphics—Sutherland is commonly referred to as the “Father of Graphics.” Other minor developments occurred until the 1990s, when researcher Tom Caudell coined the term “augmented reality,” and Australian Julie Martin brought virtual reality to television. In 1997, Ronald T. Azuma’s “A Survey of Augmented Reality” examined the varied uses of augmented reality such as medical, manufacturing, research, mechanical operation and entertainment. Later that decade and into the 2000s, Hirokazu Kato’s ARToolKit combined virtual graphics with real life, which uses video tracking to overlap computer graphics on a video camera.
Sensorama Machine— Among the first 3D Movies in history
It was in 2008 that the first augmented reality apps came to smartphones. Since then augmented reality has already seen a massive growth in use in mobile devices. Currently augmented reality is still in development for most practical implementations of the technology so most of the applications are either game or entertainment based.
Future of Augmented Reality 18
HOW DOES AR WORK? MOBILE & TABLETS
Through mobile devices like smartphones and tablets, Augmented Reality acts like a magic window; through the viewer you can see holograms and manipulate 3D models. Hundreds of Augmented Reality apps are available on iPhone, iPad, and Android.
Augment has a visualization app for sales people.
19
Augmented reality app for architecture
Siemens presents its newest range of products through augmented brochures
Future of Augmented Reality 20
COMPUTER & TV PLAYERS
On PC and connected TV players, Augmented Reality (AR) works through a webcam and relayed through the screen. Webcam AR content relies on computer software animation plugins, such as the Adobe Flash Player. Programmers essentially encode the assets of models, images, sounds and animations into the AR data stored remotely or on a printed code. These printed codes store data, signify users of AR content and contain simplyshaped markers that help AR applications track the rotation, angle and movement of virtual objects.
21
Sony has unveiled its new Wonderbook Augmented Reality PlayStation 3 peripheral at E3, a folding
User wearing HoloLens and interacting with holograms
Future of Augmented Reality 22
GLASSES & LENSES
On head mounted displays, glasses, and lenses, Augmented Reality becomes a part of your entire field of view, making for more lifelike Augmented Reality experiences. It almost feels like Ironman with the help of Jarvis.
Meta 2 headset
23
Innovega combines glasses and contact lenses for an unusual take on AR
Optical head-mounted displays, or devices that augment our vision either through full-blown glasses or fixed optics that float screens in our peripheral sight, have come to epitomize the cutting edge of wearable tech.
Future of Augmented Reality 24
25
THE PRESENT
THE CURRENT STATE OF PLAY IN AR
Future of Augmented Reality 26
REVAMPING CONTENT WITH AUGMENTED REALITY
No one can deny that physical media like print has been struggling. While this media may never go extinct, it is quickly become outpaced by digital technology. Rather than fight against emerging technologies, marketers should embrace new techniques and actively apply them to new and existing campaigns for more customer engagement. But why stop there? Why not merge the digital with the physical? That’s where Augmented Reality comes in. For those unfamiliar with the technology, Augmented Reality is a digital overlay of information on top of the real world. Usually this information is tied to a physical marker: whether it be an image, an object, geolocation coordinates, or a digitally mapped space. A range of content can be displayed (videos, 3D models, web information, sounds, etc.) and activated once the marker has been recognized. But how does that help you? By linking physical and digital assets you can create a system of tools that work together to build a better experience as a whole, instead working independently. Imagine printing a picture and using it to place life-size products throughout your home. Or browsing through a catalog and ordering by simply turning
your device towards an image and touching a button. Those aren’t hypotheticals, but real applications deployed by real companies to cut back on the barriers of purchase and help consumers understand their products. Take IKEA for instance: For the 2013 catalog, IKEA revamped their mobile application with an Augmented Reality experience that displayed 3D models of various products.
IKEA: AR Feature
27
Augmented Reality for Interior design
This seemingly simple idea of providing product visualization through a catalog resulted in the application becoming the most downloaded, branded app of 2012, despite its July release. The application’s success has led to two more generations (2014 and 2015) that include the ability to place lifesize virtual furniture in your home. On a smaller, more local scale, Phun & Dunn, a San Francisco-based agency, worked with Brewery Achouffe to create an application called “Summon The Chouffe.” The application expanded on Brewery Achouffe’s gnome mascots and beer varietals. A series of Augmented Reality experiences triggered by branded coasters and product labels was included in the application as a way for customers to interact and learn more about the brand. The above examples are all based on print and mobile devices, but the growing trend of wearable technology and more powerful mobile devices hints at a future very much filled with computer vision. As the technology grows more prominent, access to digital experiences and content becomes easier with hands-free hardware and faster mobile connectivity. By leveraging Augmented Reality, marketers can stay in
line with the technology curve as devices such as Google Glass and the Epson Moverio BT-200s become more common and capable of more interaction in future iterations without sacrificing classic mediums like print.
Feature design ideas classy ikea canada iphone app
Future of Augmented Reality 28
AUGMENTED REALITY TREATMENT REDUCES PHANTOM PAIN Phantom limb pain is a mysterious ailment: people with amputations experience aches and acute pains in an arm or leg that isn’t there—making the problem notoriously difficult to treat. But a new type of therapy using augmented reality is surprisingly effective at reducing even the most intractable phantom pain.
The AR therapy method, first proposed in 2014 by Max Ortiz Catalan of the Chalmers University of Technology, just completed its first, highly promising clinical trial. The team selected 14 amputees whose phantom limb pain was chronic and unresponsive to other therapy methods. The patients were equipped with myoelectric sensors that detect the signals in muscles that once controlled the missing limb. These signals are tracked and analyzed, and linked to movements in a virtual environment— opening the hand or twisting the wrist of an on-screen limb. Once this calibration is complete, the virtual limb is superimposed on a live webcam image of the patient, starting just where the real limb stops. The user thinks of movements, and the virtual limb executes
them. Over 12 semimonthly sessions, patients were asked to put the virtual limb into various positions, use the sensors to control a car in a racing game, and so on. Amazingly, by the end of the 12 sessions, reported pain was reduced by about half, and interruptions of daily activity or sleep from it were similarly cut down. The four patients on pain medication reduced their dose, two of them by 81 percent. Six months later, the improvements were still present, implying a lasting therapeutic benefit. “The results are very encouraging, especially considering that these patients had tried up to four different treatment methods in the past with no satisfactory results,” said Catalan in a news release. “We also saw that the pain continuously decreased all the way through to the last treatment. The fact that
29
the pain reduction did not plateau suggests that further improvement could be achieved with more sessions.” If the idea that moving a virtual limb around in AR could relieve pain strikes you as strange, don’t worry—it is. But phantom pain is a poorly understood phenomenon and sometimes the effectiveness of treatments is matched only by their strangeness. Phantom itches are also a problem, for example: imagine how maddening it must be to have an itch you can’t scratch because the limb it’s on isn’t there. The solution, some have been lucky enough to find, is to arrange mirrors so that a limb that’s present appears to be in the place of the missing one. Someone scratches it, and the phantom itch disappears. Believe it or not, this and other forms of mirror therapy are established practice—though not always effective. This AR-based method is sort of like mirror therapy taken to the next logical level, and it may prove a valuable tool in the treatment of this mysterious but very real condition.
Philips Surgical Navigation Technology based on Augmented Reality
An amputee patient tries out the arm system
Future of Augmented Reality 30
How boring is it on a scale of 9 to 10 flipping pages and reading those clustered words and trying to figure out those two-dimensional diagrams? With several developments in technology, we can thoroughly and accurately make use of augmented reality in the field of education. Imagine you are an engineering student majoring in mechanical engineering, and you just purchased this Turbocharging Performance Handbook. On the cover page of this book is a car fitted with engines, and without a chassis. Now, what if you could see the 3D image of this car? And not just from one angle! You could see the details of the engine, how the various parts of the car are connected etc. All this just by using an Augmented app and scanning the image on the book. Studying becomes so much easier when you stop picturing and start seeing and observing. How much simpler it would become if you could see the working of the human jaws and teeth, the 3D model of a molecule, or even an augmented model of a monument! AugmentedReality in automotive industry
AUGMENTED REALITY IN EDUCATION
31
Octagon Studio Kid Learning 4D+ Flash Card (Animal+Space)
Augmented Reality on tablet computer
Classroom education’s another area where reality is well worth augmenting. If you want to teach people about the stars and planets, for example, what better way than to use a smartphone or tablet that can help them label the things they can see in the sky above their heads? Sun Seeker and Star Walk are just two of many AR applications giving students an inspiring new way to find their way around the world. Satellites and passing flights can be tracked the same way; check out the app version of Flight Radar for example.
Future of Augmented Reality 32
Pokémon Go sows augmented reality seeds in South Africa
“Entertainment’s future has arrived, and it has come in the form of a developing technology known as augmented reality.”
ENTERTAINMENT APPLICATIONS IN REALITY TECHNOLOGY
33
In less than a week, the mobile game Pokémon Go has become the most downloaded app in Apple’s App Store.
The fashion industry has had tremendous growth and has been developing ever since it has adapted to augmented reality. How about shopping without having to change into the clothes on the rack physically? Sounds good, doesn’t it? This can only happen by using augmented reality. Uniqlo has given its customers the ability to try out a variety of clothes without having to keep changing into them. Now, we don’t have to go by the ‘only three clothes allowed in the trial room’ rule. We can choose anything we like, any color we like and any size we think would fit and see how we’d look in it. The famous brand Sephora has also gotten a hold of the AR concept and has teamed up with facial recognition company called ModiFace and has launched an app called Virtual Artist which allows you to see how you would look with the Cherry Skies lipstick on or the retractable waterproof eyeliner wing. Now you don’t even have to put your makeup on to test it and buy it. Just use AR and get it done with.
3D Virtual fitting dressing room and mirror
Future of Augmented Reality 34
35
THE FUTURE
FIVE WAYS AR WILL IMPROVE YOUR LIFE
Future of Augmented Reality 36
INTRODUCTION
AR has been hot on the list of future IT tech for 25 years. It has been used for various things since smartphones and tablets appeared but really hit the big time with the recent Pokemon craze. To get an idea of the full potential of augmented reality, recognize that the web and all its impacts on modern life came from the convergence of two medium sized industries—telecoms and computing. Augmented reality will involve the convergence of everything in the real world with everything in the virtual world, including games, media, the web, art, data, visualization, architecture, fashion and even imagination. That convergence will be enabled by ubiquitous mobile broadband, cloud, blockchain payments, IoT, positioning and sensor tech, image recognition, fast graphics chips, display and visor technology and voice and gesture recognition plus many other technologies. Just as you can put a Pokemon on a lawn, so you could watch aliens flying around in spaceships or cartoon characters or your favorite celebs walking along the street among the other pedestrians. You could just as easily overlay alternative faces onto the strangers passing by. People will often want to display an avatar to people looking at them, and that could be different for every viewer. That desire competes with the desire of the viewer to decide how to see other people, so there will be some battles over who controls what is seen. Feminists will certainly want to protect women from the obvious objecti-
37
fication that would follow if a woman can’t control how she is seen. In some cases, such objectification and abuse could even reach into hate crime territory, with racist, sexist or homophobic virtual overlays. All this demands control, but it is far from obvious where that control would come from. As for buildings, they too can have a virtual appearance. Virtual architecture will show off architect visualization skills, but will also be hijacked by the marketing departments of the building residents. In fact, many stakeholders will want to control what you see when you look at a building. The architects, occupants, city authorities, government, mapping agencies, advertisers, software producers and games designers will all try to push appearances at the viewer, but the viewer might want instead to choose to impose one from their own offerings, created in real time by AI or from large existing libraries of online imagery, games or media. No two people walking together on a street would see the same thing. Interior decor is even more attractive as an AR application. Someone living in a horrible tiny flat could enhance it using AR to give the feeling of far more space and far prettier decor and even local environment. Virtual windows onto Caribbean beaches may be more attractive than looking at mouldy walls and the office block wall that are physically there. Reality is often expensive but images can be free. Even fashion offers a platform for AR enhancement. An outfit might look great on a celebrity but real life shapes might not measure up. Makeovers take time and money too. In augmented reality, every garment can look as it should, and that makeup can too. The hardest choice will be to choose a large number of virtual outfits and makeups to go with the smaller range of actual physical appearances available from that wardrobe. Gaming is in pole position, because 3D world design, imagination, visualization and real time rendering technology are all games technology, so the biggest surprise in the Pokemon success is that it was the first to really grab attention. People could by now be virtually shooting aliens or zombies hoarding up escalators as they wait for their partners. They are a little late, but such widespread use of personal or social gaming on city streets and in malls will come soon.
Future of Augmented Reality 38
THE FUTURE DEVELOPMENT OF AUGMENTED REALITY AR is open and partly immersive. AR users continue to be in touch with the real world while interacting with virtual objects around them.
The likelihood of purchasing the product
61%
75%
86%
Gamification app
Virtual vehicle app
Virtual furniture app
65%
75%
88%
Catalog app
Shopping app
Virtual dressing app
39
AR games
AR theme park
AR consumer AR adspend
EnterpriseAR
AR hardware
AR film/TV
Augmented Reality Revenue Share 2020 AR voice
AR data
Commerce
Future of Augmented Reality 40
41
THE FUTURE OF HUMAN INTERACTION
A project for GE (General Electric) where we worked closely with renowned Photographer Markku Lahdesmaki, we created this very technical image, almost like an architects tenchnical drawing or a future augmented reality over one of their Jet engines.
Future of Augmented Reality 42
Social Media unlimited
43
Where is all this heading over the next few years? It’s beginning to look like a real business, just as mobile did nearly a decade ago. Mobile analyst Tomi Ahonen expects AR to be adopted by a billion users by 2020. Intel is betting that AR will be big. The chip maker is investing $100 million over the next 2 to 3 years to fund companies developing “perceptual computing” software and apps, focusing on next-generation, natural user interfaces such as touch, gesture, voice, emotion sensing, biometrics, and image recognition. Apple isn’t in the AR game yet, but the company has been awarded a U.S. patent, “Synchronized, interactive augmented reality displays for multifunction devices,” for overlaying video on live video feeds. Eyewear will evolve over the next year with comfortable stylish glasses with powerful embedded technology. They will range
from Google Glass-style glance-at displays that also replace the phone to stereoscopic 3D-viewing wearables for everyday use. “You’ll get 20/20, perfectly augmented vision by 2020, with movie-quality special effects blended seamlessly into the world around you,” said Dave Lorenzini, founder of AugmentedRealityCompany.com and former director at Keyhole.com, now known as Google Earth. “The effects will look so real, you’ll have to lift your display to see what’s really there. There’s more of the world than meets the eye, and that’s what’s coming.” He cautioned that the growth of the AR industry could be slowed by a lack of standards to connect disparate players and their formats for bringing a 3D digital layer to life. “The AR industry has to get together to power the hallucination of what’s do come,” Lorenzini said. He added that a key turning point will be the availability of the
Future of Augmented Reality 44
WYSIWYG (What You See Is What You Get) real-world markup tools needed to bring this digital layer to life. When the AR industry does take off, Lorenzini envisions a trillion dollar market for animated content, services and special effects layered into the real world. “Imagine people tagging friends with visual effects like a 3D halo and wings, or paying for a face recognition service to scan and add a floating name tag over the head of everyone in a room,” he said. “AR will grow from specific vertical uses to mass market appeal, driven by young, early adopters. “Anyone reviewing devices like Google Glass needs to take it to their kids’ school before they pass judgement,” Lorenzini added. “This is not a device from our time, it’s from theirs. They love it, use it effortlessly, and are totally unfazed by ad targeting or privacy concerns. It will be be a natural part of who they are, how they learn, connect and play.” Eventually, wearable technology will become more integrated with the human body. With advances in miniaturization and nanotechnology glasswear will be replaced with contact lens or even bionic eyes that record everything, make phone
calls and allow you to use parts of your body, or even your thoughts, to navigate the world. “Contact lenses are difficult now but the bionic eye will become commonplace and AR will just be a feature,” Kipper said. “Some may choose to have eyes in back of their heads, and some won’t. Some will want to be cyborgs. We will always use tools as advanced as they can be to help ourselves.” Brian Mullins, CEO of Daqri, an augmented reality developer of custom solutions, went even further in melding humans and technology. “Thinking is the future of AR,” he said. Mullins talked about measuring “thought intensity” with EEG machines and focusing the mind to manipulate objects during a panel discussion at the Augmented Reality Expo. Of course, the technical challenges are accompanied by issues of social etiquette and privacy. Smartphones are now a wellaccepted part of daily life in most countries, but issues around data ownership and access to the data abound. The subtlety and potentially always-on capacity of wearable technologies will create more privacy concerns and challenges to acceptance. Feiner acknowledged that it’s “scary” in terms of the information available, especially
45
Gestures & Augmented Reality in Iron Man
Future of Augmented Reality 46
Augmented Research Datapowered qualitative research for the network age
An interactive science teacher for Augmented Reality
when billions of people with cameras and microphones can capture anything in public. “There are no laws against it,” he noted. He gave Google some compliments for not overloading Glass with features. “It not suffering from doing too much too soon,” he said. Whether Google Glass is the tip of the spear for the mass adoption of far more powerful AR is uncertain, but it is doing a good job of surfacing the issues around the introduction of a disruptive, new way of computing. Nicola Liberati, a Ph.D. student in philosophy at the University of Pisa studying the intersection of humans and technology, suggested another line of thinking about AR in his presentation at the expo. “We should not focus our attention only on what we can do with the such technology, but even on what we become by using it.” So, when you are strolling down the street wearing the latest digital eyewear from Google, Apple or some as yet unformed or now early-stage company, with your continuous partial attention on the 3D holographic screen feeding you all kinds of personalized information about the environment around you, zeroing in on the people and places in your field of view or piped in remotely from around the real and virtual worlds, and spaces in between, think about what we have become.
47
Telit: Making “The Internet of Things” Video With Maxwell
Future of Augmented Reality 48
“AR has great potential to transform our cities and the way we learn and discover within them.�
49
CHANGING LIFE FOR THE BETTER Improvements to the technology means more promise for AR—and 2.5 billion AR apps are expected to be downloaded by 2017. But if you’re wondering how you’ll actually use AR, we’ve outlined seven real ways in which you, city dwellers across the globe, might soon use this bleeding-edge technology.
Future of Augmented Reality 50
Urban Exploration In a new neighborhood or exploring another city? Ditch the Fodor’s and grab an AR app that shows you what’s nearby and where you should go. These AR apps let you filter by category so you can find exactly what you’re looking for, whether it’s a coffee shop, restaurant or museum. And you won’t need to worry about getting turned around by the map—the AR app will adapt based on what you’re facing, so it’ll tell you to turn right and get you to your destination, as opposed to just indicating that you should walk northeast (how are you supposed to know which way is northeast?). This kind of AR app already exists —check out Nokia’s City Lens, Wikitude and Metaio’s Junaio and there are more to come.
Lord explained another fun use of AR beyond helping you get around. In Munich, for example, the clock known as Glockenspiel chimes at 11 a.m., and little characters act out “scenes.” But what if you’re not there at 11 a.m.? Metaio created an overlay of the animation of the routine so that you can “see” the animation when you hold your smartphone up to the Glockenspiel, in effect letting you experience the moment any time of day. This technology could be applied for other time-sensitive events at landmarks, such as the Changing of the Guards or the illumination of the Eiffel Tower during the holidays, for instance.
51
The Museum of London has just launched an iPhone app that makes use of its extensive art and photographic collections as well as geo tagging and Google Maps
A view of Bow Lane, off Cheapside in the City of London, looking south to the crossing with Watling Street and St. Mary Aldermary in the middle distance.
And another fun way to explore your city —or any city—is to overlay 3D maps that show what the city looked like at any point in history. If you’re walking down Broadway, you would be able to “see” horse-drawn carriages parked on the street in front of old parlors similar idea would be to overlay what your city used to look like on top of the current layout. If you live in New York, you could walk around downtown and your phone would geolocate your positioning and put a virtual layer on top of the street, letting you “see” the wall after which Wall Street is named, carriages parked on the street and the cobbler shop on the corner. It’d be like walking in a history book.
Future of Augmented Reality 52
Museums Visiting a museum? Metaio did an integration with the British Museum where there were AR hot spots that offered more information, and the Junaio AR browser basically “attaches” information to the art so you don’t need to buy one of those audio tours. Especially in the case of modern art, says Lord, “You could walk up to nearly any painting in any museum and the [AR] recognition will work on it,” using LLA (Longitude, Latitude, Altitude) to navigate indoors. Metaio also experimented with 3D virtual “docents,” who are placed throughout the museum—but only visible through the browser—and can tell you more about the art in nearby exhibits. Lord explains that this is a helpful tool, especially when you
are in a large museum, like the Louvre or New York’s Metropolitan Museum of Art, where maps don’t always help you find what you’re looking for.
American media artist Tamiko Thiel will launch a new exhibition this September entitled ‘Fractured Visions’, which uses Augmented Reality and smartphones
Artist’s impression of the Oceans and Reefs Gallery at the Australian Museum
53
Virtual Reality Sweeps Shoppers Into New Retail Dimension
Shopping Augmented reality is going to radically change the shape of commerce, says Rosenthal of Goldrun, who created an AR pop-up shop for Airwalk in a New York City park. AR could turn places as mundane as parks and airports into shopping destinations, which would be a great way to kill time (and a smart way for businesses to save money on commercial real estate). If there’s no Uniqlo or Crate & Barrel in your city, AR could change that, and you could browse the stores virtually, using your phone. Think of it as v-commerce, as opposed to ecommerce.
But even brick-and-mortar locations could integrate augmented reality into their design. They could show you items that will be on shelves soon, or items that aren’t sold at that location—and soon you’ll even be able to feel these items through your phone. And have you ever thought about how much storefront designs costs and that they’re simply thrown out after? AR could spice up window displays and cut costs if ornate tangible displays are instead be presented digitally.
Future of Augmented Reality 54
Travel & History If you’re looking for budget “travel” options or a quick “getaway,” you could find a solution in augmented reality. Just plop the Eiffel Tower or the Leaning Tower of Pisa right in your backyard and unlock monuments during a sort of virtual vacation, and you could learn tidbits about each one as you go. It’s be a great way to teach your kids, too—“You could have your kid pose with each monument and basically take a ‘trip around the world,’” says Rosenthal. Using AR in this way would be great at home and in classrooms, where history teachers could take students on a “class trip”
to the Great Wall of China and even pose for a picture, making education deeply personal and thus, more memorable. This, of course, is different from the AR uses mentioned in, since you wouldn’t need to be physically in front of the monuments to see them with AR.
The results obtained demonstrate the potential of AR applications for the dissemination of historical heritage
55
Airports experiment with the latest virtual technologies to improve customer service
Customer Service No one likes having to call customer service —you’ll be put on hold and stuck listening to a script recited by a rep. But in the future, if you’re having trouble setting up Apple TV, or your cable cuts off, you can have customer service come to you. Metaio’s AR software can access the user’s camera (Lord assures us it’s not as creepy as it sounds), so if you’re setting up technology at home and having problems, the support team can access the camera and in real-time, overlay instructions through the camera. So instead of hearing generic instructions, like “Unplug the red cord”
and “double-check the port,” someone could walk you through the process and see the things you’re seeing, enabling the customer service rep to point things out in more detailed, visual way and helping problems get solved in a more efficient manner.
Future of Augmented Reality 56
Safety & Rescue Operations Chris Grayson, an AR expert, says, “The enterprise space and government employees could see the first real-world benefits” of AR. Emergencies are a fact of life, and first responders, police and firefighters often arrive at chaotic scenes and need to make sense of the environment and navigate a place they’ve never been. Wouldn’t it be cool if they could see a virtual map of the site or have “X-ray vision” to see underground water and power lines?
The future of AR is firefighters with infrared breathing masks
57
Augmented reality user interface by Jaakko Ranne
Maybe this is only a problem in Manhattan, where we live in shoeboxes, but AR has fun and useful applications when it comes to moving day. There’s no worse feeling than buying furniture, paying the delivery fee, having someone schlep it up five flights of stairs, but it not fit through the doorway. What if you searched through an app and pulled up the Macy’s bedframe, IKEA dresser (IKEA has already experimented with AR, see below) and Jennifer Convertibles sofabed through an app and virtually positioned them in your home so you could see what they’d look like—and whether they would actually fit—before you head to the store and pay?
Moving & Decorating Your Home
Future of Augmented Reality 58
59
PEERING INTO THE FUTURE OF AR GAMES
Future of Augmented Reality 60
61
Not only do you have the option to play games on your mobiles, but you have the chance to try the wonders augmented reality offers.
Wearables and Tech: Expanding Horizons Wearable tech will interface both with other devices and services (PCs, smartphones, tablets) and with traditional video game tech and designs, as well as the cloud. What effect will these have on the direction developers take? Narayanan sees three basic ways in which wearable tech will interface with other technologies: “First, the interfaces between wearable devices themselves. Second, interfaces between wearables and the tethered smartphone, and lastly, a shared game space in the cloud where games as complex as MMOs could be realized with players using different devices to play in the same virtual world.” “It makes sense for wearable devices to work together to share data, gamestate, and network connections and for play experience to incorporate many different types of interactions that may take places on different devices at different times,” says Hanke. “In the end we’ll see smart, small, stylish devices, a super-reliable personal cloud, maybe a single device providing and sharing highspeed network connectivity, something for voice input and some kind of high-fidelity but discreet audio output, and maybe video.”
Ellsworth and Johnson, meanwhile, suggest some practical software-based considerations, anticipating how the devices will interface with game development tech: “Physics plays an important new role in this space. Having the user interact in 3D into the virtual world with physics makes the player feel much more connected to the environment. Audio becomes more alive as it is now based upon your position in the physical world: Move your head closer to a virtual music box to hear it. Finally, projected AR allows multiplayer games which utilize fog of war or other hidden data mechanisms to retain those concepts even when players are sitting right next to each other.” The Ultimate Potential of Wearables Already, Meta has built a Minecraft mod called Metacraft that allows you to “build with boxes in your room, really neat stuff,” says Kitchales. The Niantic team is clearly experimenting with Ingress-like shared experiences, even if they’re under wraps for now. But what’s the long-term potential for games that use this kind of technology? Johnson and Ellsworth offer a subtle but important insight: “We see AR bringing the social back into social gaming. People can now sit across from each other rather
Future of Augmented Reality 62
than side by side, watching their opponents react to the gameplay.” But they have other ideas, too: “We’ve demonstrated that you can use your head as an input device as control for a flight simulator, so exploring more ways your physical presence will have on gameplay will be interesting.” “If you move [an existing] game to Google Glass, there’s a different control scheme,” says Narayanan. “Perhaps one’s head might be used to navigate the object on the screen, or hand gestures might be the inputs to the controller. In either case, we certainly believe there will be compelling new experiences by re-imagining them on wearable computing devices.” “Bridging the gap between the physical world and the virtual world, by using miniatures, cards, board game pieces, etc. will continue to enhance the experience,” Ellsworth and Johnson note. Hanke, meanwhile, actually dismisses the idea of extending current video games into the AR space. “In terms of taking an existing game mechanic and extending it with some wearable tech, perhaps but that’s not so interesting to me,” he says. What does he want to see? “I’m thinking about a range of tech enhancement ranging from small pulses on your wrist letting you know that something just happened all the way to full-blown augmented reality that transforms your world into the one you’ve always wanted to live in.” These devices will be expected to perform
very capably by players used to rich game experiences, Narayanan says. “Consumer expectations around mobile games are rising. For example, consumers now are used to enjoying rich 60 FPS 3D game experiences on the smartphone.” “In a similar fashion we anticipate that more and more apps will take advantage of the increasingly sophisticated sensor capabilities on these devices to meet emerging customer expectations about richer experiences.” He also sees it as essential to use the data that devices can know about players and the world to create the next generation of wearable game experiences: “The real insight here is that these new wearable games will be micro-transactional in nature. Gamification of all this data that is unique to wearable technology is essential. It’s horizontal, not vertical. We can see this already with health applications. People don’t want to see a chart or graph of their heart rate. They’d rather play a game, test their stamina, clock their mile run. Not only is this beneficial to the consumer but for companies as well, as this type of interaction helps build more meaningful costumer relationships.” If that doesn’t interest you, you can think about it this way: “You can walk around the house and start a game, and you’ll see zombies coming out of the walls,” says Kitchales. “The ideas are limitless -- and there are so many cool ideas and so many ways to use this new technology. It just depends which kind of person you are.”
63
Wearable technology
Future of Augmented Reality 64
USING AR FOR INTERIOR DESIGN AND PRODUCT DESIGN
65
Future of Augmented Reality 66
Bathstore shakes up bathroom retailing with AR tool
Noken Augmented Reality App
67
Today, designers and manufacturers are turning to augmented reality (AR) to replace their physical products. Augmented reality allows for faster design iterations at a fraction of the cost. Augmented Catalogue With Augment you can link a 2D print material to either a single or multiple 3D models. For instance, your customer can scan your product catalogue using the AR app to access your digital augmented product catalogue. This allows your customer to try all your products at home in AR. It adds a new dimension to your marketing material and engages your audience. Augmented Website Augmented is also solving today’s e-commerce challenges. It offers consumers the opportunity to test products from your
online store in their home before making the purchase. This experience helps eliminate the guess work and sell more efficiently. Multiple Models No longer will you and your customer have to guess if the sofa and side table will work with the size, layout and design of the room. With Augment, you can visualize multiple models at the same time to find the perfect configuration. Static Background With Augment, even if you aren’t on site with the client, you can use a saved background to simulate your product in the their environment. This ensures your products are suited for the clients’ space during the prototyping and design process. Color Configurator What if your product comes in different colors and you want to show your customers all the choices? Augment has a color configurator feature available. It allows you to change your 3D model’s color from within the app. You can customize different components of the same product, giving your client a full array of color combinations to choice from. Sharing & Communication You can also share your product visualizations from within the Augment app, making the communication of ideas with your client and team easier than ever before. This reduces the back and forth communications and shortens your sales cycle.
Future of Augmented Reality 68
69
MAKEUP BRANDS ARE TESTING AR TO DRIVE CONVERSIONS
Future of Augmented Reality 70
While shopping for a new lipstick on Sephora’s mobile app, customers contemplating whether to buy Marc Jacobs’ Forbidden Berry shade or Stila’s Magenta Plum can see exactly how the colors will look on their faces. The Sephora Virtual Artist tool in the retailer’s app launched in January to help customers test products using AR technology powered by ModiFace. The visual artist tool lives on the Sephora app’s homepage, and as customers “try on” different lipstick shades using the front-facing camera, they can move around and the color stays put. An “add to basket” button is featured at the top of the screen as colors are tried on. The feature, Sephora believes, is more than a gimmick, as it can help customers make purchase decisions while shopping on their mobile apps. Augmented reality can help a client buy a product by allowing her to try it on, boosting her confidence that she knows how to use it, or giving her an experience that informs
Augmented Reality In the Makeup Industry
her about how and why it was created,” Bridget Dolan, vp of Sephora’s Innovation Lab, told Mobile Commerce Daily. While Sephora said it was too soon to disclose conversion, the app has seen 1.6 million visits and 45 million “try ons” in the eight weeks since launch. Sephora is not alone. According to a Demandware study, 72 percent of U.S. beauty brands are testing a form of “guided selling” to push sales, like augmented reality. L’Oreal’s MakeupGenius app launched last year to help consumers test the drugstore brand’s products and now has 6.3 million downloads. In May, Benefit Cosmetics released the “Brow Genie,” a digital tool that virtually transformed users’ eyebrows before demonstrating how to recreate the shape. Elizabeth Arden teamed with the app YouCam, which has 100 million users, to let customers virtually try on products and order them from the brand. “There are always going to be the people who need to be in store and in person,” said Zach Paradis,
71
director of experience and innovation “Generic overlays of color look clownish, strategy at Sapient Nitro. “But retailers need and that can actually hurt sales,” said Aarabi, to support new ways to buy and experience, adding that when people see blobs of and augmented reality color on their faces that don’t well represent is a very powerful way to do it, specifically the product, they’re turned off by both the in makeup and beauty.” product and the retailer. To be successful, though, the technology ModiFace tells brands that its technology has to be up to par. “For these applications, can drive conversions online and mobile it is very important to focus on realism,” by 80 percent and in store by 30 percent if said Parham Aarabi, CEO of ModiFace, the used properly. Aarabi said that the cusvirtual makeover technology company that tomer experience around the tool affects created the augmented reality experience conversion as well as the technology. for Sephora, L’Oreal and other brands. How the tool is presented, and how easily “We’re constantly improving how colors customers can add to cart and buy, can appear when they’re applied to ensure it’s make or break the sale. taking texture, light reflections and skin Paradis believes that the brands who get tone into consideration.” out first in this field, however, will be rewarded. Aarabi, a former University of Toronto “In augmented reality, what may have been professor, launched his company after gimmicky five years ago is now believable. It’s creating a simulator that could predict what crossed that threshold into usefulness,” said botox injections would look like on people Paradis. “It can manifest into meaningful of different ages. Sixty-five brands now decision support in the path to purchase that use ModiFace’s products for virtual skincare, can differentiate a retailer. It helps that it’s fun.” hair color and makeup predictions. He said that his company studies how its tools drive conversion in “excruciating detail,” because when the technology is not used properly, it can have a damaging effect for the brand.
Augmented Reality makeup artist
Future of Augmented Reality 72
CONCLUSION
73
Augmented reality is another step further into the digital age as we will soon see our environments change dynamically either through a smartphone, glasses, car windshields and even windows in the near future to display enhanced content and media right in front of us. This has amazing applications that can very well allow us to live our lives more productively, more safely, and more informatively. In the future, we will see our environments become augmented to display information based on our own interests through built-in Radio-frequency identification tags and augmentations being implemented through holographic projections surrounding the environments without a use of an enabling technology. It would be incredible to no longer wonder where to eat, where to go, or what to do; our environment will facilitate our interactions seamlessly. We will no longer be able to discern what is real and what is virtual, our world will become a convergence of digital and physical media.
Future of Augmented Reality 74
CREDITS
Articles wired.com/2014/06/well-soon-be-researching-3-d-objects-just-by-looking-at-them/ cnet.com/news/the-next-big-thing-in-tech-augmented-reality/ uxmatters.com/mt/archives/2009/08/inside-out-interaction-design-for-augmented-reality.php newsmax.com/Hirsen/Future-Entertainment-Augmented-Reality/2015/03/23/id/631923/ techcrunch.com/2016/12/02/augmented-reality-treatment-reduces-phantom-pain-in-missing-limbs/ sfama.org/revamping-content-with-augmented-reality/ augment.com/how-augmented-reality-works/ assignment10ct231.blogspot.com/2012/03/assignment-10-augmented-reality.html mashable.com/2012/09/24/augmented-reality/#PiBIpyVVFsq2 augment.com/blog/augmented-reality-for-interior-design-and-product-design/ gamasutra.com/view/news/203856/Peering_into_the_future_of_augmented_reality_games.php wareable.com/vr/virtual-reality-vs-augmented-reality-which-is-the-future mashable.com/2012/12/19/augmented-reality-city/#3MknbGee9Pqg timeguide.wordpress.com/2016/08/16/future-augmented-reality/ computer.howstuffworks.com/augmented-reality4.htm www.theguardian.com/technology/2016/oct/23/augmented-reality-development-future-smartphone interaction-design.org/literature/article/augmented-reality-the-past-the-present-and-the-future dailymail.co.uk/sciencetech/article-2567739/Streetmuseum-app-creates-hybrid-images-London.html timetoast.com/timelines/87208 t-immersion.com/augmented-reality/use-cases
75
Pictures mashable.com visualpathy.com augmentedrealitytrends.com philips.com coetail.com sfama.org obeyproximity.com cnet2.cbsistatic.com static.leiphone.com img-2.newatlas.com coolthings.com augment.com media.licdn.com .webanywhere.co.uk whatafuture.com mir-s3-cdn-cf.behance.net s-media-cache-ak0.pinimg.com sae.org visualpathy.com designdirectory.com bridginggaps.org retail-week.com fashionandmash.com cdn.psfk.com
Designed by Yitian Ma mayitian025@163.com Fonts Avenir by Adrian Frutiger Archer by Tobias Frere-Jones and Jonathan Hoefler Printed by Epson Artisan 1430 Binding by Yitian Ma Paper Epson Premium Presentation Paper
77
Future of Augmented Reality 78