Future of AR

AR coming of age

Microsoft’s HoloLens and the use of reality meshes have got round tracking issues, paving the way for exciting new AR applications from Bentley, writes Greg Corke

Bentley Fellow Stéphane Côté lives and breathes Augmented Reality (AR). For years, he’s been a driving force behind AR in the built environment, and has led several research and development projects.

The applications have been far reaching — from construction planning and subsurface utility visualisation to asset queries and plant maintenance. All use the physical world as a visual gateway to digital data, whether that’s an augmented model, hologram, datasheet or task notes. Simply point a tablet or look through an AR headset and the relevant information is there, instantly, in context. No more looking up file names, searching for references, or working off paper drawings.

Côté admits that tracking issues — not being able to augment the physical world with sufficient accuracy — have held back developments.

Unless the augmentation appears, and behaves, exactly as if it were in the physical world, mistakes will be made, he says. And if mistakes are made, the technology will not be used.

Côté has experimented with various positioning techniques, including beacons, total stations, GPS and feature recognition, none of which have been perfect. However, as he revealed at the recent Bentley YII conference, there have been two major breakthroughs. For outdoor applications, Bentley has developed an alternative approach to AR that uses a reality mesh instead of a live camera feed. For indoor applications, it’s Microsoft’s HoloLens, the AR headset that everyone is talking about.

Outdoor AR - mesh augmentation Most AR tracking techniques capture a video stream from a tablet’s camera to ‘recognise’ visual features in the physical world, such as a lamp post, hydrant or steel section. However, due to changing conditions, such as shadows and wind, or moving objects like cars, Côté’s team has found that augmentations can be inaccurate or ‘shaky’.

Rather than exploring more ways to master the physical world with a live video feed, this has led to a change in tack at Bentley – the use of the reality mesh for context.

Reality meshes are created in advance, by a collection of photographs taken by a drone and processed with Bentley ContextCapture.

Then, later, wherever the user points the tablet on site, it displays the corresponding part of the mesh. This is tracked by the tablet’s GPS.

While GPS isn’t 100% accurate at tracking position and orientation, it is good enough to enable the user to visually match features displayed on the mesh to those in the real world. The important thing here is that the mesh is always perfectly aligned to the augmentation. This means that accuracy can be quantified and certified, so users can rely on the information, which is essential for engineering applications.

In addition to ‘solving’ the tracking issue, Côté says there are other advantages. Augmentation sessions can be carried out during the night, or when there is snow or rain, as the technique does not rely on visual tracking. Also, augmentation can be done remotely. This, says, Côté, could enable teams to plan site visits in advance.

There are downsides. First, it requires the 3D reality mesh to be captured in advance, and this mesh must be kept up to date. Reality meshes for entire cities are already available, says Côté, citing a 1,000km2 city model of Paris that exisits, but he admits that the realism of the augmentations will depend on the resolution of the images used to produce the meshes — so drone photos will likely produce more detailed meshes than aerial photos.

Check out the example video to see the technology in action tinyurl.com/ARmesh.

Indoor AR - Microsoft HoloLens The HoloLens is a self-contained, holographic AR headset, which displays a hologram directly in the sight line of the wearer, so it looks like the object is in the physical world. Users can simply look at objects to access data, pinch their fingers together to select, or use voice controls. The big advantage over a tablet is that both hands are kept free.

With multiple sensors, the device offers truly exceptional tracking for indoor applications, says Côté, solving Bentley’s long-standing tracking issue in an instant.

The HoloLens is not geo-referenced — i.e. does not know its location in the physical world. But it is very good at recognising interior spaces and its exact position within them.

Bentley has been exploring several applications for the HoloLens, including “Construction planning in context”, “X-Ray vision for building maintenance”, and “Plant maintenance”. These are best illustrated by video, so we strongly recommend you check out the Bentley Research YouTube channel tinyurl.com/ BentleyAR.

Côté says the HoloLens does not work that well outdoors, as it can be affected by sunlight. Its form also needs to change so it can be used on construction sites — i.e. ruggedized and built into a hard hat. However, as a first release technology, it’s very exciting. There are also many more AR headsets in development.

Conclusion Bentley sees AR becoming a hugely important technology for the built environment — and we tend to agree. But it is only starting to scratch the surface of what is possible.

Now that it no longer has a tracking issue to solve, it can focus its R&D efforts on developing more practical use cases.

We expect 2017 to be a pivotal year for AR as Bentley starts to work with partners in industry on real-world applications, hopefully taking it one step closer to delivering a commercial AR solution.

■ bentley.com

1 Bentley’s mesh augmentation technology - designed to offer an accurate and stable alternative to live augmented reality. View the video at tinyurl.com/ARmesh 2 Mixed reality mixed reality experience of building construction using MicroStation and a HoloLens. View the video at tinyurl.com/HoloLens-building 3 Using the HoloLens for plant maintenance. Voice commands are used to help keep the user’s hands free in order to carry out the maintenance work. View the video at tinyurl.com/HoloLens-plant

LumenRT embraces VR

LumentRT may be owned by Bentley, but it’s not just for Bentley customers. The easy-to-use viz tool offers Revit, ArchiCAD and SketchUp users a gateway into a powerful ‘game engine’ environment and it now supports VR, writes Greg Corke.

Last year, Bentley acquired design viz specialist e-on software and its real time rendering tool, LumenRT. If you haven’t seen the software in action, the output is quite spectacular – think interactive ‘game engine’ experiences with realistic trees, people, water, wind and moving vehicles.

Now LumenRT is embracing Virtual Reality (VR) in a big way. The software currently works with Google Cardboard, but full-blown immersive VR experiences with the Oculus Rift and HTC Vive will be available before year-end.

Google Cardboard Google Cardboard lets you turn your smartphone into a VR headset for around £15. Simply plug in your Samsung Galaxy or iPhone and away you go.

Most AEC-focused Google Cardboard experiences are static, where the viewer can experience a 360 degree panorama from a fixed point. However, the proposed highway design demo we were shown at the recent Bentley YII event in London was much more dynamic, allowing us to view the proposed design from any angle as we flew along a fixed path. Even though we were essentially looking at a succession of static VR panoramas, for an entry-level (and low-cost) VR experience, it was very impressive.

Fully immersive VR Things should get even more exciting next month when Bentley ships LumenRT Connect Edition Update 2, which will feature support for the Oculus Rift and HTC Vive. This will deliver a fully immersive, interactive experience where users can freely explore LumenRT’s 3D worlds populated with moving vehicles, animated characters and wind-blown skies.

Three years in development, Bentley explored a whole range of VR game engines, such as Unreal and Unity, but decided to adapt its own engine instead. This decision was based on two factors: First, the ability of its engine to scale from small buildings to giant infrastructure projects, above and below ground, and second, the fact that it already had built-in tools for generating realistic AEC-specific features such as cars with wheels that move and traffic flows that are real, thanks to direct interfaces to MicroStation Traffic and VISSIM.

CAD users outside of Bentley’s customer base will benefit, too. In addition to MicroStation, AECOsim, SITEOPS and other Bentley tools, the software works with Revit, ArchiCAD, CityEngine and SketchUp. Bentley says the workflow from CAD to VR is simple. Getting CAD data into LumenRT only takes minutes to process, even with big scenes. Then, because the geometry is already optimised, moving into VR only takes a matter of seconds.

For distribution to clients and colleagues, projects can be packaged up as a ‘LiveCube’, a self-extracting executable.

LumenRT doesn’t just bring across geometric data. The software can read in levels, layers, lights, animation paths and metadata.

Once inside VR, ‘BIM hotspots’ will allow users to view the underlying BIM data simply by hovering over an object and clicking. According to Bentley, as everything within the model is ‘instanced’, the memory footprint has been reduced dramatically, helping keep 3D performance levels high, even with large models.

We first saw LumenRT last year, and were hugely impressed by its quality, performance and ease of use. Now, with new VR capabilities, the software looks set to become an even more compelling tool for AEC professionals. The big challenge for Bentley will be letting designers, engineers and architects know that it’s not just for Bentley customers.

Bentley: the future

Bentley may still be preparing for an IPO, but there has been some important news on its relationship with Siemens that could have big ramifications in the future

November 2016 was the second Year in Infrastructure event at which Bentley has been under its IPO rules, which means it can’t reveal its financial earnings - something which has been a bit of a tradition, despite being a private company and, as such, not required to do. The company is biding its time as to when the market conditions best suit a Bentley Systems IPO. It’s still one of the largest privately held software firms in the USA.

While the election of Trump may be a shock to many of us, one of the things he has said he is committed to is a deal to allow US firms to repatriate income earned outside of the USA without punitive taxation (USA corporate and state taxes are over 40%).

Many American Corporations keep money outside of the US — so many that it’s estimated that there is over $2 Trillion of offshore profits. One wonders how this will impact US tech firms and the whole investment and shares market should this money make it to America. Combine that with a campaign strategy to rebuild America’s Infrastructure, with Bentley customers including the majority of US State Department of Transports, this could very well change market conditions in its favour

Siemens Invests For a few years now it’s been rumoured that Bentley and Siemens could be more than strategic business partners. The companies work closely together in the Factory Design market combining manufacturing, architectural and structural technology with factory components and layout tools. Both firms are ‘corporate’ in style and have superb engineering pedigrees.

In the week following YII 2016, an announcement was made indicating that Siemens and Bentley Systems are in fact getting closer together with a formalised, strategic alliance agreement to advance infrastructure project delivery and asset performance in complementary business areas. Siemens and Bentley will initially invest at least €50 million in developing joint solutions to enlarge their respective offerings for infrastructure and industry

by martyn Day

to the benefit of the end-customers.

This work will uniquely leverage new cloud services for a connected data environment to converge respective digital engineering models from both companies. In addition to those elements of the agreement, approximately €70 million of secondary shares of Bentley’s common stock has been acquired by Siemens, under a company program that will continue until such time as Bentley Systems’ stock is publicly traded.

This ongoing acquisition of Bentley stock is extremely significant. The last time a software / engineering firm owned so much Bentley stock was back in the days when Intergraph owned a significant number of shares. With Siemens having also just acquired Mentor Graphics (electronics CAD software firm), it’s clear that Siemens has an appetite for AEC, and is astutely watchful of what its main rival Dassault Systèmes is doing in the space.

Siemens and Bentley Systems have a track record of complementing their respective portfolios through the licensing of each other’s technology. For example, Bentley’s reality modelling software has been integrated into Siemens Process Simulate to leverage laser-scanned point clouds in modelling the existing context of brownfield industrial environments.

The automotive industry manufacturer Turnkey Manufacturing Systems (TMS) successfully employed the innovative point cloud capabilities to create a “digital twin” of its production line to significantly enhance its planning and validation processes, while saving time and costs.

The new investment initiatives will involve virtually all Siemens divisions. The major benefit will be accumulating intelligence from Siemens solutions throughout Bentley’s complementary applications for design modelling, analytical modelling, construction modelling and asset performance modelling. As a result, the integrated and accessible digital engineering models, such as the “digital twin” viewed through an immersive 3D interface, will, according to Bentley, enable unprecedented operational performance, visibility and reliability. This work will converge digital engineering models: physical engineering models in their 3D physical reality context by way of Bentley’s software solutions and the corresponding functional engineering 2D models within Siemens’ solutions.

Siemens and Bentley Systems have identified opportunities to work together in Energy Management, Power Generation, Building Technology and Mobility where each company can leverage their respective technology and industry expertise to bring new business value to the market. For example, Bentley’s applications for the 3D modelling and structural analysis of industrial and infrastructure assets, complement Siemens’ solutions and domain expertise in electrification and automation. Siemens and Bentley Systems will each provide software from the other to deliver complete solutions from either company to the benefit of their respective customers in order to improve their project and asset performance through simulation and virtual commissioning. Development work will benefit from and extend Siemens’ and Bentley Systems’ established commitments to openness and interoperability.

Bentley Systems CEO Greg Bentley explained, “Only with Siemens could we so purposefully advance beyond merely linking the ‘Industrial Internet of Things’, to ultimately leverage digital engineering models for visual operations and connected infrastructure asset performance. Given our long history of sharing complementary technologies, we are very excited to now contribute so broadly to Siemens’ industrial digitalisation leadership.”

Conclusion Looking at route to market, style of customer, depth of product and a total focus on team and document management Bentley and Siemens are very well matched in this alliance. Siemens for the first time is actively looking outside of factory design and can see that Bentley’s technology and user base would also benefit from existing Siemens technologies and wide array of automation products. It will be interesting to see the fruits of the co-development and to see what the net result of these initial moves will be.