construction

Aligning real and virtual construction

From drone-gathered site data to augmented reality (AR), advanced technologies are bringing new perspectives on real-world environments and works in progress, writes Randall S Newton

The construction industry in time and costs,” says Fujitoshi the gathered imagery; and a cloud intelliJapan has a problem: the Takamura, CTO of Komatsu. gence platform, for turning imagery and shrinking birth rate is creating scans into actionable data. a shortage of skilled construc- Eyes inside the project One example of the data processing is tion workers. So heavy equipment manu- “We are a data company that happens to in creating topographic charts. There can facturer Komatsu has created a Smart use drones to collect data,” says Patrick be no trees, shrubs and so on in the final Construction division, building automat- Stuart, director of products at Skycatch. data. Removing all the flora by hand, ed excavators and dozers. The robotic The company’s combination of drones, using a computer workstation, can take equipment pushes dirt without a human mobile apps and cloud processing turns hours or weeks, depending on the scale of driver, following excavation plans with raw data from the drones into usable the project. The Skycatch cloud platform great accuracy. The automated heavy information for autonomous equipment does it in minutes, scaling to use as many equipment is so fast and precise that or for use by engineers in existing CAD or processors as available to complete the existing methods of ground-based data BIM workflows. task. The software can export to common gathering (surveying, ground-based For example, a utilities engineer can AEC data formats for CAD and BIM. scanning) cannot keep up. monitor how utilities are being installed Running robotic heavy equipment

To tackle this issue, Komatsu Smart in near real time. By tracking progress so and monitoring construction progress Construction worked with San Francisco- closely, the engineer can see if and when are only two of the most obvious uses of based start-up Skycatch to improve data a design change must be made, then take UAVs. One Skycatch client started using collection and processing. Skycatch specialises in industrial applications for Unmanned Aerial Vehicles ‘‘ BIM no longer has to be only the the technology after ground had been broken and foundation work had begun on a campus-sized (UAVs), commonly known as drones. Now, whenever Komatsu Smart design/engineering model; it can be the constantly updated record of project. Management started by using Skycatch technology to gather curConstruction vehicles are on site, construction rent site data, comparing it to the they are guided by one or more drones gathering terrain data. Komatsu refers to the resulting workflow measurements and decide what to do. ’’ project design. On the first day, they found a $400,000 slab of concrete was eight inches misaligned. Work as the world’s first machine-to-machine What took days or weeks with manual stopped; an investigation revealed flaws in automated construction system. data gathering and oversight can be now the original survey that echoed through Skycatch’s drones photograph job sites, done in minutes. the design phase and into construction. If capturing imagery and automatically gen- “We give people in construction some- not found so quickly, a $400,000 concrete erating 3D site data. The data can be quick- thing they never had before – eyes into an mistake could have ballooned, costing sevly laid over site drawings or models to entire construction project,” says Stuart. eral times more than that sum, depending automatically calculate area and volume of Traditionally, construction managers on when traditional construction overearth to be moved. gathered information from a variety of sight methods spotted it.

The results are transmitted as instruc- sources and methods, from discussions “I use the Skycatch data on the site daily tions to Smart Construction machinery for on-site to marked-up plans, to email and if not hourly,” says Victoria Julian, a superfully autonomous work on the site. Think phone exchanges. intendent for DPR Construction. “We can of it as drone-to-phone construction man- “It is like a phone tree game,” says take the Skycatch plan, colour-code it, agement. Stuart. “Usually the data is late and not mark it up, label it and use it for logistics

“Once we have efficiently planned the very accurate.” planning, daily, weekly, monthly – whatevwork amount, [clients] can use the The Skycatch technology stack com- er we need.” Komatsu machinery. Smart Construction prises three layers: the drone hardware “Everybody who sees this is amazed by can successfully plan the efficient system and control technology (Skycatch sells it. Loves it. And is using it. It’s really an from the beginning to the end so clients drones but also works with other incredible tool that I don’t want to have to save significant amount of workload, brands); apps for data review and editing do without on the next big project.”

Komatsu Smart Construction transformed its autonomous earth moving equipment when it added near real time data from drones to its site models. (Source: Komatsu)

Recurring imagery as sequential data Skycatch is one of several vendors working to unite the real and the virtual in construction. The new element is recurring imagery. A ‘single source of truth’ has always been the holy grail, but the construction process has made it elusive. The ability to add recurring highresolution actionable imagery data to BIM models or CAD drawings is a gamechanger. BIM no longer has to be only the design/engineering model; it can be the constantly updated record of construction.

The use of high-res recurring imagery can mitigate risk by providing evidence of deviation from the design, as in the case above. Instant aerial 2D linear and 3D volumetric measurements replace manual estimates or hands-on retrieval, with significant time savings and improved safety. Using imagery for job site flow analysis can identify high traffic areas immediately, allowing managers to create workarounds. Terrestrial 3D laser scanners have been in use on large projects for years, but the addition of drones adds both the new dimension of fly-over viewing and the ability to turn site analysis into a new form of sequential data.

A construction site is constantly changing; research shows the constant state of flux is one key reason construction has not made efficiency gains in recent years compared to other industries. Specifically, the issues of Flying Superintendent Flying Superintendent on-site coordination, planning uses drones to create a real-time construction Turner Construction is workand communication have not monitoring process. ing with researchers from the reaped significant automation (Source: University University of Illinois (UI) to gains, with the obvious excepof Illinois and Turner Construction) create a real-time construction tion of making data accessible monitoring system using drone on mobile devices. The creation of data imagery. Flying Superintendent uses and editing of data has remained a desk- both still images and video to guide qualtop activity. Using drones to gather recur- ity control, safety compliance and logisring imagery means site maps can tics. “The analytics we conduct on these become daily reports. Planning logistics survey-grade 3D visual production modand managing assets is simplified when els offer construction managers a transexact coordinates replace estimates. parent view into what’s happening on Communication with quality control, site each day, empowering them to safety and other departments becomes improve reliability in short-term plans faster and more accurate. and eliminate problems before they hap-

Would you be able to walk the plank in the vertigo-inducing WorldViz demo?

Skycatch claims using drones to gather 3D site data can cut days from the process of surveying a typical construction site (Source: Skycatch)

pen,” says Mani Golparvar-Fard, lead tal funding and is working in close cooper- ed in scans. A360 is a cloud collaboration principal investigator and an associate ation with Autodesk on SiteScan, an aerial tool that works in conjunction with professor of civil engineering at UI. analytics platform designed to integrate ReCap, simplifying access to data for all

The research team is currently testing the drone-and-camera hardware with var- stakeholders. The resulting data can be and refining Flying Superintendent on the ious Autodesk products including ReCap used in all Autodesk AEC and civil engiconstruction of a new professional sports and A360. Earlier this year a SiteScan- neering applications including AutoCAD, complex in Sacramento, California. The enabled 3DR drone created 3D imagery of Revit, Navisworks, Infraworks and others. goal is to create a predictive computer the famed Red Rocks Amphitheatre near vision system for construction site data Denver as part of a renovation project. Bentley Systems analysis that runs with minimal human Autodesk helped scan Red Rocks a few Bentley brings its enterprise approach to interaction and can navigate using con- years ago using terrestrial 3D laser scan- AEC and geospatial engineering with a struction site data. By not relying on GPS ners, but this update significantly line of software products and service for data as its only navigation method, drones improved upon the existing data because what it calls “reality modelling”. can be used for indoor work where the of the aerial dimension. Following a bor- ContextCapture turns photos into detailed GPS satellite signal is weak or non-exist- row-not-build ethic, 3DR is also collabo- 3D mesh models that integrate with ent. The research is being commercialised rating with Sony (for mobile computing) MicroStation and other Bentley design by a university spin-off, Reconstruct Inc. and GoPro (for photography) as it builds and engineering products as well as CAD Flying Superintendent is also currently out the capabilities of SiteScan. or GIS software from other vendors; the being used on a high-rise project in models are precise enough to measure Arizona and by a leading Japanese con- Autodesk from or use for comparing as-built with struction company on multiple projects. Years of research and a variety of acquisi- design. A set of photos up to 100 gigapixtions has given Autodesk a rich treasure els in size can be converted into a single State of the art trove of relevant technologies for aligning detailed model. The software includes The technology for aligning the real world real and virtual construction data. Until georeferencing. Other modules in the of construction data with the virtual world recently most of it was scattered in vari- ContextCapture line can create videos or of design and BIM is very much a work in progress. Start-ups like Skycatch and Reconstruct and ‘‘ Construction and operations are interactive web models. Trimble others are working from the looking closely at AR as a new way to It seems each of the major AEC drone-and-imagery side, while experienced AEC technology deliver assembly, operations and repair vendors has a different phrase to describe using 3D site data in the vendors like Autodesk, Bentley data to the job site construction workflow; for and Trimble are working from the design data side. An overview of the leading vendors and their ous divisions. Last year, Autodesk ’’ Trimble, it is “mixed reality”, to describe the blending of realworld objects with digital content from work reveals a variety of approaches to the launched the Forge Initiative, designed to design in real time. Trimble’s roots are in challenge of aligning the real and the vir- consolidate its various technologies and construction, not design; in recent years it tual in construction. Major players and services related to mixing the use of real- has been building a BIM portfolio from a technologies include: world data (“reality capture” in construction tech point of view, and also Autodesk’s words) with design automa- working closely with other vendors where 3D Robotics tion. Some of the technology is designed interests align. Trimble has interoperabili3D Robotics (3DR) was an early player in for product development and 3D printing, ty agreements with both Autodesk and the commercialisation of drones and but there are also tools for AEC. ReCap Bentley, and is using software from both recently dropped out of the consumer 360 provides automatic stitching and reg- companies in their mixed reality R&D. market to focus on enterprise applications. istration of scan data to CAD/BIM files Trimble sees mixed reality as a form of It has received $99 million in venture capi- and can be used to measure items depict- spatial interaction, in which design teams

and construction teams both review, interact and share using 3D models in the context of the physical environment instead of the virtual design space. “In the context of the building industry, this is the phase in which digital and real content co-exist,” says Aviad Almagor, Trimble’s director of Mixed Reality. “Architectural design collides with reality and construction teams transform digital content into physical objects.”

Trimble is working closely with Microsoft to bring Hololens technology to AEC. Hololens is a wearable, self-contained holographic display computer. Users can interact with 3D models that blend into the view of real objects. The Hololens display can be adjusted on a continuum from ‘real objects only’ to ‘digital model only’ – or anywhere in between. Trimble is working from a deviceagnostic perspective. In addition to Microsoft Hololens, it’s also exploring the use of the Google Tango platform to develop augmented reality applications and Facebook Oculus Rift for virtual reality.

Two on-going pilot projects with very large companies highlight what Trimble is developing.

With engineering giant AECOM, Trimble is developing a system for collaborative review of engineering design from a construction perspective. Engineering data and site data collected by drones are overlaid for viewing inside a Microsoft Hololens display system. Design alternatives can be explored in a collaborative context and team members can immediately see how the changes will interact with the existing site.

In June 2016, Trimble announced a mixed reality pilot program with CSCEC Group 1, one of the largest construction firms in China. Similar to the AECOM pilot, drone data will be blended with design, but for construction operational overview, not engineering review. Given the need for on-site collaboration in the Group 1 pilot, the Google Tango tabletbased augmented reality technology may play the lead role from the display side.

1

Would you be able to walk the plank in the vertigo-inducing WorldViz demo?

2

Unlike the products from Autodesk and Bentley, the Trimble mixed reality technology is still in the proof-of-concept and pilot programme stage.

The Big Data future Virtual reality (VR) and augmented reality (AR) are attracting a lot of attention in AEC at the moment. Project visualisation (whether for sales or architectural design review) is an obvious use for VR. Construction and operations are looking closely at AR as a new way to deliver assembly, operations and repair data to the job site. These are interesting applications, each worthy of their own articles. But there is something unique to realtime site information in the BIM model that transcends either specific display technology.

Virtual reality and augmented reality freeze data at a point in time, yet construction sites change constantly. Superimposing drone-gathered site data

4

onto the BIM model in real time makes new technologies possible; Komatsu’s autonomous earth movers are just a first step. As all construction processes become accessible as data, technologies now used in so-called big data applications such as analytics and artificial intelligence can be applied. It should not take too long for autonomous robotic welders and riveters to assemble highrises and autonomous cement trucks to deliver just-in-time loads where they are needed on the job site without a specific advanced order or schedule. Adding other data sources such as weather and traffic could improve logistics and oversight. Add financial data to the mix and, over time, data analysis could more accurately predict costs for specific design styles and construction methods.

Randall S. Newton is Principal Analyst at Consilia Vektor. He has been writing about AEC since 1987.

1 Trimble and Microsoft are developing a mixed reality platform that combines physical objects and digital models into one holographic view. (Source: Trimble) 2 Bentley ContextCapture turns photographs into 3D mesh models suitable for use inside CAD and BIM software. (Source: Bentley Systems) 3 3D Robotics works closely with Autodesk on equipping drones for use in construction monitoring. (Source: 3D Robotics) 4 Autodesk ReCap and A360 and 3D Robotics SiteScan were used to create a 3D data set of Red Rocks Amphitheater near Denver. (Image courtesy of Autodesk, 3D Robotics and Kimley-Horn)