Machine Learning as a Tool

Detecting Boulders in a Multibeam Point Cloud

Machine Learning as a Tool

Machine learning is currently experiencing a surge of interest. Of course, what first come to mind are self-driving cars, face recognition and internet algorithms. However, while these are great examples of machine learning in action, artificial intelligence can also be applied for hydrographic purposes. A Bachelor’s research project was carried out at QPS B.V. into the detection of boulders in multibeam point cloud data using machine learning.

The objective of the research was to develop a working tool that uses machine learning to detect boulders or rocks in multibeam point cloud data. The approach differs from the tools that processors are currently familiar with, such as filter tools that the user can employ to locate boulders. While there is nothing wrong with the current tools, machine learning aims to automate the entire process, mimicking the steps that a human processor would take to detect objects.

REQUIREMENTS OF MACHINE LEARNING

Machine learning requires input data, and lots of it. In fact, the more data, the better. An integral part of any machine learning is the training of the model using real world sample data. Rather than creating fixed algorithms, machine learning adapts the criteria of what is a boulder or rock while learning. The training data needs to be checked and prepared ahead of time by a professional. In this case, a good example of this preparation might be to start with available multibeam point cloud data containing certain objects of interest. These objects would be selected within the data, adding a class to each data point (boulder or seafloor). The more thoroughly the data is prepared, the better the algorithm functions. Due to the nature of machine learning, the aim of this algorithm is to mimic the input data, thus indirectly learning from processors who prepare the input data. This means that the aim is to achieve the level of accuracy that a human processor produces.

When properly prepared, the machine learning algorithm can produce locations of objects almost as accurately as professional processors are able to do. With more input data and classification of that data by different professionals, the algorithm gains more insight into the output that it is expected to achieve. Following this cycle, the algorithm can potentially perform better than a single human processor, as it combines the knowledge of various processors and situations. On the other hand, the algorithm will never be able to cope with a situation that is completely unlike one that it has encountered before.

CREATING THE TOOL

The process of creating the tool was undertaken in a simple Jupyter environment, as the tool was developed in Python 3.8. Creating a machine learning algorithm from scratch is timeconsuming, so for this research a library, or a

Figure 1: Simple prediction interface for the tool. Figure 2: Point cloud data and classification into boulders (red) and seafloor (blue).

Figure 3: Untrained boulder shapes detected incorrectly.

preprogrammed set of usable code, was used. In this case, the library Open3D was used. This allowed the research to follow the basic principles that have been precoded by professional machine learning experts.

The result is a working prototype that is able to import data, run the detection algorithm, and export its predictions on object locations. Training of the algorithm was done using a different set of objects for which boulders needed to be predicted (these pretrained models can be saved for later use). The training process is very computer resource heavy, and this research was executed on a Nvidia GTX 1070, which proved to be a bottleneck. The training process depends heavily on the hardware and the amount of input data, and in this case it took at least 1 hour for 10 million points. With the current setup, the predicted boulders were found while classifying 300,000 data points per 10 minutes. To compare, a modern multibeam produces between 10,000 and 40,000 points per second.

QUALITATIVE RESULTS

The algorithm was trained to produce a list of boulders from the supplied and prepared data. Qualitative results indicated that the algorithm was differentiating between the boulders according to the preprocessed data used to train it.

This conclusion is based on the processing of a small area by several processors, trying to reach the ground truth as much as possible. The actual analysis was therefore based on an assumed ground truth: the accuracy of human processors. It therefore represents the ability of the algorithm to match the accuracy of manually processed data and picked boulder locations. Several issues remained after visual analysis of the results, mainly concerning the shape of the detected boulders. The algorithm did not completely fill in the boulder locations with the classification ‘boulder’ and had issues with boulders that it was not familiar with, as expected. This led to the conclusion that more training and more or better data would be required.

Another conclusion related to the size of the found objects. As part of the algorithm, the software smartly scales the boulders to new situations. As a result, the software started to find tiny boulders that, based on their size, should not be classified as such, but as pebbles. A future adaptation to the algorithm could allow for a traditional bandwidth filter that discards objects below a certain threshold.

QUANTITATIVE RESULTS

To verify the quantitative results of the tool, a comparison was made between the results of the tool and those of ten different data processors. For this, seven MIWB students and three QPS data processors were asked to classify boulders in the same dataset using the QPS Qimera Geopicking tool. This resulted in a series of objects in which differences could be seen between manual processors. No further analysis was done on why different processors picked different sets of boulders, but the commonly picked boulders were used as ground truth for the algorithm validation. The resulting list of manually picked boulders was then compared with the output of the tool to gauge the accuracy of the tool.

A statistical analysis of the comparison yielded an overall reliability value of 55%, which means that the results of the tool had a 55% similarity to the manually processed dataset. This is built up from three metrics: accuracy, precision and recall. Accuracy indicates the percentage of correct predictions, while precision indicates how many of the positive predictions were actually correct. Recall indicates how many of the actual positive values were also predicted as positive. The three parameters each have a weight in the final reliability (Wa, Wr and Wp). As can be seen in the table, accuracy is comparatively higher than precision and recall, the reason being a class imbalance between seafloor (many) and boulder points (few), thus misrepresenting the total reliability.

This means that the algorithm is not yet ready to be trusted. The accuracy consists of both ‘false positives’, boulders detected by the algorithm but not by the human operator, and ‘false negatives’, boulders detected by the human operator but not by the algorithm. The number of false positives, or locations where the algorithm thinks a boulder is present where in reality it is not, and false negatives, where the algorithm missed out an existing boulder, represents the error of the algorithm, at first glance already too great to be trustworthy as of now.

Although the algorithm in its current state is far from correct, it might still be of help to the surveyor as a tool that can roughly detect objects. It also became clear that future standards for input ‘training’ data preparation must be set, which would further increase the reliability of the algorithm. Of interest, but not further studied, is the question why different processors indicate sometimes very different locations as boulders. Using this as further input would allow the algorithm to take these situations into account as well.

MACHINE LEARNING IN HYDROGRAPHY

Although the results of this research are not as reliable as was hoped, this tool in specific, and machine learning in general, can be of great help once made more reliable. Even the rough prototype created during this research could help

Figure 4: Statistical analysis of the results (TP = true positive, FP = false positive, TN = true negative, FN = false negative).

the processor to save time by detecting the obvious boulders and smaller, harder to notice objects. Ideally, results will be obtained at just the press of a button, resulting in less manpower required in the processing department, less time to process and, more importantly, more reliable results. Perhaps even more important is that it could also lead to more consistent results across processing datasets as the algorithm provides the same output every time.

The second use for this kind of algorithm is the live detection of boulders. In the case of automated cars, it is possible to display objects that have been detected live to the autonomous system. This can also be achieved for surveying vessels. As the vessel moves and the surveying systems are running, machine learning algorithms can show dangerous or interesting locations to the crew. This could increase safety and awareness of the surroundings. However, the processing time needs to be reduced, as more data is currently accumulated than can be processed in the same time.

Object detection algorithms already exist, and work by trying to figure out a common property of a certain object. Machine learning could add flexibility, human-like processes and applicability. Machine learning cannot only detect objects, but can also predict locations based on its previous experience, in the same way that a processor finds these locations. It can also use any data property to its advantage, as it can use shape, size and other properties that objects share. This forms a reliable algorithm which can, more than existing filters, present the client with a complete picture. A machine learning algorithm can be trained to detect anything that is deemed important. The training cannot however be done in the field yet, and will therefore require specific adaptations to the software package.

CONCLUSION

Machine learning seems to have a lot of potential, as it aims to further automate the process of object detection and even show it live as feedback. Applying it to survey routines may yield great improvements in data processing, as well as having other advantages. Before getting to that stage, however, more research is required. The currently presented research forms a small step in this direction.

Further reading

- Open3D: http://www.open3d.org - Accuracy, recall and precision: https://www.analyticsvidhya.com/ blog/2020/11/a-tour-of-evaluation-metrics-for-machinelearning/

Pascal van Unen completed his Bachelor’s in Ocean Technology (Cat A) at the Maritime Institute Willem Barentsz. His graduation thesis research was conducted at QPS BV. His interest lies in the software side of hydrography, for which he strives to find new and interesting applications.  pascalvunen@gmail.com

Huibert-Jan Lekkerkerk is a senior lecturer in Cat A Ocean Technology at the Maritime Institute Willem Barentsz as well as a senior lecturer on the Skilltrade Cat B course and a hydrographic consultant on a variety of projects.  huibert-jan.lekkerkerk1@nhlstenden.com

Phoenix 5: A Versatile, Stable Survey ASV

Aquatic Drones, based in the Netherlands, recently introduced the Phoenix 5 Autonomous Surface Vessel (ASV). The unit, with a length of five metres, provides a stable platform for a range of survey and inspection operations. It can be deployed for bathymetric surveys, sub-bottom profiling, inspection of civil engineering structures, safety and security surveillance, water quality monitoring and other applications. Phoenix 5 is designed for operations on inland waterways, in ports and nearshore.

Maarten Ruyssenaers, founder and CEO of Aquatic Drones, explains the history of the Phoenix: “I worked on marine robotics at the RDM Campus in the Port of Rotterdam, where industry, SMEs, research institutes and government bodies cooperate on innovation projects. I could see great opportunities for autonomous vessels so I set the company up in 2016 to develop these as commercial products. We worked with partners such as the Dutch Ministry of Infrastructure and Water Management, the Port of Rotterdam, Deltares and Boskalis to analyse their requirements and test the pre-production vessels. We learned a lot over a period of five years, surveying ports and large rivers in the Netherlands and supporting dredging operations. And now we are expanding our field of operations towards the sea.”

PHOENIX 5 ASV

Although earlier models were catamarans, Aquatic Drones has now adopted an aluminium monohull design. This proved to provide the highest stability at survey speeds of 4 to 5 knots and makes it easier to integrate the propulsion and control systems and survey sensors. The length overall is 5m, the beam 2m and the draught 0.55m, and it can be launched by a single operator. The vessel is powered by lithium-ion batteries and the propulsion system has two ducted propellers driven by electric motors with a combined power of 8 or 20kW, depending on the model. Up to 2kW, at several DC and AC voltages, is available for payloads. Solar panels on deck recharge the batteries during deployment.

PHOENIX 5 MODELS:

- E: standard model with an endurance of up to 20 hours - EP: higher capacity battery model for operation at higher speeds - HP: diesel generator hybrid model for surveys lasting several days

AUTONOMY AND SITUATIONAL AWARENESS

Autonomous operation is based on a sophisticated situational awareness system which receives input from radar, Lidar, forward looking sonar, video cameras and an AIS receiver. Positioning relies on GNSS-RTK, providing an accuracy of approximately two centimetres, and an inertial measurement unit. The communications links to the operator station include Marine Broadband Radio, 4G and WiFi, all securely encrypted. The remote operator, onshore or on a nearby vessel, has displays for hydrographic data and for vessel status and control. Aquatic Drones also provide a cloud solution to give personnel at the operator’s base immediate access to the data gathered, anywhere in the world.

SAFETY AND RELIABILITY

The vessel has an automatic fire suppression system, bilge pumps and back-up power for essential systems. Redundancy of essential control, propulsion and steering systems ensures safety in the case of component failure. The remote operator can always monitor the health of the systems, intervene and even control the vessel manually. If the communications links are interrupted, the Phoenix will loiter at its current position or return to a predefined home point.

The Phoenix 5 is an Autonomous Survey Vessel for surveying ports, large rivers and nearshore.

Control Centre showing live camera feed, system health monitoring, water map with line planning, screen alerts and more. Survey screen using Teledyne’s PDS – data acquisition software.

SURVEY OPERATIONS

One of the key applications of the Phoenix is undertaking bathymetric surveys using a multibeam echosounder. The high stability of the vessel and optimized sensor position minimize the impact of waves and the wake of other vessels on the survey operation. Ruyssenaers notes: “During a survey of the IJssel river in the Netherlands, we were still able to obtain valid measurements at gale force 8, which was much higher than we had expected.”

The operator defines the survey area, after which the Phoenix control system determines the optimum survey grid and route to the area. During operations, the grid is automatically refined, depending on the water depth and bottom profile. If data acquisition is disturbed by external influences, the vessel automatically goes back and rescans the area.

The ASV can be fitted with a range of sensors, tailored to the application. Examples include single or multibeam sounders, side-scan sonar, turbidity and nutrient sensors, and so on. The onboard winch is used to lower sensors, for example to obtain a vertical temperature and conductivity profile for sounder and sonar calibration.

In addition to bathymetric and environmental surveys, Phoenix ASVs can be used for surveillance operations when fitted with low light and thermal imaging cameras. Lidar can scan above-water structures such as quay walls, to supplement side-scan sonar data.

CASE STUDY: RIVER IJSSEL

(BY JEROEN VAN REENEN, BSC HYDROGRAPHY, HYDROGRAPHX) The advantages of ASV surveys were demonstrated when the riverbed morphology of the River IJssel in the Netherlands was repeatedly measured following the construction of temporary breakwaters with a new design. The stable platform provided by the Phoenix ensured excellent repeatability of the measurements. Furthermore, the low noise and vibration of the platform with electric rather than diesel propulsion greatly improved the performance of the multibeam echosounder. The raw data required little post-processing, only the correction of a few outliers associated with steep slopes. This was automated and a clean dataset was available within an hour of the completion of each survey. Weather conditions varied significantly between surveys; however, the stability of the Phoenix meant that this did not affect the results, see graph below.

PARTNERS AND MARKETS

Ruyssenaers concludes: “Undertaking surveys and other operations was essential when developing the product, and we will continue to offer those services in the Netherlands and surrounding countries. We will serve other international markets by supplying Phoenix ASVs to researchers, survey companies, dredging companies, marine contractors and ship leasing operators. Public authorities such as port operators, river authorities and the police and coastguard are also a key market. Aquatic Drones has a network of agents and dealers in Europe, Asia, North Africa, the Middle East and South America who can support their customers with maintenance and advice. We are currently preparing several international projects.

Our company focuses on listening to customers, building up good relationships and optimizing our products for their applications – we work closely together with them. We are currently also supporting the Netherlands Ministry of Infrastructure and Water Management, which is developing regulations for uncrewed vessels.

Customers are concerned about their environmental impact. A Phoenix has a fully electric propulsion system and operates with zero emissions at the point of use when running on batteries. That avoids the CO2 emissions of a larger, crewed survey vessel with diesel engines. In the future, we would also like to replace the diesel generator on our HP model with a hydrogen fuel cell.”

For more information, please go to www.aquaticdrones.eu

Periodic surveys of the IJssel riverbed morphology.

Teledyne Optech and Teledyne CARIS Merge to Form Teledyne Geospatial

Welcome to the New Geospatial

Teledyne Optech and Teledyne CARIS have united to form Teledyne Geospatial in response to the evolving marketplace needs. Under the Teledyne Imaging group, the harmony between the two business units is clear and the two teams were brought together to offer added value to customers through solutions across hardware, software and workflows.

Andy Hoggarth, vice president, Sales and Marketing at Teledyne Geospatial, has been with Teledyne CARIS for almost 20 years. Beginning his geospatial career as a multibeam sonar processor using CARIS software at sea, he transitioned from working on the technical side of the business and is considered an expert in ocean mapping. Hoggarth’s role at the helm of sales and marketing is to understand trends and customer needs as they evolve and ensure that these are translated from idea to product. Teledyne CARIS has carved out a solid niche within the marine sector only to become broader via this new collaboration and, for Hoggarth, the synergies were clear: fusing the two companies reflects the natural evolution of the geospatial industry.

WHAT OPPORTUNITIES DOES THIS NEW COMBINED BUSINESS PROVIDE FOR YOUR CUSTOMERS?

“What’s new about this model is our ability to streamline and centralize our service provision, offering more to both new and existing customers across the spectrum of needs. Having a single contact point across all requirements means efficiency gains in terms of workflow, saving time and ensuring that our customers get what they need earlier.

“The needs of professionals are evolving to no longer focus solely on land or sea, and both geospatial and maritime enterprises are increasingly expected to offer a more comprehensive suite of services. Bringing Optech and CARIS together enables leveraging the world-leading expertise of both companies, ensuring that customers can take full advantage of their ability to provide targeted solutions for land, air and sea. We see new opportunities to connect existing hydrographic customers to the broader geospatial sector. We can offer solutions to a new spectrum of clients on the topographic mapping side through the latest geospatial workflow software, including tools like AI.”

Andy explains that he is most excited about the new possibilities that the merger provides for Teledyne customers to gain a competitive advantage in terms of what they can deliver. For example, processes will be further streamlined: customers can log their high-density, high-resolution Lidar

Teledyne Geospatial provides solutions and workflows for seamlessly mapping land and sea. (Image courtesy: Dave Somers, Ocean Surveys)

data, store it in the CARIS cloud for immediate processing and analysis, and then move into downstream product creation workflow. This gives a clear operational advantage. This, in turn, enables customers to offer added value to their own clients in terms of the breadth of services they’re able to offer.

WHAT DOES THE MERGER MEAN FOR THE CARIS AND OPTECH TEAMS?

“Joining forces means that the teams are gaining new skills across disciplines, and ensuring retention of the field-specific expertise they’re known for. The market is familiar with the strong customer service and support model, from engineering to sales, and nothing will change there. Teledyne’s arsenal includes hydrographers, physicists, surveyors and GIS professionals, as well as AI specialists, engineers and computer scientists. Going forward, the skills and expertise needed most in the field are at the intersection of traditional domain knowledge and the diverse aspects of computer science, as the market transitions to digital. I would suggest that there is an evolution underway in industry, and enterprises that can recruit highly skilled teams with expertise across the geospatial spectrum will be the ones to engage with. Our teams are industry professionals who understand both hardware and software and we see this as an opportunity to develop holistic solutions for hydrographic and topographic mapping.”

HOW DOES TELEDYNE GEOSPATIAL DELIVER A COMPETITIVE ADVANTAGE?

“Geospatial will have access to full range sonar, Lidar and software solutions so customers can get full value from their Teledyne investment with the most advanced sensors in both ocean and land environments. We are unique: while some providers have topographic sensors with integrated software, even bathymetric Lidar, our software workflow goes much further; others are not involved in the final delivery of data, whereas we have end-to-end capability. This is a huge advantage for our customers, who will be able to integrate workflows, create efficiencies and reduce silos. We will reduce the need for separate infrastructure for land and sea since we can support our customers on both. This is strategic for our customers at the business level, but also makes life easier in certain environments, for example in challenging and extreme settings such as the Arctic. This industry acumen means that Teledyne Geospatial is the ideal partner for a full range of projects.”

WHAT HAVE BEEN SOME OF THE KEY CHALLENGES FACED BY THE MARINE SECTOR IN THE PAST YEAR?

“No industry has survived the COVID-19 pandemic unscathed, including the marine sector. Operational challenges have been immense, with businesses forced to pivot their operations from the field or officebased work to dealing with the need to complete projects while necessarily working from home. Limitations like lower-power computers in home offices combined with the inability to travel to worksites to install and maintain hardware systems have only increased this complexity. Despite the roadblocks posed by COVID-19 restrictions, Teledyne has taken the opportunity to explore how to adjust operations and retain the highest level of service provision. A key priority has been to support our customers to transition their software to the cloud, enabling data processing and cartographic production. Digital and virtual interaction is a key part of our strategy, and we are now conducting much of our teaching and training virtually. On the upside, the pandemic has enabled us to reduce our carbon footprint from travel during this period, which also means less travel burden for our teams. As such, Teledyne is evolving with the industry and has managed to transition through the pandemic period into an even stronger position, having identified ways to make remote operations work successfully.”

An example of Teledyne’s recent innovation in terms of remote and autonomous operations involves hydrographic data scientists based in Canada, who would normally be on location in Brazil to perform a survey. Due to the COVID-19 pandemic, they worked remotely with colleagues in Brazil to operate survey equipment including AUVs through software workflows to command and control. As just one of many examples, Teledyne Geospatial will continue to apply this remote operations

Ease of data collection through to final products is enhanced with efficiency-driving AI algorithms and real-time quality control. (Image courtesy: Mike Stecher)

methodology where possible during the pandemic and its aftermath. In terms of the benefits for customers, some project costs are reduced and the reduced environmental burden is obvious, although companies may need to accelerate their investment in digital to make such remote operations successful.

WHICH TECHNOLOGICAL DEVELOPMENTS SHOULD WE BE ON THE LOOKOUT FOR IN THE COMING YEARS?

“Adaptive capabilities, whereby enterprises can use the same systems for a multitude of purposes and adapt solutions to different survey tasks, will be imperative. With Teledyne Geospatial, customers will be able to use the same setup for ocean mapping and corridor mapping for powerlines, delivering a wider range of services. Lidar for both topographic and bathymetric purposes is a pertinent example of how one powerful tool can be applied across sectors. Going forward, UAVs and USVs will be critical, and our software can process the data from these drones, supporting automation.”

What has worked well in the past in shallow water has since demonstrated success in deeper water due to better capabilities, thanks to advances in Teledyne’s portfolio of solutions. For example, Teledyne recently began a partnership with the UK Hydrographic Office on The Nippon Foundation-GEBCO Seabed 2030 Project. This project aims to map the entirety of the world’s ocean floor by 2030, contributing to supporting maritime trade and protecting oceans, a critical weapon in the fight against climate change. Teledyne’s AI capability is used to process incoming multibeam bathymetry datasets more rapidly and accurately than previously possible, overcoming major bottlenecks created by the backlog of data. Enterprises should similarly consider next-generation tools and technologies, and AI will continue to gain momentum, for its capacity to enable accurate, fast data classification and data extraction.

WHAT CAN WE EXPECT FROM TELEDYNE GEOSPATIAL IN THE YEARS TO COME?

“At the heart of Teledyne Geospatial lies integrated solutions. The offering will include turnkey systems, Lidar, sonar,

Customers benefit from a competitive edge, mapping and delivering data products inside of one workflow. (Image courtesy: Chris Esposito)

integrated workflows and a range of systems and solutions that support full, precise data collection. Geospatial will also work closely with Teledyne Marine, in particular concerning sonar systems, and both sonar and Lidar can be logged in CARIS’s recently launched Onboard360 acquisition software. With Onboard360, a near real-time and autonomous data acquisition and processing solution, it will become possible to create a common workflow using AI and cloud-native technology. For example, customers will be able to log data to the cloud with their hardware, streamlining their product timeline. A streamlined workflow for customers is a central, underpinning goal of Teledyne Geospatial.”

WHAT NEW PRODUCT DEVELOPMENTS CAN WE EXPECT TO SEE IN THE COMING MONTHS?

Andy regards software as a differentiator, enabling customers to get the most out of their hardware: “We will be releasing new software products in the coming months, which will allow users to get the most out of their sensors. An example of this is the recently launched CZMIL SuperNova, which combines Optech’s powerful bathymetric Lidar with CARIS’ comprehensive processing software.” Terratec AS, a leading Norwegian mapping company, has purchased and flown CZMIL SuperNova for airborne survey projects, collecting and processing marine coastal zone base maps and mapping Arctic lakes. Working with the CZMIL SuperNova sensor and AI techniques, they conducted automatic discrimination and noise classification, capitalizing on the system’s advanced capabilities and software workflow integration.

To get a first look at CZMIL SuperNova and to hear more about Teledyne’s other upcoming combined solutions, customers are invited to join the conversation under the new banner Teledyne Geospatial at its booths at key industry events in the coming months. Teledyne Geospatial will be at Ocean Business in October and GEO Business in November, both in the UK. In the meantime, readers should reach out to Andy Hoggarth or Jennifer Parham for anything related to current or future Lidar, software and sonar needs.

Andy Hoggarth is vice president of sales and marketing at Teledyne Geospatial.  andy.hoggarth@teledyne.com