BioSeer Technical Report by Caelin Sutch

BIOSEER REPORT

JESUIT HIGH SCHOOL

BIOSEER

CARMICHAEL, CA

Michael Equi - CEO Caelin Sutch - Web Development Lead Alden Parker - Web Development Jaiveer Gahunia - AI Development James Monroe - Hardware Development

MATE 2019

JESUIT ROBOTICS

Visit Us: Bioseer.io

JESUIT R

JESUIT ROBOTICS

BIOSEER REPORT

TABLE OF CONTENTS I. INTRODUCTION

Abstract

Introduction and Terms of Reference

II.

DEVELOPMENT RATIONALE

Innovation

Company Organization

Product Architecture

Product Evolution

Microsoft Azure Platform

User Interface

Data Processing and Deep Learning

H. Hardware

Pricing Structure

Future Development

III. FINDINGS A.

Market Research

B. Partnerships

VI. CONCLUSION A.

Challenges and Risks

16 16

B. Recommendations

C. Acknowledgments

D. References

V. APPENDICES

Software Flowcharts

Data Schema

Links to Datasets and Other Resources

BIOSEER REPORT

JESUIT ROBOTICS

I. INTRODUCTION

B. INTRODUCTION AND TERMS OF REFERENCE

A. ABSTRACT

According to the latest World Wildlife Fund (WWF) report, goods and services from coastal and marine environments amount to $2.5 trillion each year. Additionally, “more than two-thirds of the annual value of oceans relies on healthy conditions to maintain its current output” (Recognizing the Value of Marine Ecosystem Services). The multitude of benefits of the ocean ecosystem, such as food production, carbon storage, and tourism make the conservation of marine habitat health not only a moral but also an economic priority (Figure 1).

BioSeer redefines how rivers, lakes, and oceans are monitored. Current methods for aquatic habitat monitoring focus on either indirect measurement through the water’s chemical content or laborintensive studies that require teams of scientists. Due to the continued urbanization of natural habitats, the preservation and maintenance of water bodies have become increasingly necessary in maintaining the economic value of the habitat and overall ecosystem health. Without enough high-quality data, wellinformed decision-making is impossible. BioSeer provides users with data, information, and analytical capabilities to help communities, researchers, governments, and companies play their part in maintaining strong and healthy marine environments. BioSeer brings together cutting-edge technology and an intuitive website to make highfrequency and high-resolution direct marine habitat monitoring accessible to a large, nontechnical audience at a low cost. Cloud-based Artificial Intelligence (AI) and Machine Learning (ML) algorithms are able to process location and time-stamped image data to determine not only habitat health but also the rate of change in system conditions. Data is organized and displayed intuitively to assist users in identifying and monitoring the causes and effects of human activity on the environment. BioSeer’s “mobile buoy” technology takes image data-processing and visualization tools a step further by proving a low-cost platform for monitoring large areas of lakes, rivers, and oceans. The power efficiency of the mobile buoy over traditional Autonomous Underwater Vehicles (AUVs) means that it can run for longer periods of time and collect more data before needing to be recharged. Data collected by the mobile buoys are sent to the database where they are processed and relayed to the website to be displayed to the user in real time. The global database lets users openly view and compare data with each other. Bringing together data from the diverse conditions met across the globe makes understanding unique and complex environments effortless.

Figure 1. Oceans Economic Contribution (Consultancy.uk)

Due to human influence on the environment, such as pollution and overfishing, our oceans’ resources are constantly at risk of being eliminated. By 2050, as the planet’s population swells towards 9 billion, the strain on Earth's natural systems, such as marine ecosystems will continue to grow (Ocean Wealth). To ensure these systems keep pace with our human needs, we must harness science and technology. Currently, collecting long-term data for marine analysis is a time-consuming and costly job. Human researchers come at a cost, and often cannot be deployed frequently enough or fast enough to see, let alone address, complex trends. Existing buoy solutions are expensive and antiquated, limiting deployment options. With this in mind, BioSeer has oriented its mission toward providing the technology and the ecosystem to prepare humanity to take better

JESUIT R

JESUIT ROBOTICS

BIOSEER REPORT

Figure 2. Rovotics

care of marine environments and preserve their economic and human value for centuries to come. BioSeer is developed by the Research and Development (R&D) arm of Rovotics, a team of five members within the 17-person company (Figure 2). BioSeer is a product designed to help researchers and industries gather and analyze data on local marine environments in an automated way. BioSeer is not another static sensor buoy equipped with an array of indirect habitat sensors such as pH, salinity, and oxygen content. BioSeer combines the power of AUVs and buoys to provide an endurance-focused sensor and an expandable API to make introducing custom sensor solutions into the ecosystem a breeze. BioSeer is targeted at communities, researchers, governments, and companies looking for a method to perform comprehensive environmental studies and form long-term plans for maintaining the value of our aquatic environments. Common Terms 1. Autonomous Underwater Vehicle - AUV: Underwater vehicles that are able to navigate without human interference using sensors to detect their surroundings. AUVs are used for a variety of monitoring and research tasks. 2. Artificial Intelligence - AI: An area of computer science that deals with giving machines the ability to seem like they have human intelli-

gence. 3. Machine Learning - ML: Machine learning is an application of artificial intelligence (AI) that provides systems the ability to automatically learn and improve from experience without being explicitly programmed. 4. Wireless Sensor Node - WSN: commonly a buoy or a set of buoys (network) that contain a number of sensors such as pH, salinity, turbidity, and depth.

II. DEVELOPMENT RATIONALE A. INNOVATION Big data analytics in the form of image processing is a new and thriving technology that is making its waves through every market and industry. From autonomous cars to biological cell detection and segmentation, image processing algorithms are the way of the future for augmenting the human ability to deal with our environment. Recognizing this trend, BioSeer has expanded the technologies application into the marine industry by turning raw image data into high-resolution semantic maps that characterize marine habitats. Not only can

BIOSEER REPORT

JESUIT ROBOTICS

those intervals down to less than a month in order to allow for large and detailed time series to be put together that help users develop stronger relational understandings of river conditions. BioSeer sets itself apart from the existing market. Most of today's automated marine monitoring systems focus on water chemistry sensing through Wireless Sensor Nodes (WSNs) and buoys

Figure 5. SEAWATCH Buoy Collecting Data

Figure 3. Image Processed by BioSeer AI

BioSeer distinguish between species, but it can also qualitatively and quantitatively measure plant life, population numbers, and detailed habitat conditions from just a series of images (Figure 3). Part of the inspiration for the implementation of this project came from research that has been performed on the South Fork Holston River in

(Figure 5). Bioseer, on the other hand, relies on direct image data which carries more information than what typical WSNs are able to gather. Additionally, BioSeer reduces time intervals between data collections and provides an easy-to-use website for the visualization and organization of the data. The key technological innovation that allows for all this is the application of today's cutting-edge semantic image segmentation and ML technology that can make sense of the large amounts of unstructured data captured from sources

Figure 6. Semantic Segmentation Technology Figure 4. Researchers in the South Fork Holston Rive, TNr

Tennessee (Figure 4). Costly and time-consuming efforts to classify aquatic health and biodiversity were undertaken at various time intervals varying between three and ten years. BioSeer hopes to shrink

such as cameras (Figure 6). Since data collection with cameras is also inexpensive and easy, this new innovation will widen the market of those who can participate in monitoring lakes, rivers, and oceans. The democratization of this ability will prove a breakthrough in maintaining the

JESUIT R

JESUIT ROBOTICS

BIOSEER REPORT

health of our aquatic environments that contribute to a higher quality of life for many people. Stated simply, more users means more deployed sensors across the globe which, in turn, provides BioSeer with more data to help us better understand complex interactions between humans and aquatic habitats.

B. COMPANY ORGANIZATION The BioSeer Development team is a subset of Rovotics, a 17 person underwater robotics company based in Carmichael, California. There are five key members of the development team. Our CEO, Michael Equi, leads the project by developing and maintaining the long-term vision for BioSeer’s success for which all intermediary decisions are made. He is also the lead AI and ML developer. Jaiveer Gahunia is one of the teams AI developers. Caelin Sutch leads the development of the web interface, with Alden Parker providing front end support. James Monroe is in charge of hardware and mechanical design. Similar to Rovotics, BioSeer team goals and deadlines are outlined in project specs to provide a clear structure for development. Also taken from Rovotics, the BioSeer team uses Trello, a cloud-based project management tool, to facilitate the Plan, Build, Test, and Release (PBTR) process when organizing

C. PRODUCT ARCHITECTURE The BioSeer product consists of three main components: the user interface, the image processing algorithms, and the BioSeer mobile buoy (Figure 8). Together, the three systems make BioSeer convenient, simple, and powerful. The user interface developed by our web development team uses Bing Maps to provide a geographically-accurate dashboard that displays

Figure 8. BioSeer Mobile Buoy

information from the global database. Bing Maps combined with the Angular 7 front-end framework provides the tools to create “zones” for BioSeer buoys to scan and a slider to move through the time series stored in the global database. The experience provided by the user interface brings marine research to the modern era and makes it easy for anyone to use (Figure 9). Other buoy

Figure 7. BioSeer Team’s Trello Board

deliverable and keeping track of overall deadlines (Figure 7). The versatile array of project management skills learned and practiced in Rovotics have carried over and created a culture of success within the BioSeer team.

Figure 9. BioSeer Mobiloe Buoy

BIOSEER REPORT systems, such as those provided by USGS, market to those who are equipped to deal with complex data and are able to put up with a less-than-ideal user interface. BioSeer’s mission to allow more people to perform in-depth marine environment tests has driven BioSeer toward making as user-friendly an interface as possible. To store user data, the interface uses one of two data frames in a NoSQL database schema served using Microsoft Azure Cosmo DB. The user database stores personal information and information about their mobile buoy sensor platforms. The sensor platforms update their location and status through the user database for security. Any valid image data that is collected and processed is sent to the global database where it is automatically analyzed and can be viewed by a large set of users across the globe. The web interface serves as the portal to all this information. The core of BioSeer is in its ability to make sense of image data (Figure 10). Traditional chemistry sensors are a good source of indirect data but are prone to losing calibration over time and do not provide direct insights about the quality of marine habitat. BioSeer AI fixes this by making images the ideal source of data for understanding complex marine environments. Cameras are not only cheap but are capable of providing far more information than what can be provided by a chemical sensor. The AI algorithms that make this possible are deployed through the Microsoft Azure Machine Learning Workspace. More about this system can be found in the Microsoft Azure Platform and Data Processing and Deep Learning sections later in this report.

Figure 11. BioSeer Smart Camera Platform

JESUIT ROBOTICS

The BioSeer mobile buoy is a custom-designed smart camera platform made specifically for the BioSeer product ecosystem (Figure 11). Designed around simplicity, reliability, and mission endurance, the BioSeer mobile buoy is able to monitor large zones autonomously and report back to the user when the data is ready to be analyzed by BioSeer’s image processing algorithms. A key feature of the platform is its adjustable buoyancy system that can efficiently submerge the vehicle so it can always get a good view of marine habitats without compromising endurance. A similar adjustable buoyancy system has been proven in past Rovotics projects, such as Mako in 2018 (Figure 12).

Figure 12. Mako’s Adjustable Buoyancy Tubes

D. PRODUCT EVOLUTION BioSeer began as a simple idea to unify marine research into one large global database in order to provide tools that can predict environment states based off of changing factors, including industrial development and climate change. However, early research and development showed that such a highquality global database does not exist. It was also found that more high-quality and standardized data would need to be collected in order to realize this end goal. Reorienting the project led us to focus on creating a product ecosystem that collects, analyzes, stores, and displays data from sensor nodes that consisted solely of an internet connection, GPS, and

JESUIT R

JESUIT ROBOTICS

BIOSEER REPORT

Algorithm Examples

The images on the left are processed by the BioSeer AI algorithm to produce the output images in the middle. Rocks, algae, plants, and human interference are picked out of the image by the algorithm with a high level of accuracy. After processing the images, another algorithm compresses the processed image into a series of metrics (see metrics column). On the left are the labels that were made by the BioSeer team. The human-produced labels are far from perfect yet the algorithm still learned the correct correlations between various features. The segment of data shown is from the test data set and was not used for training the model underscoring its ability to generalize on unseen data.

SOURCE IMAGES

OUTPUT IMAGES

BIOSEER ALGORITHM

METRICS

OUR LABELS

OUR METRICS

Rock 0.493 Aquatic Plant: 0.089 Algae: 0.247 Non-Aquatic Plant: 0.019 Soil/Clay: 0.003 Human Interference: 0.002

Rock 0.621 Aquatic Plant: 0.102 Algae: 0.019 Non-Aquatic Plant: 0.000 Soil/Clay: 0.000 Human Interference: 0.000

rock: 0.519 aquatic plant: 0.127 algae: 0.330 non-aquatic plant: 0.038 soil/clay: 0.009 human interference: 0.0548

rock: 0.155 aquatic plant': 0.135 algae: 0.265 non-aquatic plant: 0.000 soil/clay: 0.080 human interference: 0.000

rock: 0.488 aquatic plant: 0.097 algae: 0.250 non-aquatic plant: 0.020 soil/clay': 0.002 human interference: 0.001

rock: 0.0 aquatic plant: 0.134 algae: 0.0 non-aquatic plant: 0.0 soil/clay: 0.0 human interference: 0.0

rock: 0.465 aquatic plant: 0.145 algae: 0.256 non-aquatic plant: 0.041 soil/clay: 0.010 human interference': 0.017

rock: 0.449 aquatic plant: 0.137 algae: 0.195 non-aquatic plant: 0.028 soil/clay: 0.006 human interference: 0.008

rock: 0.411 aquatic plant: 0.119 algae: 0.208 non-aquatic plant: 0.038 soil/clay: 0.012 human interference: 0.020

rock: 0.506 aquatic plant: 0.103 algae: 0.289 non-aquatic plant: 0.024 soil/clay: 0.004 human interference: 0.004

rock: 0.456 aquatic plant: 0.105 algae: 0.178 non-aquatic plant: 0.018 soil/clay: 0.003 human interference: 0.004

rock: 0.735 aquatic plant: 0.080 algae: 0.743 non-aquatic plant: 0.039 soil/clay: 0.014 human interference: 0.052

Figure 10. Algorithm Examples

rock: 0.0 aquatic plant: 0.301 algae: 0.0 non-aquatic plant: 0.0 soil/clay: 0.0 human interference: 0.0

rock: 0.0 aquatic plant: 0.089 algae: 0.0 non-aquatic plant: 0.0 soil/clay: 0.007 human interference: 0.0

rock: 0.0 aquatic plant: 0.062 algae: 0.0 non-aquatic plant: 0.147 soil/clay: 0.0 human interference: 0.0

rock: 0.648 aquatic plant: 0.084 algae: 0.0 non-aquatic plant: 0.080 soil/clay: 0.0 human interference: 0.0

rock: 0.793 aquatic plant: 0.186 algae: 0.263 non-aquatic plant: 0.0 soil/clay: 0.0 human interference: 0.0

rock: 0.829 aquatic plant: 0.0 algae: 0.598 non-aquatic plant: 0.0 soil/clay: 0.0 human interference: 0.162

rock: 0.882 aquatic plant: 0.0 algae: 0.853 non-aquatic plant: 0.0 soil/clay: 0.0 human interference: 0.093

BIOSEER REPORT camera. However, it was quickly decided that the new method of data collection that would open the door for direct biodiversity monitoring and high frequency, high-resolution scans of ocean, river, and lake marine habitats required a new form of sensor platform. This is what led BioSeer to develop a mobile buoy with the intent to combine the mission endurance of a buoy or stationary WSN with the mobility and search range of an AUV thanks to thrusters, adjustable buoyancy, and a set of advanced navigation sensors and algorithms. The mobile buoy system provides the main data source for which our software, and hence our goals can be realized. In order to make sense of the image data the BioSeer team immediately turned to machine learning and specifically neural networks. The ability for neural networks to find correlation in high-dimensional unstructured data, such as that provided by cameras, made them ideal for providing BioSeer with the ability to make use of the images collected by the mobile buoy sensor platform. Since then, proof-of-concept neural networks have been developed and tested with our most mature algorithm being deployed in order to be interfaced with the BioSeer.io website. The intuitive and well-designed website and NoSQL flexible database structure have always been in line with the goals of BioSeer. A clean and user-friendly ecosystem that is also highly flexible opens up the door for more individuals, companies, and communities to use our product and therefore contribute to a growing global database. Once a database of standardized, high-quality, and highfrequency data can be collected, BioSeer will finally be able to see into the future and help humanity by predicting the state of our precious marine environments using advanced algorithms.

E. MICROSOFT AZURE PLATFORM Microsoft Azure web products helped the BioSeer team put together this project in a way that is scalable, fast, and easy to use. Cosmo DB and Bing Maps formed a critical foundation for our website and data handling capabilities. Azure Machine Learning Workspaces helped us fit our algorithms to their custom dataset and deploy them with ease.

JESUIT ROBOTICS

In the website, Bing Maps was used to provide a map in which data is displayed. Key Bing Maps features, such as pushpins and infoboxes provided an easy way to place markers and information in GPS coordinates. Cosmo DB is a NoSQL database used for storing information about sensors and users in an easy to access format. On the webpage, Cosmo DB provides an API for the backend NodeJS server that the front end Angular framework interacts with. Azure Machine Learning Workspaces provided BioSeer with the compute and the deployment tools to make the machine learning development pipeline simple (Figure 13). Our custom model based on the U-Net architecture was deployed seamlessly through the Notebook VM and then accessed by our web developers through HTTP requests on the provided

Figure 13. Azure Machine Learning Workspace

score URL. The score script used to handle calls takes in a series of images that form a zone and returns the semantically segmented images along with metrics about the zone including biodiversity, species count, algae level, and how much of each class searched was detected.

USER INTERFACE

One of the major development challenges of this project was organizing the large amounts of data in a way thatâ&#x20AC;&#x2122;s easy to read and use when making decisions. To that end, the website was carefully planned out using common User Interface (UI) and user experience techniques and tools such as wireframing. The website was designed in Adobe

JESUIT R

JESUIT ROBOTICS

BIOSEER REPORT

XD and built using the front-end software framework Angular 7. The backend API for interacting with the Azure Cloud was built in NodeJS. The front end makes use of open source libraries for components and layout to speed up development times. Bootstrap 4, a common layout library, and Angular Material, a Google-Material oriented component library, were used so that the development team did not have to hand make all the components. Bing Maps provides support for the maps in the dashboard of the website. Because of Angular 7, the front end was built to be as modular as possible. Different pages are broken up into separate components. For example, the dashboard is broken up into two components, the Bing Map and Map Overlay components. The Map Overlay component is then broken up into the overview, map settings, and sensor group settings components. This modularity ensures that only data transfer points are exposed, and allows parallel development of different components without introducing breaking changes. Two different user scenarios were anticipated when designing the platform. Either the user can upload their own data (Figure 14), adding the appropriate metadata, or use BioSeerâ&#x20AC;&#x2122;s mobile buoy

Figure 14. Image Upload

platform to collect data. Data can be examined on the Bing Maps map along with helpful overlays such as the locations of various sensors and metrics that make large amounts of data easily digestible. Locations of sensors are divided into zones, with each sensor being responsible for its own zone.

Each zone has its own water quality score, and data is averaged out among the whole zone, with hotspots or major anomalies being noted (Figure 15).

Figure 15. Two Sensor Zones on the map

The platform is expandable, and as more sensors and data are added, trends on a global scale are easier to recognize by both humans and machines. All data is open sourced and each user can see different zones and their water quality reports in relation to their own based on standardized data. This makes collaboration between researchers easier than ever. Due to each data point being timestamped during collection, BioSeer is able to display and analyze trends over time. The time slider also allows the user to easily choose specific timestamps to filter data by. The trends in data can be examined over time in customizable graphs as well. User Authentication is handled via JWT, an open source authentication library that stores auth tokens in cookies. Authentication is used to ensure the privacy of critical sensor data, such as current location. User accounts are also used to organize preferences of zones and customize the user experience. The API for interacting with the Azure Cloud platform was built in NodeJS and makes use of Express, a common routing library. The front end Angular application interacts with the API through HTTP requests. Each API endpoint is protected, ensuring overall application security.

BIOSEER REPORT

JESUIT ROBOTICS

G. DATA PROCESSING AND DEEP LEARNING The BioSeer AI team put to work a modified U-Net architecture for semantically segmenting images of the aquatic environments. U-Net is a powerful yet simple Fully Convolutional Neural Network (FCNN) that does not require as much training data or time as similar Regional Convolutional Neural Networks (RCNNs). Based on the requirements of the project, U-Net provided more than enough accuracy to perform the required tasks (Figure 16). Semantic Segmentation is a step up from other localization and box classification techniques that

Figure 16. U-Net

are also common in the AI computer vision industry. Rather than just recording bounding boxes or a yes or no value for an object in an image, semantic segmentation provides more accurate insight into exactly where each object is and how much of it is present in the image. This pixel-level knowledge is critical for our system to measure biodiversity and habitat health. Similar technology is used in autonomous vehicles, underscoring its maturity and reliability (Figure 17). In order to get our custom U-Net to label the classes of objects that we needed, we increased the number of output channels to match that of our number of classes, six, and then trained the network

Figure 17. Semantic Segmentation in Autonomous Cars

on our own custom dataset. No current dataset exists that would suffice for this network. In order to test the network with more classes and on our custom dataset, before we applied it to application-specific data we ran a series of tests. One such test network was SkyAI, a dog-recognizing algorithm trained on a custom dataset (Figure 18).

Figure 18. Sky AI Picture (Left) and Prediction (Right). Blue is greenery, green is the labradoodle, and red is the husky

With preliminary testing completed, to create the custom dataset, the BioSeer team went to the American River in California to collect images in a similar manner to that of the mobile buoy (Figure 19). Once we collect several thousand images, we uploaded them to Labelbox and distributed the costly work of labeling them on a pixel-by-pixel basis to the entire team (Figure 20). Once prepared, custom python Jupyter Notebook scripts prepped the data and U-Net model for training. Throughout the training process, we tuned the hyperparameters of the model until we were satisfied with the modelâ&#x20AC;&#x2122;s performance.

JESUIT R

JESUIT ROBOTICS

BIOSEER REPORT

Figure 19. Caelin Collecting Data in the River

Figure 21. Training Process Graph

Figure 20. LabelBox Image Labeling

The current version of U-Net that is deployed through the Azure Machine Learning Workspace generalized best with a validation set binary cross-entropy loss of 0.37. The training process clearly indicated that more training data is needed for improvement. The addition of regularizes did not confer any noticeable benefit other than creating a greater averaging effect that limited the efficacy of the FCNN (Figure 21). In order to prove the model's competency with a wide array of data we also trained it on a custom data set collected based on MATE props and the Rovotics ROV, Boxfish. The model was trained the same way and was able to fit the model and provide us information about the quality of aquatic habitats in the mock river and lake environment. Since the BioSeer U-Net AI is not capable of doing precise species classification, Microsoft's Species Classification AI is used to enhance the quality of the information we are able to gain from each image.

Figure 22. Species Classification AI

Metrics such as algae content from the BioSeer AI and species count based on Microsoft AI can be precisely determined through a single call by the web interface to process a zone of data (Figure 22).

H. HARDWARE In order to create a cohesive and accessible product ecosystem, BioSeer designed a rugged adaptive mobile buoy that works in any aquatic environment. The BioSeer mobile buoy consists of three main parts: The buoyancy foam, internal aluminum frame, and the bottom protective shell. The bottom shell contains HDPE plastic,

BIOSEER REPORT

JESUIT ROBOTICS

Figure 24. Active Buoyancy System

Figure 23. Bottom of the Mobile Buoy

reinforced with aluminum, and a polycarbonate curved bottom to provide stability and shield the cameras and internal structure from the environment. (Figure 23). The internal aluminum frame houses the two Active Adaptive Buoyancy Systems (AABS). The AABS was inspired by the success of Rovoticsâ&#x20AC;&#x2122; Manual Adaptive Buoyancy System (MABS), which was introduced and won the innovation award at the 2018 MATE competition. Each AABS utilizes a waterproof linear actuator, a polycarbonate tube with a capped ends, and a sliding piston to pressurize and depressurize volumes of air, therefore increasing and decreasing buoyancy. To attain negative and positive buoyancy both in saltwater and freshwater, the buoy can alter its buoyancy by 80 newtons. The large cavities minimize the change in pressure necessary to achieve a substantial change in volume. This active buoyancy also replaces any vertical thrusters, allowing the BioSeer mobile buoy to meet the sweet spot between mobility and autonomous mission endurance (Figure 24). The internal aluminum structure also connects to the two thrusters, the main electronics housing, and the dual-camera rig. The top buoyancy unit contains R-3318 subsea buoyancy foam, with a small aluminum skeleton, and solar panels to extend mission endurance and reliability (Figure 25). The

three sections of the buoy are connected vertically by four large screws. Conservatively, the cost of each buoy is around $3900. (Figure 26)

Figure 25. Solar Panels and Buoyancy

PRICING STRUCTURE

In order to maximize the efficacy of BioSeerâ&#x20AC;&#x2122;s mission, multiple pricing structures will be available for the product. With the majority of the functions running in software, a major source of overhead is in the cost for data storage and compute. To pay for this, BioSeer plans to charge customers $20 a month for the use of online services. Without factoring in discounts incurred through production at scale, a single unit build estimate of the mobile buoy is

JESUIT R

JESUIT ROBOTICS

BIOSEER REPORT

Figure 26. Hardware Cost Breakdown

estimated at $3,900. To maintain a healthy profit while also making it accessible, BioSeer plans to sell the devices starting at either a $4,300 down payment or a $2,000 down with $100 per month lease. In the latter case, the cost of the $4,300 product can be paid off in 23 months. Due to limited time, BioSeer team was unable to get precise market estimates, so the values above are subject to change as we gain further information regarding the number of buyers and the price they are willing to pay for our services.

FUTURE DEVELOPMENT

On the webpage, more time and resources are needed to develop additional features in the user dashboard, such as more comprehensive data viewing (including a map overlay based on semantically segmented image data), the ability to export data in a variety of formats, exposed API endpoints for other applications to use, and more user customization of the dashboard will provide a better user experience. Unfortunately, the web development team did not have the time to fully develop the user account features of the webpage. Although accounts and

authentication were implemented, with more time more features will be added to save user settings and add more customization options for each user. A key requirement for improving the accuracy and scaling up our system is more high-quality data that can be used to train our AI algorithms. With more time and resources BioSeer would be able to invest in getting the needed labeled data to improve our algorithms in providing higher accuracy and the segmentation of more classes of objects. As part of BioSeerâ&#x20AC;&#x2122;s main vision, more data will eventually give BioSeer the ability to forecast aquatic ecosystem health and recovery. As we continue to develop the preliminary version of BioSeer, the long-term goals will drive decision making. Further contact with industry professionals would also help in ensuring the scientific accuracy of the system. Currently, the BioSeer algorithms have yet to be scrutinized by the scientific community, possibly isolating key parts of the market. Industry and scientific professionals would also help us tailor the data and metrics to the most important components of marine ecosystems and conservation.

BIOSEER REPORT

III. FINDINGS A. MARKET RESEARCH BioSeer addresses an industry-wide problem, the lack of comprehensive direct data when monitoring marine environments. There are a few other programs, both governmental and private, that monitor marine environments, but none as comprehensive as BioSeer. A few competing programs are addressed below, and their strengths and weaknesses are noted. Traditionally, oceanographic research vessels are used to monitor marine environments which is a very expensive and time-consuming process that has a low resolution in both time and space. Today, however, a Wireless Sensor Node (WSN) approach (similar to BioSeer) can dramatically improve the access to real-time marine data. Most wireless sensor nodes focus on stationary chemical sensors and long periods of deployment without human intervention. Taken from an analysis of WSNs (Xu): “For marine environment research, a WSN-based approach can dramatically improve access to real-time data covering long periods and large geographical areas [23]. According to Tateson et al. [24], a WSN-based approach is at least one order of magnitude cheaper than a conventional oceanographic research vessel.” BioSeer builds off the idea of WSNs by providing direct data and more comprehensive insight into marine environments. One existing static buoy WSN provider is Smart Environmental Monitoring and Assessment Technologies, or SEMAT. SEMAT buoys are loaded with a sensor payload to monitor ocean

Figure 27. SEMAT Buoy

JESUIT ROBOTICS

environments. At $154 per buoy, they offer a low-cost solution that can be deployed and managed easily on a large scale (Trevathan) (Figure 27). Again, BioSeer expands off of this platform by using visual data to get a better idea of species diversity. Currently, very few solutions exist for largescale marine monitoring. The Copernicus Programme, established by the EU, tracks marine data through the use of satellites. Data such as currents, temperature, and salinity are taken on a large scale and provided free for use. However, solutions such as this fail to monitor marine biodiversity, instead of focusing on data that can be taken from orbit. Satellites can detect trends at a large scale, but lack the resolution to measure detail within smaller marine ecosystems that prove no less important than the larger ones. Without the availability of tools including species identification, Copernicus lacks the depth needed for researchers seeking to combine industrial and climate data with its impact on marine life (About EU Copernicus). Pollution Tracker tracks pollution in the environment through the analysis of harmful chemicals in sediment and mussels. This requires researchers to physically collect samples in the ocean, limiting the scale at which this can be executed. This solution is also localized in British Columbia and is not implemented on a global scale (Monitoring Helps Us Identify Priority Contaminants of Concern). Ocean.org conducts SCUBA surveys for long term research on biodiversity. They’ve developed a program to train divers in species identification and send SCUBA expeditions to collect data. However, this program is expensive, has a high risk for inaccuracies, and is hard to execute routinely and at high frequency. The human component that makes these programs expensive and hard to maintain can be replaced using our image recognition algorithms to track species and marine habitats over long periods of time (Ocean Wise Research). Analysis of current benthic monitoring techniques, such as a Finnish program, shows once again that biodiversity monitoring itself is still notably prohibitive due to its high cost. As pulled from that analysis, “The yearly costs for the Finnish national marine biodiversity monitoring program were around

JESUIT R

JESUIT ROBOTICS

BIOSEER REPORT

€ 5.9 million. The largest costs were generated by the fish monitoring (€ 2.58 million), where the gathering of information for the Common Fisheries Policy accounted for 2.21 million €, as well as by the offshore pelagic and benthic monitoring (2.20 million €), was running the research vessel constituted a major expense” (Henrik). This paper also puts into context the value of monitoring marine habitats, as the benefits can reap hundreds of times more than what it takes to invest in monitoring in socioeconomic gains. A more unique solution for marine monitoring is a project by the Queensland University of Technology to monitor coral reefs. This project, funded by a Microsoft AI for Earth grant, monitors coral reefs through hyperspectral cameras mounted on Unmanned Aerial Vehicles (UAV). The data collected by their UAVs is processed by cloud computers to analyze the health of marine species (Figure 28). This is similar to the Copernicus Program in its aerial technique. By using UAVs opposed to satellites, the researchers have a more detailed and comprehensive view. However, this solution still does not delve into marine species and instead focuses on water-based metrics (Johnston).

proves the capability of ML when applied to marine science. BioSeer seeks to take this further in being able to predict entire states of marine environments.

B. PARTNERSHIPS Marine monitoring requires a multidisciplinary approach. As a high school team, none of us have graduate- or professional-level knowledge of marine biology. To help get more input from industry professionals, the BioSeer team reached out to establish connections with organizations to get advice on the product and more insight into our target market. Working with organizations will help us gauge interest in our product and its probability of success. When performing market research, we developed a list of contacts to inquire about possible applications of BioSeer and the needs of the industry. Below is a list of the groups that we reached out to: - University of Drexel - Eastman Company - Rivernetwork.org - Spokaneriverkeeper.org - Mining.ca - Miningamerica.org - Marinebio.org - Marine.copernicus.eu - Marine-conservation.org - Stanford biology department - Heatherspence.net - Simonjpierce.com - ocean.org

VI. CONCLUSION A. CHALLENGES AND RISKS Figure 28. UAV Equipped with Hyperspectral Cameras

In South Korea, researchers used machine learning to predict algae blooms in water bodies. The researchers harnessed algorithms to analyze large amounts of data to predict when the next algae bloom will happen. This project shows the success of combining data and ML to predict future states and

BioSeer has met with and tackled several challenges and continues to do so moving forward. As of the time of the publication of this report, the key immediate challenges of the BioSeer team have been listed below: A major challenge faced by the project was the quality of the data provided for training the custom U-Net. The time constraints placed by the competition and the team's other activities, including

BIOSEER REPORT school and the regular MATE ROV competition, made getting all the data labeled difficult. Additionally, the quality and bias in the data due to our limited size of the zone used for collection drastically limits the versatility of the existing algorithm. These challenges limited the algorithm to proof-of-concept functionality and demo work. The major innovation of this project is also its major risk. The uncertainty in BioSeer’s future acceptance by the scientific community and the versatility of the algorithm limits its current capabilities and leaves many unknowns in further developing the project. Knowing this risk will help us more strategically handle the growth of the project. With the goal to eventually predict high-dimensional marine environment states, BioSeer’s system of collecting that data carries with it the possibility of bringing down the project. Another risk is in the pricing structure. Due to limited market data and testing, it is unknown whether we would be able to fully realize our goals from the set cost of the system. While much cheaper than other WSNs and with more functionality in monitoring marine habitats, it is not certain whether it would be preferred over the emerging low-cost WSNs that make BioSeer look like a more high-end product.

B. RECOMMENDATIONS Below is a summary list of the key recommendations, future improvements, and current needs of the project in order to maximize the chances of success: 1. Receive further market data in order to tailor the current version of BioSeer in order to gain market dominance and widespread adoption. 2. Gain more marketing information in order to improve the price structure of the product. 3. Begin building an early version of the mobile buoy sensor platform. 4. Receive scientific guidance and scrutiny in order to improve the validity of data collected by the BioSeer AI algorithms. 5. Add more trained web developers and data scientists to the team in order to expedite the development. 6. Begin sensor deployment as soon as possible so that more data can be collected and labeled

JESUIT ROBOTICS

to improve the versatility and accuracy of the AI algorithms. 7. Expand the AI species classifier to more species of aquatic animal and plant life. 8. Continue developing out the web interface for more user customizability. 9. Fully test the web interface with test benches for sensors sending mock data.

C. ACKNOWLEDGMENTS - Microsoft for the free Azure Credits - Microsoft Earth for AI for use of their species classification algorithm - MATE for competition support - The Equi Family for T-Shirts and Website Hosting with a custom domain name - Labelbox for free image labeling tools - U-Net Github repository by zhixuhao for providing the base network used in the BioSeer AI

D. REFERENCES Ocean Wise Research, research.ocean.org/program/ howe-sound-biodiversity#vulnerable-speciesand-habitats. “About.” Mapping Ocean Wealth, oceanwealth.org/ about/. “About Eu Copernicus.” Copernicus, marine. copernicus.eu/about-us/about-eu-copernicus/. Henrik, et al., “Price vs. Value of Marine Monitoring.” Frontiers, Frontiers, 3 Oct. 2016, www.frontiersin. org/articles/10.3389/fmars.2016.00205/full. Johnston, Matt. “How Microsoft's AI Grant Will Help QUT Monitor Coral Reef Health.” ITnews, 23 Apr. 2019, www.itnews.com.au/news/how-microsoftsai-grant-will-help-qut-monitor-coral-reefhealth-524197. “Monitoring Helps Us Identify Priority Contaminants of Concern.” Pollution Tracker, pollutiontracker. org/methods/. “Recognising the Value of Marine Ecosystem Services.” WWF, wwf.panda.org/our_work/ oceans/solutions/recognising_the_value_of_ marine_ecosystem_services/. “Rivers and Streams Across the United States - Water Quality Summaries.” NAWQA Annual Reports, 15 Apr. 2015, cida.usgs.gov/quality/rivers/sites.

JESUIT R

JESUIT ROBOTICS

BIOSEER REPORT

Trevathan, Jarrod, and Ron Johnstone. “Smart Environmental Monitoring and Assessment Technologies (SEMAT)-A New Paradigm for LowCost, Remote Aquatic Environmental Monitoring.” Sensors (Basel, Switzerland), MDPI, 12 July 2018, www.ncbi.nlm.nih.gov/pmc/articles/ PMC6068521/. Xu, Guobao, et al. “Applications of Wireless Sensor Networks in Marine Environment Monitoring: a Survey.” Sensors (Basel, Switzerland), MDPI, 11 Sept. 2014, www.ncbi.nlm.nih.gov/pmc/articles/ PMC4208207/.

V. APPENDICES A. SOFTWARE FLOWCHARTS

See Figure 29 and 30.

B. DATA SCHEMA BioSeer uses a database schema that is broken up into three parts for organizing data in the NoSQL CosmoDB database. The user document holds personal data about the user, such as email or password, as well as data about what sensors they own and what zones they’re interested in. The sensor document contains location data, zone data, and image data. The global document contains zone data and information about what sensors are in each zone, the name of the zone, and the coordinates of the sensors. The schema is organized for efficiency and ease of development. See Figure 31 for the Schema.

C. LINKS TO DATASETS AND OTHER RESOURCES Labelbox link to the dataset: https://bit.ly/2Rld17K Microsoft AI for Earth Species Classifier Github: https://github.com/Microsoft/SpeciesClassification U-Net Github: https://github.com/zhixuhao/unet

U-Net Research Paper: https://arxiv.org/abs/1505.04597 Azure: https://azure.microsoft.com/en-us/ Github link to the BioSeer web interface: https://github.com/JHSRobo/Bioseer-Web-Interface Github link to BioSeer AI: https://github.com/JHSRobo/bioseer Github Link to SkyAI: https://github.com/Michael-Equi/SkyAI BioSeer main website: https://bioseer.io/

BIOSEER REPORT

JESUIT ROBOTICS

Figure 29. Overall Software Flowchart

JESUIT R

JESUIT ROBOTICS

BIOSEER REPORT

Figure 30. Algorithm Flowchart

BIOSEER REPORT

JESUIT ROBOTICS

Figure 31. Data Schema

JESUIT R