Human Assist: Decision automation for high value fruit industries

Decision automation for high value fruit industries Human Assist: Wearable devices and Artificial Intelligence systems for orchardists and vintners to maximise fruit value Human Assist is a national research collaboration of experts and industry representatives working to overcome challenges around worker shortages and variability in the high-value fruit industries.

Human Assist will help optimise labour-intensive tasks and create a tech‑augmented expert workforce in the orchard and vineyard through: • A suite of new AI and augmented reality tools to help workers make better and more consistent decisions • Turning workers into experts through skills-transfer and automation • Reducing labour shortages as workers are upskilled into high‑quality jobs • Delivering a step-change in profitability for producers and exporters • Fully automating some orchard activities.

Human Assist’s new technologies will be exportable as it directly addresses the global need for skilled and accurate manual labour in high-value orchards and vineyards. It has the potential to be applied to other industries and crafts that are also relying on skilled and accurate labour.

3

AUGMENTED REALITY

1. Time of flight camera 2. RGB-D camera (depth sensor) 4 3. Augmented realty display 4. LIDAR (pulsed lazer light to image) 5. Power pack 5 6. Embedded computer 7. Electric lopper

6

What is the need?

The orchard/vineyard industries say they need: • Rapid and accurate decision-making by workers in the field • Improved human operator consistency • Skills-transfer to address the shortage of experienced workers • Automation technology to ease the upward cost pressures • Greater predictability of the quality and yield higher value fruit.

How will it work?

We’ll combine four key elements: • Data: Rich database on the crop, tree and workers over space and time. AI tools to optimise orchard/vineyard decisions and actions • Sensor Fusion: Suite of machine vision, optical, acoustic and microwave sensors capturing information on the crop and operators • Human Assist Tools: Data and knowledge gained from AI models to develop tools such as augmeted reality, virtual reality and dexterious tools to guide decision making and guide the actions of worker • Communities: How the technology affects workers and communities will be investigated. A focus will be placed on Mātauranga Māori, cultural heritage, knowledge and IP in the co-design and adoption of new technologies.

2 SENSORS

SENSORS

EMBEDDED COMPUTER

1 MECHATRONIC DEVICE ACTUATION

7

A.I. LEARNING AND DECISION MAKING

AUGMENTED DEVICE SHOWS LOCATION

MECHATRONIC DEVICE ACTUATION

ROBOT ARM MOVES TO LOCATION

These elements then become the Human Assist Platform

Looking ahead

The project focused on 3 case studies:

We will fill a significant gap in global fruit industries by creating a high-quality tech augmented workforce (e.g. pruning, thinning, harvesting).

1. Apple fruitlet thinning 2. Grape pruning 3. Blueberry harvesting The platform technology can then be adapted to other orchard and vineyard tasks. Down the track, we can link this to automated machines.

• New wearable sensors will assess the crop canopy in front of an experienced operator (e.g. branches, buds, fruitlets) and monitor their manual actions (gestures, events, tasks) • New AI methods to learn actions from sensor data

How will growers, exporters and New Zealand benefit?

• We can apply the automation platform to other orchard and vineyard operations

By 2028, this programme is expected to deliver $416 million p.a. in economic benefits to New Zealand in:

• Being highly transferrable and addressing a global need, the technology has significant export potential

• High-value manufacturing exports through human-assist systems and fully automated systems

• The new knowledge New Zealand gains will increase our science capabilities and put us into the R&D frontier of AI‑based utomation for horticulture

• High-value fruit crops/wine export receipts through higher yields and improved quality • Domestic high-value fruit production through reduced labour costs.

• Learnt knowledge will support inexpert staff using interface methods such as AR; reducing operator inconsistency

What new technology will the project deliver?

• We will understand technology adoption in the workforce and how automation affects hoticultural communities

• A new, rich synchronised database of vine/tree structure and operator’s movements and activities including:

• We will advance Human Assist technologies into a control system for automated robots

- assessing plant development across seasons (temporal)

- assessing variations across the orchard/vineyard (spatial).

• We will better understand the constraints around technology adoption and consider how we can support social benefits from the introduction of new AI technologies.

• A new technology platform for:

- automated learning of “best practice” actions from expert operators leading to a digital model of the expert operator

- human-assist system (AR/VR) and wearable hardware to inform, guide and run quality control of manual orchard tasks

- digital action models to guide robots for task automation.

From here, growers will be able to: • Support less experienced workers with augmented reality so they make better and more consistent decisions that will enhance crop quality • inform autonomous robots to ultimately automate the tasks.

The Human Assist project will be led by the following research experts: Dr Jaco Fourie Lincoln Agritech. Jaco has extensive knowledge in heuristic optimisers and machine learning.

Professor Bruce MacDonald University of Auckland, Principal Investigator. Bruce is the director of the Centre for Automation and Robotic Engineering Science (CARES).

He leads the coordination of sensing, robotics and automation research across New Zealand.

Associate Professor Patrice Delmas University of Auckland. Patrice leads the Intelligent Vision Systems Lab and has expertise in the 3D Computer Vision pipeline, from the theoretical development of robust stereo matching algorithms to the application of complete solutions.

Dr Armin Werner Lincoln Agritech, Research Aim lead.

Armin has experience in precision agriculture using science expertise in crop physiology and growth modelling to create solutions for digital horticulture.

Dr Marama Muru-Lanning University of Auckland, Māori engagement leader. Marama is the Research Director of the James Henare Māori Research Centre. Her research is concerned with debates and critical challenges in social anthropology where she focuses on the cultural specificity of iwi-Māori and their unique sense of place and belonging in Aotearoa.

Marama is from Tūrangawaewae Marae and is of Waikato Tainui and Ngāti Maniapoto descent.

Professor Richard Green University of Canterbury, Research Aim lead. Richard is a computer vision and AI expert, with experience in autonomous robot vision for horticulture, forestry and marine applications.

Professor Mike Duke University of Waikato, Research Aim lead. Mike leads the Waikato Robotics Automation and Sensing (WaiRAS) research group, specializing in agricultural robotics.

Dr Peter Schaare Plant and Food Research, Research Aim co-lead. Peter specialises in the development of optical and physical measurement technologies for evaluating composition and quality of fresh produce.

Dr Katharine Legun University of Otago, Lead for community technology adoption. Katharine research considers how plants, measurement systems, and new artificial intelligence technology shape economic agency and industry dynamics in agri-food systems.

Dr Henry Williams University of Auckland, Research Aim lead. Henry is the technical integration lead for this project. He specialised in utilising machine learning to allow robots to learn their own approaches through interactions with the environment.

Level 10, 49 Symonds Street, Auckland. Private Bag 92019 Victoria St West Auckland 1142 New Zealand http://tiny.cc/QS_UOA

2

http://tiny.cc/Times_Rankings

3

http://tiny.cc/THE_UOA_Impact

The University supports economic growth locally and nationally by fostering innovation and entrepreneurship to create quality jobs and high-value businesses. Our graduates make a significant contribution to our economy and society, to the benefit of all.

The University of Auckland is New Zealand’s world-ranked university. It is the leading New Zealand university in the QS World University Rankings 20191 and in the Times Higher Education World University Rankings (THE)2, and is ranked number one in the world in the sustainability-focused inaugural THE Impact Rankings3.

UniServices At UniServices, we bring ideas to life.

We partner with the best minds at the University of Auckland to apply intelligent thinking to ideas that have the potential to change the world.

Together with our partners, we look to the future, imagine the possibilities, and innovate for public and private good. For over 30 years, we’ve collaborated with hundreds of organisations on thousands of projects in New Zealand and around the world.

We are the most innovative university in New Zealand in Reuters’ Top 75: Asia’s Most Innovative Universities rankings.4

CONTACT research@uniservices.co.nz www.uniservices.co.nz | www.auckland.ac.nz

1

The University of Auckland

4

http://tiny.cc/Reuters_Top75