Tesi dell’Ing. Fabio Castellini presentazione del 20 aprile 2024 al Collegio Ingegneri Venezia

Master’s Degree in Computer Engineering for Robotics and Smart Industry

Master’s Thesis

DEVELOPMENTANDCHARACTERIZATIONOFA 3DPRINTEDHEMISPHERICALSOFTTACTILE SENSORFORAGRICULTURALAPPLICATIONS

Supervisor: Candidate:

Prof.RICCARDOMURADORE FABIOCASTELLINI

Co-supervisor: Student ID:

FRANCESCOVISENTIN,PhD

VR464639

Fabio Castellini

Thesis’motivation

 The agricultural sector is facing the challenge of increasing productivity to meet the growing global food demand but is hindered by a shortage of workers due to labor-intensive tasks and poor working conditions.

 To support the Agriculture 4.0 transition, we explored the field of soft robotics and grippers. By exploiting these technologies, we can rethink the way tasks are performed in agriculture, increasing efficiency and safety while keeping an eye on sustainability, food quality and handling.

Fabio Castellini

Thesis’objectives

 Design and manufacture a soft tactile sensor with a suited patternof fiducial markers to be tracked;

 Design a structure that can hold the sensing device in place and mountit onthe FrankaEmikaPandarobot’send-effector;

 Calibrate the sensing device in order to perform online estimation of bothshear and normal forces;

 Attempt a simple harvesting task exploiting the developed device within a force control loop on the gripper and an externaldepth camera mounted on the roboticarm.

Fabio Castellini

Master’s Degree in Computer Engineering for Robotics and Smart Industry

Designandmanufacturing

Thesofttactilesensor

Left: the justprinteddome; middle: the washed dome; right: the polymerdome after curing.

Fabio Castellini

Master’s Degree in Computer Engineering for Robotics and Smart Industry

Designandmanufacturing

Thesofttactilesensor

Left: the chosenmarkers’pattern; middle: the four analyzedpatterns; right: render of the initialprototype.

Fabio Castellini

Designandmanufacturing

Thecustomgripper

 Modular design: it allows to rethink and print a single part instead of the whole prototype, making the overall process faster and minimizing products’ waste. The final design consists in 4 flat elements, the main through hole box and the back-mount requiring a total printing time of about 5h.

 Mainhardware components (sensor&gripper):

 3Dprinted polymerdome (Elastic 50A Polymer Resin);

 Adafruit LEDNeoPixel Ring 12 RGB

 3Dprinted PLA (Polylactic Acid) case;

 RGB fish-eye camera (1.7mm,5MP, 175°FoV) for RaspberryPi;

 RaspberryPi 3 board;

 Intel RealSenseD435idepthcamera;

 Franka Emika Hand gripper and Franka Emika Panda 7-DoFrobotic arm

Fabio Castellini

Deformable dome

12RGBLEDs

Modularcase

RGBcamera

Master’s Degree in Computer Engineering for Robotics and Smart Industry

3Dshapereconstruction

Left: 3D meshesobtained triangulating markers’ coordinates and exploiting the undeformed CADmodel;

Right: scatter plotof the initial and final estimated 3D coordinates of themarkers.

Master’s Degree in Computer Engineering for Robotics and Smart Industry

Sensorcalibration

Dataacquisition&processing

Fabio Castellini

Engineering for Robotics and Smart Industry

Sensorcalibration

Featureextraction,selectionandmodeltraining

 Extractedfeatures:

 Markers’ trajectories with 8 variants(from 2 to 87 features);

 Raw RGB and Binarized images (only for the Deep-CNN)

 Trainedandtestedmodels:

 Linearelastic force approximation;

 Non-linearlycompensatedelastic force approximation;

 LinearRegressionmodel

 K-NeighborsRegressormodel

 Support VectorRegressionmodel

 NeuralNetworkSequentialmodel

 DeepConvolutionalNeuralNetwork(ResNet50) model

Fabio Castellini

Master’s Degree in Computer Engineering for Robotics and Smart Industry

Models’evaluation

Left: the ATINano17-Eforce/torque sensorused for calibrationandvalidation; Right: our soft tactilesensorinstalledon the FrankaHand.

Fabio Castellini

ATI Nano 17-E

Intel RealSense depth camera

Our force sensor

Models’evaluation

 8 types of extracted features with the option of feature scaling (for a total of 16 variants);

 6 models tested on numerical features; 1 Deep Neural Network tested on 2 types of inputimages;

 Models’ hyperparameters were obtained through a GridSearchprocess;

 KNN has found to be the best performing model according to the Mean Squared Error metric (minimum of 1,99% error when evaluating with thecustomroboticgripper)

Fabio Castellini

Engineering for Robotics and Smart Industry

Models’evaluation

Summary of the bestperformance achievedbyeverymodel evaluated on the testset(MSEvalues referto the Fzcomponent).

Models’evaluation

(1)

Estimatednormal forcesagainst Fz groundtruth duringthe validation(1)andtest(2)phases.

(2)

 Validation: testing the trained models on the unseen portion of the dataset.

 Testing: checking models’ performance when the custom gripper is installed on the Franka Emika robot

 Forces are predicted along all three directions but due to how data was gathered, Fz is the component with the most variance inside the dataset.

 KNN has found to be the best performing model according to the Mean Squared Error metric (minimum of 1,99% error when evaluating with the custom robotic gripper)

Master’s

Computer Engineering for Robotics and Smart Industry

Real-timeforcefeedbackandfruitdetection

Strawberry detection (2D and 3D localization) & classification (ripe or unripe);

Trajectory planning of the manipulator keeping the gripper wide open;

Force feedback activation to allow closed loop gripper’s closure;

Harvesting of the fruit with a suited picking pattern.

Fabio Castellini

Master’s

Degree in Computer Engineering for Robotics and Smart Industry

Real-timeforcefeedbackandfruitdetection

Asoft,sensorizedgripperfordelicateharvestingofsmallfruits(ComputersandElectronicsinAgriculture,2023) Fabio Castellini

Conclusions&futuredevelopments

 Enrich the dataset with more samples, to improve force estimation accuracy and robustness;

 Develop afullyautomatedpipelinetocalibratethesensingdevice;

 Improve fruit detection training the object detection Neural Network model on a widerdataset;

 Implement a more sophisticated trajectory planning of the manipulator, that might include collision avoidance; end-effector’s orientation based on the strawberry’s position; placement of the harvested strawberry in a suitable container;

 Adopt another approach for 3D shape reconstruction (e.g. using a structured lightsetup and a photometric stereoalgorithm).

Fabio Castellini

Thanksforyourattention! Fabio