Tracking and measuring of catch events in stereo video for longline fisheries

Tracking and Measurement of Catch Events in Stereo Video for Longline Fisheries Tsung-Wei Huang, Jenq-Neng Hwang

Department of Electrical Engineering, University of Washington, USA Sheng-Ting Shen Department of Electrical Engineering, National Cheng Kung University, Taiwan

Suzanne Romain, Farron Wallace Alaska Fisheries Science Center, NOAA Fisheries, USA Session: Emerging Technologies in Fisheries-Dependent Science and Catch Monitoring American Fisheries Society 147th Annual Meeting, Tampa, FL, August 21, 2017

Outline • Motivation • System Overview • Tracking Method • Segmentation and Measurement • Experiment Results • Summary

2

Motivation – Track and Measure

3

Motivation • Goal: • Track and measure the fish captured on rail

• Challenges: • Deformation of fish • Environmental noise

• Proposed method: • Tracking by detection using deep convolutional neural network (D-CNN) with Kalman filter rescoring • Segmentation in disparity image with background plane classification 4

System Overview Tracker Input frame

L

Object detection

Kalman filter rescoring

RGB + disparity

Object segmentation

Update tracks

R detection

disparity

results 5

Deep Convolutional Neural Network (D-CNN) Detector • Single Shot MultiBox Detector (SSD) [1] • Boxes with high detection scores are preserved as object proposals -> multiple object proposals

[1] Wei Liu, et al., “SSD: Single Shot MultiBox Detector,” European Conference on Computer Vision (ECCV), 2016.

6

Tracking Method • • • •

How to select the best object proposals from detector?

Rescoring based on tracking -> Kalman filter Compare: position, size and orientation Final score: đ?‘ đ?‘?đ?‘œđ?‘&#x;đ?‘’ = đ?‘ đ?‘?đ?‘œđ?‘&#x;đ?‘’đ?‘‘đ?‘’đ?‘Ąđ?‘’đ?‘?đ?‘Ąđ?‘–đ?‘œđ?‘› + đ?œ†đ?‘Ąđ?‘&#x;đ?‘Žđ?‘?đ?‘˜ đ?‘ đ?‘?đ?‘œđ?‘&#x;đ?‘’đ?‘Ąđ?‘&#x;đ?‘Žđ?‘?đ?‘˜ C

score

B

high A Ground truth low

đ?‘Ąâˆ’1

Kalman filter prediction

đ?‘Ąâˆ’2

7

Results – tracking comparison (1/3)

With rescoring

Without rescoring (detection only)

(white bounding boxes are ground truth per fish)

8

Results – tracking comparison (2/3)

With rescoring

Without rescoring (detection only)

(white bounding boxes are ground truth per fish)

9

Results – tracking comparison (3/3)

With rescoring

Without rescoring (detection only)

(white bounding boxes are ground truth per fish)

10

Segmentation and Measurement • Length measurement requires reliable segmentation • Initial segmentation • Background subtraction in RGB image -> contains noise

• Segmentation refinement • Disparity image -> 3D information helps refining segmentation

Input

Segmentation in RGB

Disparity

Bg. plane score

11

Disparity and Depth • Disparity of correspondence in stereo image pair • Depth âˆ? (disparity)-1 đ?‘? = đ?‘‹, đ?‘Œ, đ?‘?

Object

đ?‘Ľ0

đ?‘Ľ1

Image plane

L

1 1 đ?‘?âˆ? = đ?‘Ľ0 − đ?‘Ľ1 đ?‘‘

R

far Camera center

boat đ??ś0

đ??ś1

sea disparity

near 12

Disparity Plane • Define disparity plane đ?œƒ = đ??´, đ??ľ, đ??ś : đ?‘‘đ?œƒ đ?‘Ľ, đ?‘Ś = đ??´đ?‘Ľ + đ??ľđ?‘Ś + đ??ś ďƒł Plane in 3D • The disparity plane fit to a pixel đ?‘Ľ, đ?‘Ś is the tangent plane in 3D

Disparity plane passing đ?‘–1 đ?‘–1

disparity 13

Background plane – sampling-and-scoring • Classify the foreground and background using disparity planes • Sampling: Sample several disparity planes from initial background area • Scoring: Compatibility of each pixel in the image to the sampled planes

Compatibility of pixel đ?&#x2018;Ľđ?&#x2018;&#x2014; , đ?&#x2018;Śđ?&#x2018;&#x2014; to a disparity plane đ?&#x153;&#x192;đ?&#x2018;&#x2013; = đ??´đ?&#x2018;&#x2013; đ??ľđ?&#x2018;&#x2013; , đ??śđ?&#x2018;&#x2013; : đ?&#x2018;&#x2018; đ?&#x2018;Ľđ?&#x2018;&#x2014; , đ?&#x2018;Śđ?&#x2018;&#x2014; â&#x2C6;&#x2019; đ?&#x2018;&#x2018;đ?&#x153;&#x192;đ?&#x2018;&#x2013; đ?&#x2018;Ľ, đ?&#x2018;Ś < Î&#x201D;đ??ˇ

đ?&#x2018;&#x2014;1 đ?&#x2018;&#x2013;1

đ?&#x2018;&#x2013;2

đ?&#x153;&#x2021; đ?&#x2018;&#x2014;, đ?&#x2018;&#x2013; =

đ?&#x2018;&#x2014;2 đ?&#x2018;&#x2013;3

1

đ?&#x2018;&#x17D;đ?&#x2018;&#x203A;đ?&#x2018;&#x2018; đ?&#x2018;&#x17D;đ?&#x2018;&#x203A;đ?&#x2018;&#x2018;

0

đ?&#x153;&#x2022;đ?&#x2018;&#x2018; đ?&#x2018;Ľđ?&#x2018;&#x2014; , đ?&#x2018;Śđ?&#x2018;&#x2014; â&#x2C6;&#x2019; đ??´đ?&#x2018;&#x2013; < Î&#x201D;đ??´ đ?&#x153;&#x2022;đ?&#x2018;Ľ đ?&#x153;&#x2022;đ?&#x2018;&#x2018; đ?&#x2018;Ľđ?&#x2018;&#x2014; , đ?&#x2018;Śđ?&#x2018;&#x2014; â&#x2C6;&#x2019; đ??ľđ?&#x2018;&#x2013; < Î&#x201D;đ??ľ đ?&#x153;&#x2022;đ?&#x2018;Ś đ?&#x2018;&#x153;đ?&#x2018;Ąâ&#x201E;&#x17D;đ?&#x2018;&#x2019;đ?&#x2018;&#x;đ?&#x2018;¤đ?&#x2018;&#x2013;đ?&#x2018; đ?&#x2018;&#x2019; 14

Growing object plane â&#x20AC;˘ Grow the object segmentation by comparing the outside pixels with neighborhood object pixels â&#x20AC;˘ Include the pixels with are compatible in both disparity and color

Compatible in disparity and color

Compatible in color only

đ?&#x2018;&#x2014;1

đ?&#x2018;&#x2014;2

đ?&#x2018;&#x2013;1

đ?&#x2018;&#x2014;3

Incompatible

15

Results â&#x20AC;&#x201C; background plane

Left

Initial Foreground

Disparity

Bg. plane score

16

Length Measurement â&#x20AC;˘ â&#x20AC;˘ â&#x20AC;˘ â&#x20AC;˘

Reproject the segmented fish to 3D space Use PCA to find the orientation (major axis) of fish in 3D Measure the length by connecting the endpoints along major axis Estimate length from multiple frames during tracking:

â&#x20AC;˘ Take the average of those within 90% of longest length because the fish is not always straight â&#x20AC;˘ Do not trust the longest length if it is much larger than the second largest -> possibly outlier đ?&#x2018;Śâ&#x20AC;˛

head

đ?&#x2018;Ľâ&#x20AC;˛

đ?&#x2018;§â&#x20AC;˛ đ??ś tail 17

Results • Clip 1 (early morning), clip 2 (morning) and clip 3 (afternoon with strong sunlight and shadows) • Tracking Performance Method baseline rescoring rescoring + seg.

TP 59 72 76

Clip 1 (79) FN FP 20 6 7 4 3 2

TP 48 57 58

Clip 2 (60) FN FP 12 5 3 5 2 1

TP 47 50 58

Clip 3 (62) FN FP 15 13 12 10 4 2

TP, FN, FP: true positive, false negative, false positive

• Length Measurement (Mean Absolute Error) Method w/o bg. planes

Clip 1 (16) 11.7%

Clip 2 (20) 14.1%

Clip 3 (14) 14.9%

with bg. planes

7.6%

8.0%

10.0% 18

Example Results (1/3)

19

Example Results (2/3)

20

Example Results (3/3)

21

Visualization in different 3D angle

22

Summary • We propose an efficient and effective tracking and segmentation framework for monitoring fish catching on wild sea surfaces • A Kalman filtering-based tracking system is used to rescore the object proposals given by object detector and track the low detection score objects • Background plane detection and growing of the object segmentation based on disparity planes improve both segmentation and tracking

23

Thank you!

24