Fruit detection and 3D location using instance segmentation neural networks and structure-from-motion photogrammetry

Gené Mola, Jordi; Sanz Cortiella, Ricardo; Rosell Polo, Joan Ramon; Morros Rubió, Josep Ramon; Ruiz Hidalgo, Javier; Vilaplana Besler, Verónica; Gregorio López, Eduard

doi:https://doi.org/10.1016/j.compag.2019.105165

View/Open

Postprint (1.856Mb)
Sol·licita una còpia

Issue date

2020-01-02

Author

Gené Mola, Jordi

Sanz Cortiella, Ricardo

Rosell Polo, Joan Ramon

Morros Rubió, Josep Ramon

Ruiz Hidalgo, Javier

Vilaplana Besler, Verónica

Gregorio López, Eduard

Suggested citation

Gené Mola, Jordi; Sanz Cortiella, Ricardo; Rosell Polo, Joan Ramon; Morros Rubió, Josep Ramon; Ruiz Hidalgo, Javier; Vilaplana Besler, Verónica; Gregorio López, Eduard; . (2020) . Fruit detection and 3D location using instance segmentation neural networks and structure-from-motion photogrammetry. Computers and Electronics in Agriculture, 2020, vol. 169, article number 105165. https://doi.org/10.1016/j.compag.2019.105165.

Abstract

The development of remote fruit detection systems able to identify and 3D locate fruits provides opportunities to improve the efficiency of agriculture management. Most of the current fruit detection systems are based on 2D image analysis. Although the use of 3D sensors is emerging, precise 3D fruit

location is still a pending issue. This work presents a new methodology for fruit detection and 3D location consisting of: (1) 2D fruit detection and segmentation using Mask R-CNN instance segmentation neural network; (2) 3D point cloud generation of detected apples using structure-from-motion (SfM) photogrammetry; (3) projection of 2D image detections onto 3D space; (4) false positives removal using a trained support vector machine. This methodology was tested on 11 Fuji apple trees containing a total of 1455 apples. Results showed that, by combining instance segmentation with SfM the system performance increased from an F1-score of 0.816 (2D fruit detection) to 0.881 (3D fruit detection and location) with respect to the total amount of fruits. The main advantages of this methodology are the reduced number of false positives and the higher detection rate, while the main disadvantage is the high processing time required for SfM, which makes it presently unsuitable for real-time work. From these results, it can be concluded that the combination of instance segmentation and SfM provides high performance fruit detection with high 3D data precision. The dataset has been made publicly available and an interactive visualization of fruit detection results is accessible at http://www.grap.udl.cat/documents/photogrammetry_fruit_detection.html. Dades primàries associades a l'article http://hdl.handle.net/10459.1/68505

Is part of

Computers and Electronics in Agriculture, 2020, vol. 169, article number 105165

Collections

The following license files are associated with this item:

Creative Commons

Except where otherwise noted, this item's license is described as cc-by-nc-nd (c) Elsevier, 2020