Fruit detection and 3D location using instance segmentation neural networks and structure-from-motion photogrammetry

View/ Open
Issue date
2020-01-02Author
Morros Rubió, Josep Ramon
Ruiz Hidalgo, Javier
Vilaplana Besler, Verónica
Suggested citation
Gené Mola, Jordi;
Sanz Cortiella, Ricardo;
Rosell Polo, Joan Ramon;
Morros Rubió, Josep Ramon;
Ruiz Hidalgo, Javier;
Vilaplana Besler, Verónica;
Gregorio López, Eduard;
.
(2020)
.
Fruit detection and 3D location using instance segmentation neural networks and structure-from-motion photogrammetry.
Computers and Electronics in Agriculture, 2020, vol. 169, article number 105165.
https://doi.org/10.1016/j.compag.2019.105165.
Metadata
Show full item recordAbstract
The development of remote fruit detection systems able to identify and 3D locate fruits provides opportunities to improve the efficiency of agriculture management. Most of the current fruit detection systems are based on 2D image analysis. Although the use of 3D sensors is emerging, precise 3D fruit location is still a pending issue. This work presents a new methodology for fruit detection and 3D location consisting of: (1) 2D fruit detection and segmentation using Mask R-CNN instance segmentation neural network; (2) 3D point cloud generation of detected apples using structure-from-motion (SfM) photogrammetry; (3) projection of 2D image detections onto 3D space; (4) false positives removal using a trained support vector machine. This methodology was tested on 11 Fuji apple trees containing a total of 1455 apples. Results showed that, by combining instance segmentation with SfM the system performance increased from an F1-score of 0.816 (2D fruit detection) to 0.881 (3D fruit detection and location) with respect to the total amount of fruits. The main advantages of this methodology are the reduced number of false positives and the higher detection rate, while the main disadvantage is the high processing time required for SfM, which makes it presently unsuitable for real-time work. From these results, it can be concluded that the combination of instance segmentation and SfM provides high performance fruit detection with high 3D data precision. The dataset has been made publicly available and an interactive visualization of fruit detection results is accessible at http://www.grap.udl.cat/documents/photogrammetry_fruit_detection.html. Dades primàries associades a l'article http://hdl.handle.net/10459.1/68505
Related resource
http://hdl.handle.net/10459.1/68505Is part of
Computers and Electronics in Agriculture, 2020, vol. 169, article number 105165European research projects
The following license files are associated with this item:
Related items
Showing items related by title, author, creator and subject.
-
Fuji-SfM dataset: A collection of annotated images and point clouds for Fuji apple detection and location using structure-from-motion photogrammetry
Gené Mola, Jordi; Sanz Cortiella, Ricardo; Rosell Polo, Joan Ramon; Morros Rubió, Josep Ramon; Ruiz Hidalgo, Javier; Vilaplana Besler, Verónica; Gregorio López, Eduard (Elsevier, 2020)The present dataset contains colour images acquired in a commercial Fuji apple orchard (Malus domestica Borkh. cv. Fuji) to reconstruct the 3D model of 11 trees by using structure-from-motion (SfM) photogrammetry. The data ... -
Fuji-SfM dataset [Research data]
Gené Mola, Jordi; Sanz Cortiella, Ricardo; Rosell Polo, Joan Ramon; Morros Rubió, Josep Ramon; Ruiz Hidalgo, Javier; Vilaplana Besler, Verónica; Gregorio López, Eduard (Universitat de Lleida, 2020)The Fuji-SfM dataset includes: (1) a set of 288 colour images and the corresponding annotations (apples segmentation masks) for training instance segmentation neural networks such as Mask-RCNN; (2) a set of 582 images ... -
Fruit detection in an apple orchard using a mobile terrestrial laser scanner
Gené Mola, Jordi; Gregorio López, Eduard; Guevara, Javier; Auat Cheein, Fernando A.; Sanz Cortiella, Ricardo; Escolà i Agustí, Alexandre; Llorens Calveras, Jordi; Morros Rubió, Josep Ramon; Ruiz Hidalgo, Javier; Vilaplana Besler, Verónica; Rosell Polo, Joan Ramon (Academic Press. Published by Elsevier, 2019-09-21)The development of reliable fruit detection and localization systems provides an opportunity to improve the crop value and management by limiting fruit spoilage and optimised harvesting practices. Most proposed systems for ...