Multi-modal deep learning for Fuji apple detection using RGB-D cameras and their radiometric capabilities
Vilaplana Besler, Verónica
Morros Rubió, Josep Ramon
Ruiz Hidalgo, Javier
MetadataShow full item record
Fruit detection and localization will be essential for future agronomic management of fruit crops, with applications in yield prediction, yield mapping and automated harvesting. RGB-D cameras are promising sensors for fruit detection given that they provide geometrical information with color data. Some of these sensors work on the principle of time-of-flight (ToF) and, besides color and depth, provide the backscatter signal intensity. However, this radiometric capability has not been exploited for fruit detection applications. This work presents the KFuji RGB-DS database, composed of 967 multi-modal images containing a total of 12,839 Fuji apples. Compilation of the database allowed a study of the usefulness of fusing RGB-D and radiometric information obtained with Kinect v2 for fruit detection. To do so, the signal intensity was range corrected to overcome signal attenuation, obtaining an image that was proportional to the reflectance of the scene. A registration between RGB, depth and intensity images was then carried out. The Faster R-CNN model was adapted for use with five-channel input images: color (RGB), depth (D) and range-corrected intensity signal (S). Results show an improvement of 4.46% in F1-score when adding depth and range-corrected intensity channels, obtaining an F1-score of 0.898 and an AP of 94.8% when all channels are used. From our experimental results, it can be concluded that the radiometric capabilities of ToF sensors give valuable information for fruit detection.
Is part ofComputers and Electronics in Agriculture, 2019, vol. 162, p. 689-698
European research projects
The following license files are associated with this item:
Showing items related by title, author, creator and subject.
KFuji RGB-DS database: Fuji apple multi-modal images for fruit detection with color, depth and range-corrected IR data Gené Mola, Jordi; Vilaplana Besler, Verónica; Rosell Polo, Joan Ramon; Morros Rubió, Josep Ramon; Ruiz Hidalgo, Javier; Gregorio López, Eduard (Elsevier Inc., 2019)This article contains data related to the research article entitle 'Multi-modal Deep Learning for Fruit Detection Using RGB-D Cameras and their Radiometric Capabilities' . The development of reliable fruit detection and ...
Gené Mola, Jordi; Vilaplana Besler, Verónica; Rosell Polo, Joan Ramon; Morros Rubió, Josep Ramon; Ruiz Hidalgo, Javier; Gregorio López, Eduard (Universitat de Lleida, 2020-05)The KFuji RGB-DS dataset is composed by 967 multi-modal images of Fuji apples on trees captured using Microsoft Kinect v2 (Microsoft, Redmond, WA, USA). Each image contains information from 3 different modalities: color ...
Gené Mola, Jordi; Sanz Cortiella, Ricardo; Rosell Polo, Joan Ramon; Morros Rubió, Josep Ramon; Ruiz Hidalgo, Javier; Vilaplana Besler, Verónica; Gregorio López, Eduard (Universitat de Lleida, 2020)The Fuji-SfM dataset includes: (1) a set of 288 colour images and the corresponding annotations (apples segmentation masks) for training instance segmentation neural networks such as Mask-RCNN; (2) a set of 582 images ...