In the context of Robotics in Agricultural Engineering there is an increasing interest in the use of data produced by sensors that go beyond the visible RGB spectrum, such as hyper and multi spectral cameras. This data can be correlated to quality related physical characteristics, such as the the content of specific chemical substances (sugars, flavonoids, etc). Hyperspectral sensors are used to find the spectral signature of a substance that is dissolved in the fruit or vegetable juices. The spectral signature tells how the substance responds to different light wavelengths, however it is usually measured in laboratory controlled environments. Since robotic platforms work on the field, where the lighting conditions are non ideal, it is not possible to know in advance how the spectral signature would be influenced by the environment conditions. In addition, hyperspectral sensors are extremely expensive, and usually not compatible with the costs of regular agronomic operations. Cheaper multispectral cameras, with lower accuracy and fewer wavelengths than the hyperspectral devices, are often used for other more forgiving estimation tasks, such as the NDVI estimation. We hypothesize that the multispectral cameras could be used for spectral signature estimation in the wild compensating through data analysis and machine learning the loss of information. We propose a systematic study of the environmental noise sources while collecting multispectral images in different lighting conditions in order to model the correlation between the ideal spectral signature and the real approximate data collected with less accurate sensors. We also aim to explore semi-supervised learning strategies to reduce the labeling effort and make the approach more generally applicable to the Agronomic industry. As a real case study, we consider the estimation of sugar content in table grapes, for which the ideal spectral signature has been studied, and for which new data could be easily acquired.
Agriculture has a central role in the development of the world economy and sustainability of world population, however, the current research in robotics and artificial intelligence is only scratching the surface of the field, since the contributions are a constellation of solutions to very different problems, with no major methodological or practical problem to help the community develop a coherent set of methods and techniques.
As the research on this topic advances, we expect to see a set of general questions emerge in the Robotics community that will guide the development of more general methodologies and best practices.
With this in mind, one of the key aspects of this proposal is to advance the methodology
of work with spectrum detection devices, in order to propose a general approach to multi-spectral estimation in agricultural field robotics. It is easily perceivable that the table grape ripeness estimation problem is a workable example of a much wider category of agricultural problems: the proposed methodology is applicable to other cultivations with the same requirements (e.g. monitoring the ripeness of strawberries, or tomatoes) with minor practical considerations, but it could eventually be extendable to different crops with different acquisition modalities (e.g. monitoring wheat crops for pest control using autonomous aerial vehicles). For this reason, we are planning to generalize the outcomes of this work to the point of being able to draw the general methodology for a class of agricultural problems.
Solving a larger class of problems in perception for Agricultural Robotics will also help in going deeper in advancement in the field of Computer Vision itself. As shown in [12], where a semi-supervised approach to crop detection and counting was introduced, the specific problem is able to open the way to a much more general approach to learning strategies that could be applied in a more general way to other problems in different contexts.
With this in mind, another aspect of the proposal that should be stressed is the opportunity to advance state-of-the-art data driven computer vision with respect to semi-supervised or self-supervised techniques. While, given enough labelled data, training a supervised model with good performances is a relatively easy goal, when, as in this case, the labels are incomplete, partially available, noisy, or the information is split in mixed data modalities (images, tabular data, temporal series, ...), the methodologies to train an estimation algorithm have to be innovative and use semi-supervised or self-supervised approaches, such as variational methods that could be used to extract latent variables from noisy approximate inputs, multiple instance learning of domain adaptation, to name a few.
Finally, there are the advancements related to Robotics itself: enabling a robotic platform with the capability of assessing the ripeness of the fruit is an enabling factor to fully automated robotic harvesting. With this proposal we plan to test the estimation algorithms on a real robotic platform in the field, merging the autonomous navigation and mapping layers with the algorithms to guide the robot through the vineyard and map the ripeness level of all grape bunches. This test will hopefully lead, in future work, to the development of a fully automated harvesting platform, able to cut the grape bunches and collect them for transportation.
Summarizing, this project will address the following points:
- Modelling the non idealities of spectral response estimation with respect to the laboratory grade spectral signature.
- Applying advanced machine learning techniques to compensate for these non idealities and model the noise patterns and other sources of uncertainty in collecting data in the field
- Integrating the estimator on a real robotic platform and using it during the two following harvesting seasons.