Hand gesture recognition is still a topic of great interest for the computer vision community. In particular, sign language is a foremost area of interest due to their importance in Human-Human Communication (HHC). The sign language derives from the composition of the movements of the hand (and his fingers) with respect to the pose of the body (precisely the upper part and the entire region of the torso) of the person performing the gesture.
Any hand movement can be represented by sets of feature vectors that change over time. Recurrent Neural Networks (RNNs) are suited to analyse this type of sets thanks to their ability to model the long term contextual information of temporal sequences.
Instead, the Convolutional Neural Networks (CNNs) are suitable to learn the relevant features able to discern the various poses taken by the person while performing a complex gesture.
For this reasons, in this project, we will go to design an innovative network based on a combination of CNN and LSTM. This network will be trained by using as features the angles formed by the finger bones of the human hands and a video sequence of the movements of the upper part of the bust of the human body. A further work required for this project concerns the creation of a very huge dataset containing the data of the gestures, based on the Italian Sign Language (LIS), performed by a large number of people. Acquiring data from many people will capture the different nuances between the gestures performed by different people.
In the state-of-the-art of the gesture recognition, the main works are only focused on recognizing the subcategory of the semaforic gestures. Moreover, among these methods, deep learning is limitedly explored. The absence of big datasets containing sign language gestures is a further demonstration of how this problem is not fully managed by the scientific community. This is a limitation, since sign language offers a valid substitute for verbal communication with other people and machines. In addition, the results obtained in this field of research would help many people affected by mutism in their daily lives.
So, respect the state of the art, in this project we will want to use the potential of deep learning, since it has not yet been explored in the field of sign language recognition and we will propose a new innovative method based on the fusion of CNN and RNN networks. In addition, we will propose the first existing big dataset regarding the sign language .