Kelsey Atemie-Hart

SIGN LANGUAGE RECOGNITION APPROACHES IN DEEP LEARNING

INTRODUCTION

• Sign Language is a form of communication for the disabled with hearing and speaking disability.
• There are technologies that are developed to aid these communication method.
• The implementation of deep learning is a very useful approach to develop Sign Language Recognition (SLR) systems.
• There are currently two approaches, namely Sensor-based approach and Vision-based approach that are used to develop a SLR system in deep learning.

MOTIVATION

• The purpose of this research is to identify the approaches used under machine learning to develop a SLR system and to identify the weaknesses with these current approaches and provide a solution.
• Solution to accuracy and efficiency.
• To improves the SLR system for the disabled, making communication with other individuals easier.

PROBLEM

• The problem is bridging the communication gap between the hearing and speech impaired and the other individuals. The implementation of machine learning to aid SLR should be able to better the communication between individuals at a cheaper cost than employing a translator, effectively and accurately.

DEEP LEARNING APPROACHES FOR SLR SYSTEMS

• SENSOR-BASED APPROACH
  • This approach brings out the hand measurements, such as the hand positions and the orientation of the joints. It makes use of sensor gloves, digital camera, depth camera, accelerometer, leap motion controller, etc.

• VISION-BASED APPROACH
  • The vision based approach is the use of cameras to capture the hand gesture done by the individual without the use of gloves or any external tool.

COMPARISON OF BOTH APPROACHES

• Vision based approach:
  • Makes use of just camera to capture gesture making it more convenient
  • It has little to no building cost
  • It is more portable for end users to make use of
  • It has more computational cost due to the process taken to extract gesture from image
  • According to table below has more accuracy level from works done

• Sensor based approach:
  • It is not so convenient to use cause of the need to wear sensor gloves.
  • It has less computation cost due to the sensors within the gloves been able to directly report relevant data
  • It has more building cost due to gloves, accelerometers, microcontrollers, flex sensors etc.
  • It is not so portable to use.

ACCURACY COMPARISON OF BOTH APPROACHES

VISION BASED APPROACH METHOD

BEST APPROACH TO ADOPT

The gestures captured depict the fingers, palms, and face in terms of posture, location, and motion .

It is necessary to perform image processing in order to separate the signers' hands from the background items. However, picture noises from a variety of sources (camera, light, color matching, and backdrop) are associated with the extraction of the user's facial and body actions, limiting dynamic image recognition. Despite this, error filters are utilized to repair the harm.

This approach also makes use of classification algorithms and methods in deep learning that increases its efficiency such as the Hidden Markov Model (HMM) and Neural Networks.

In addition, this system can be easily used by individuals anywhere and anytime, it is more scalable, portable and efficient

IMAGE ACQUISITION:

This is actually the first step in recognition. The built-in desktop/laptop web camera will be used to capture the sign gestures carried out by the user. This camera needs to be in high definition so as to capture quality imagery of the hand gestures and increase accuracy of interpretation.

IMAGE PREPROCESSING/SEGMENTATION

This involves the cropping, filtering, the brightness and contrast adjustment, etc. of the image taken. To carry out the task, image segmentation method is used alongside image cropping and enhancement. The images captured are in RGB form, which are then converted to binary images. Cropping of the image is done so that the unwanted part of the image is extracted. The selected area will undergo image enhancement. Edge detection is another method that is done in image segmentation, it is the detection of the boundary of the cropped areas of the image used later for feature extraction. With the use of OpenCV, the images used are revised to the same size to eliminate differences in images of different gestures.

FEATURE EXTRACTION

This is useful to create a database of sign recognition. For letter image similarity characterization, both local and global visual features are recovered to quickly and effectively characterize the varied visual principles of letters in the alphabet. The two methods of feature extraction include; Contour-based shape representation and description method and the other is the Region-based shape representation and description method.

CLASSIFICATION

Image classification is a supervised learning problem in which a model is trained to recognize a collection of target classes (things to identify in images) using labeled sample photos. Raw pixel data was used as the input to early computer vision models.

RECOMMENDATION

REFERENCES

• Shukor, A. Z., Miskon, M. F., Jamaluddin, M. H., Ali@Ibrahim, F., Asyraf, M., & Bahar, M. (2015). A New Data Glove Approach for Malaysian Sign Language Detection. 2015 IEEE International Symposium on Robotics and Intelligent Sensors (IRIS 2015), (pp. 60-67).
• Sutarman, Majid, M. B., & Zain, J. N. (2013). Vision-Based Sign Language Recognition Systems: A Review. Proceedings of the 2013 International Conference on Computer Science and Information Technology (CSIT-2013), (pp. 195-200).
• Tsuchiya, T., Shitara, A., Yoneyama, F., Kato, N., & Shiraishi, Y. (2020). ensor Glove Approach for Japanese Fingerspelling Recognition System Using Convolutional Neural Networks. ACHI 2020 : The Thirteenth International Conference on Advances in Computer-Human Intera, (pp. 34-39).
• Tubaiz, N., Shanableh, T., & Assaleh, K. (2015, August). Glove-Based Continuous Arabic Sign Language Recognition in User-Dependent Mode. IEEE Transactions on Human-Machhine Systems, 45(4), 526-533.
• Wehle, H.-D. (2017). Machine Learning, Deep Learning, and AI: What’s the Difference?

• Lokhande, P., Prajapati, R., & Pansare, S. (2015). Data Gloves for Sign Language Recognition System. International Journal of Computer Applications, 11-14.
• Mehdi, S. A., & Khan, Y. N. (2002). Sign Language Recognition using Sensor Gloves. z C, Leu MC (2011) American Sign Language word recogni- tion with a sensory glove using artificial neural networks. Eng Appl Artif Intell 24(7):1204–1 (pp. 2204-2206). IEEE.
• Pansare, J. R., & Ingle, M. (2016). Vision-Based Approach for American Sign Language Recognition Using Edge Orientation Histogram. 2016 International Confrence on Image, Vision and Computing (pp. 86-90). IEEE.
• Pigou, L., Herreweghe, M. V., & Dambre, J. (2017). Gesture and Sign Language Recognition with Temporal Residual Networks. 2017 IEEE International Conference on Computer Vision Workshops (ICCVW) (pp. 3086-3093). IEEE.
• Rosero-Montalvo, P. D., Godoy-Trujillo, P., Flores-Bosmediano, E., & Carrascal-García, J. (2018). Sign Language Recognition Based on Intelligent Glove Using Machine Learning Techniques. 2018 IEEE Third Ecuador Technical Chapters Meeting (ETCM), (pp. 1-5).
• Sharma, S., & Singh, S. (2021). Vision-based hand gesture recognition using deep learning for the interpretation of sign language. Expert Systems with Applications

• Ahmed, M. A., Zaidan, B. B., Zaidan, A. A., Salih, M. M., & Modi bin Lakulu, M. (2018). A Review on Systems-Based Sensory Gloves for Sign Language Recognition State of the Art between 2007 and 2017. 18(7).
• Al-Ahdal, M. E., & Tahir, N. M. (2012). Review in Sign Language Recognition Systems. 2012 IEEE Symposium on Computers & Informatics(ISCI), (pp. 52-57).
• Albawi, S., Mohammed, T. A., & Al-Zawi, S. (2017). Understanding of a Convolutional Neural Network. 2017 International Conference on Engineering and Technology (ICET) . Antalya, Turkey: IEEE.
• Cheok, M. j., Omar, Z., & Jaward, M. (2019). A review of hand gesture and sign language recognition techniques. International Journal of Machine Learning and Cubernetics, 10, 131-153.
• Kang, B., Tripathi, S., & Nguyen, T. Q. (2015). Real-time Sign Language Fingerspelling Recognition using Convolutional Neural Networks from Depth map.
• Lei, L., & Dashun, Q. (2015). Design of data-glove and chinese sign language recognition system based on ARM9. 2015 IEEE 12th International Conference on Electronic Measurement & Instruments, (pp. 1130-1134).