Machine Learning-Based Gesture Recognition for Communication with the Deaf and Dumb

Prasanthi Yavanamandha; Bodduru Keerthana; Penmetsa Jahnavi; Koduganti Venkata Rao; Chatikam Raj Kumar

doi:10.52756/ijerr.2023.v34spl.004

Prasanthi Yavanamandha Department of CSE-AIML & IoT, Vallurupalli Nageswara Rao Vignana Jyothi Institute of Engineering & Technology, Hyderabad, Telangana, India
Bodduru Keerthana Department of Computer Science and Engineering, Gayatri Vidya Parishad College of Engineering (A), Visakhapatnam, Andhra Pradesh, India https://orcid.org/0009-0000-6820-514X
Penmetsa Jahnavi Department of Computer Science and Engineering, SRKR Engineering College Bhimavaram, Andhra Pradesh, India https://orcid.org/0000-0002-5245-4173
Koduganti Venkata Rao Department of Computer Science and Engineering, Guru Nanak Institutions Technical Campus, Hyderabad, India https://orcid.org/0000-0002-1633-7236
Chatikam Raj Kumar Department of Computer Science and Engineering, Gayatri College of Science and Management, Srikakulam, Andhra Pradesh, India

DOI: https://doi.org/10.52756/ijerr.2023.v34spl.004

Keywords: Sign Language, Computer Vision, Machine Learning, Convolutional Neural Network (CNN)

Abstract

A deep learning model specifically designed to recognize signs in sign language is the foundation of the Sign Language Recognition system. Sign Language is a visual language used by the deaf and hard of hearing community to communicate with one another and the general public. Sign language is a kind of nonverbal communication based on the use of hand gestures. The ability to communicate socially and emotionally is greatly aided when the speech and hearing challenged have access to sign language. The model developed in this paper captures the images through live web cam and displays the sign language meaning on the screen as text output. The model is trained and built by deep learning framework using Convolution Neural Networks (CNN) in this work. The model is trained with images of hand gestures captured through webcam using Computer Vision and then after successful training, the system performs recognition process through matching parameters for a given input gesture and finally displays the sign language meaning of the gesture as text output on the screen.

References

Abdul W., Alsulaiman, M., Amin, S.U., Faisal, M., Muhammad, G., Albogamy, F.R., & Ghaleb, H. (2021). Intelligent real-time Arabic sign language classification using attention-based inception and BiLSTM. Comput. Electr. Eng., 95, 107395. https://doi: 10.1016/j.compeleceng.2021.107395.

Adaloglou, N., Chatzis, T., Papastratis, I., Stergioulas, A., Papadopoulos, G.T., Zacharopoulou, V., Xydopoulos, G.J., Atzakas, K., Papazachariou, D., & Daras, P. (2021). A Comprehensive Study on Deep Learning-Based Methods for Sign Language Recognition. IEEE Transactions on Multimedia, 24, 1750-1762. https://doi.org /10.1109/TMM.2021.3070438

Agarwal, S.R., Agrawal, S.B., & Latif, A.M. (2015). Article: Sentence Formation in NLP Engine on the Basis of Indian Sign Language using Hand Gestures. Int. J. Comput. Appl., 116, 18–22. https://doi.org /10.5120/20428-2757.

Ali, M., Abbas, A.H., & Shahadi, H.I. (2022). Real-time sign language recognition system. International Journal of Health Sciences, 6(S4), 10384-10407. https://doi.org/10.46501/IJMTST0801006.

Breland, D.S., Dayal, A., Jha, A., Yalavarthy, P.K., Pandey, O.J., & Cenkeramaddi, L.R. (2021). Robust Hand Gestures Recognition Using a Deep CNN and Thermal Images. IEEE Sens. J., 21, 26602–26614. https://doi: 10.1109/JSEN.2021.3119977.

Aggarwal, D., Ahirwar, S., Srivastava, S., Verma, S., & Goel, Y. (2023). Sign Language Prediction using Machine Learning Techniques: A Review. Second International Conference on Electronics and Renewable Systems (ICEARS), Tuticorin, India. pp. 1296-1300, https://doi.org/10.1109/ICEARS56392.2023.10084924.

De Coster, M., Herreweghe, M.V., & Dambre, J. (2020). Sign Language Recognition with Transformer Networks. In Proceedings of the Conference on Language Resources and Evaluation (LREC 2020), Marseille, France, pp. 6018–6024. https://aclanthology.org/2020.lrec-1.737

Dhulipala, S., Adedoyin, F.F., & Bruno, A. (2022). Sign and Human Action Detection Using Deep Learning. J. Imaging., 8(7), 192. https://doi: 10.3390/jimaging8070192.

Goel, A., Wasim, J., & Srivastava, P. (2023). A Noise reduction in the medical images using

hybrid combination of filters with nature-inspired Black Widow Optimization Algorithm. International Journal of Experimental Research and Review, 30, 433-441. https://doi.org/10.52756/ijerr.2023.v30.040.

Itkarkar R.R., Nandi, A.K.V., Mungurwadi, V.B. (2021). Indian Sign Language Recognition Using Combined Feature Extraction. In: Mukherjee, M., Mandal, J., Bhattacharyya, S., Huck, C., Biswas, S. (eds) Advances in Medical Physics and Healthcare Engineering. Lecture Notes in Bioengineering. Springer, Singapore. pp. 1-7. https://doi.org/10.1007/978-981-33-6915-3_1

Kanisha, B., Mahalakshmi, V., Baskar, M., Vijaya, K., Kalyanasundaram, P. (2022) Smart communication using tri-spectral sign recognition for hearing Impaired people. J. Supercomput., 78, 2651–2664. https://doi.org/10.1007/s11227-021-03968-1.

Kothadiya, D., Bhatt, C., Sapariya, K., Patel, K., Gil-González, A.B., & Corchado, J.M. (2022). Deepsign: Sign Language Detection and Recognition Using Deep Learning. Electronics, 11, 1780. https://doi.org/10.3390/electronics11111780

Krishnan, G., Liya, B. S., Lakshmi, D.S.V., Sathyamoorthy, K., & Ganesan, S. (2023). Monkeypox Detection Using Hyper-Parameter Tuned Based Transferable CNN Model. International Journal of Experimental Research and Review, 33, 18-29. https://doi.org/10.52756/ijerr.2023.v33spl.003.

Likhar, P., Bhagat, N.K., & Rathna, G.N. (2020). Deep Learning Methods for Indian Sign Language Recognition. 2020 IEEE, 10th International Conference on Consumer Electronics (ICCE-Berlin), pp. 1-6. https://doi.org/10.1109/ICCE-Berlin50680.2020.9352194.

Liu, Y., Nand, P., & Hossain, M.A. (2023). Sign language recognition from digital videos using feature pyramid network with detection transformer. Multimed Tools Appl., 82, 21673–21685. https://doi.org/10.1007/s11042-023-14646-0.

Mittal, A., Kumar, P., Roy, P.P., Balasubramanian, R., & Chaudhuri, B.B. (2019). A Modified LSTM Model for Continuous Sign Language Recognition Using Leap Motion. IEEE Sensors Journal, 19, 7056-7063. https://doi.org/10.1109/JSEN.2019.2909837.

Mekala, P., Gao, Y., Fan, J., & Davari, A. (2011). Real-time sign language recognition based on Neural network architecture. In Proceedings of the IEEE 43rd Southeastern Symposium on System Theory, Auburn, AL, USA, 14–16 March. https://doi.org/10.1109/SSST.2011.5753805.

Rao, M.S., Uma Maheswaran, S.K., Sattaru, N.C., Abdullah, K.H., Pandey, U.K., & Biban, L. (2022). A Critical Understanding of Integrated Artificial Intelligence Techniques for the Healthcare Prediction System. 2nd International Conference on Advance Computing and Innovative Technologies in Engineering (ICACITE), Greater Noida, India, 2022, pp. 728-731. https://doi: 10.1109/ICACITE53722.2022.9823678.

Nandy, A., Prasad, J., Mondal, S., Chakraborty, P., & Nandi, G. (2010). Recognition of Isolated Indian Sign Language Gesture in Real Time. Commun. Comput. Inf. Sci., 70, 102–107. https://doi.org /10.1007/978-3-642-12214-9_18.

Papastratis, I., Chatzikonstantinou, C., Konstantinidis, D., Dimitropoulos, K., & Daras, P. (2021). Artificial Intelligence Technologies for Sign Language. Sensors, 21, 5843. https://doi.org/10.3390/s21175843.

Rakesh, S., Bharadhwaj, A. & Sree, H.E. (2021). Sign language recognition using convolutional neural network. In Innovative Data Communication Technologies and Application, 59, 7-7-719. https://doi.org/10.1007/978-981-15-9651-3_58.

Rajam, P.S., & Balakrishnan, G. (2011). Real time Indian Sign Language Recognition System to aid deaf-dumb people. 2011 IEEE, 13th International Conference on Communication Technology, pp. 737-742. https://doi: 10.1109/ICCT.2011.6157974.

Rao, M., Kumar, S., & Rao, K. (2023a). Effective medical leaf identification using hybridization of GMM-CNN. International Journal of Experimental Research and Review, 32, 115-123. https://doi.org/10.52756/ijerr.2023.v32.009.

Reddy, N. S., & Khanaa, V. (2023). Diagnosing and categorizing of pulmonary diseases using Deep learning conventional Neural network. International Journal of Experimental Research and Review, 31(Spl Volume), 12-22. https://doi.org/10.52756/10.52756/ijerr.2023.v31spl.002.

Rao, M.S., Kumar, S.P., & Rao, K.S. (2023b). A Methodology for Identification of Ayurvedic Plant Based on Machine Learning Algorithms. International Journal of Computing and Digital Systems, 2023, 10233-10241. http://dx.doi.org/10.12785/ijcds/140196.

Saha, A., & Yadav, R. (2023). Study on segmentation and prediction of lung cancer based on

machine learning approaches. International Journal of Experimental Research and Review, 30, 1-14. https://doi.org/10.52756/ijerr.2023.v30.001.

Sharma, P., & Anand, R.S. (2021). A comprehensive evaluation of deep models and optimizers for Indian sign language recognition. Graph. Vis. Comput., 5, 200032. https://doi: 10.1016/j.gvc.2021.200032.

Suneetha, M., Shivananda, B., Neha, B., Kiran, S.B. (2023). Hand gesture recognition and voice conversion for deaf and Dumb. 4th International Conference on Design and Manufacturing Aspects for Sustainable Energy (ICMED-ICMPC 2023). https://doi.org/10.1051/e3sconf/202339101060.

Wadhawan, A., & Kumar, P. (2019). Sign Language Recognition Systems: A Decade Systematic Literature Review. Archives of Computational Methods in Engineering, 28, 785 - 813. https://doi.org/10.1007/s11831-019-09384-2.

Wangchuk, K., Riyamongkol, P., & Waranusast, R. (2021). Real-time Bhutanese Sign Language digits recognition system using Convolutional Neural Network. Science Direct. ICT Express, 7, 215–220. https://doi: 10.1016/j.icte.2020.08.002.