Home > Published Issues > 2024 > Volume 15, No. 8, 2024 >
JAIT 2024 Vol.15(8): 923-940
doi: 10.12720/jait.15.8.923-940

An Intelligent Android System for Automatic Sign Language Recognition and Learning

Soha A. Shaban * and Dalia L. Elsheweikh 3
Department of Computer Science, Faculty of Specific Education, Mansoura University, Mansoura, Egypt
Email: prof_soha@mans.edu.eg (S.A.S.); dalia_lotfy@mans.edu.eg (D.L.E.)
*Corresponding author

Manuscript received March 25, 2024; revised May 6, 2024; accepted May 29, 2024; published August 7, 2024.

Abstract—This paper presents an intelligent Android system developed for automatic recognition of both Arabic and American sign languages, as well as for the teaching and learning of these sign languages. It contains two subsystems. The first subsystem, the Sensory Smart Glove System (SSG-Sys), is based on Internet of Things (IoT) and is designed for automatic sign language recognition. It comprises a smart glove equipped with five flex sensors, which measure the bending of fingers according to the gestures being performed, and an MPU-6050 accelerometer sensor to track the hand’s position across three axes (X, Y, Z). The sensed data are processed by an Arduino Nano microcontroller, and the text of the recognized gesture is transferred via HC-05 Bluetooth module to an Android phone. This phone displays the text and converts it into audible voice using an Android application. The SSG-Sys results demonstrated high recognition accuracy rates for Arabic Sign Language (ArSL) (98.42%) and American Sign Language (ASL) (98.22%). The second subsystem is the Mobile Augmented Sign Language Learning System (MASLL-Sys). It is a mobile educational app that leverages marker-based augmented reality technology, to enhance and make the sign language learning process more realistic and effective. It consists of five main modules: registration, learning, augmented learning, tests, and student module. Overall, the performance of the proposed intelligent system was evaluated by a group of experts, who revealed that it is a promising tool for sign language recognition and learning.
 
Keywords—Sign Language (SL), American Sign Language (ASL), Arabic Sign Language (ArSL), intelligent glove, Augmented Reality (AR)

Cite: Soha A. Shaban and Dalia L. Elsheweikh, "An Intelligent Android System for Automatic Sign Language Recognition and Learning," Journal of Advances in Information Technology, Vol. 15, No. 8, pp. 923-940, 2024.

Copyright © 2024 by the authors. This is an open access article distributed under the Creative Commons Attribution License (CC BY-NC-ND 4.0), which permits use, distribution and reproduction in any medium, provided that the article is properly cited, the use is non-commercial and no modifications or adaptations are made.