Artificial Intelligence-Based Indian Sign Language to Speech Conversion Using Computer Vision and Deep Learning: A Review

Meshwa Bhatt; Nirali Dave; Hemangini Patel

doi:10.55524/ijircst.2026.14.5.8

Abstract

Indian Sign language provides an important means of communication for deaf and hard of hearing individuals in India. However, communication might be difficult when people who do not understand ISL have to interact with Sign language users. Artificial Intelligence, Computer Vision and deep learning offer possible ways for recognizing signs from visual inputs and converting recognized information into text and speech. This paper reviews existing research on ISL recognition, translation, datasets and sign to speech conversion. It also discusses some techniques such as Convolutional Neural Network (CNN), Long Short-Term Memory (LSTM) Networks, Gated Recurrent Units (GRUs), and transformer-based models. It also examines datasets, such as INCLUDE and ISL Translate, alongside international sign language datasets used in related research work. The reviewed literature shows the distinction between recognizing individual signs and translating continuous sign sequences into full sentences. Some important challenges include limited and diverse ISL data, changes in the variation among signers, changes in lighting and background, and the need for effective real time processing. Based on the reviewed studies, the paper shows a conceptual framework that combines sign recognition, language processing, and text to speech conversion. The framework provides a direction for further future research; its complete implementation and performance evaluation remain outside the scope of this review.

Keywords

Artificial Intelligence, Computer Vision, Deep Learning, Indian Sign Language, Sign Language Recognition, Text-to-Speech

References

[1] K. Tripathi and N. B. G. C. Nandi, “Continuous Indian sign language gesture recognition and sentence formation,” Procedia Computer Science, vol. 54, pp. 523–531, 2015. Available: https://doi.org/10.1016/j.procs.2015.06.060

[2] Sridhar, R. G. Ganesan, P. Kumar, and M. Khapra, “Include: A large-scale dataset for Indian sign language recognition,” in Proc. 28th ACM Int. Conf. Multimedia, 2020, pp. 1366–1375. Available: https://doi.org/10.1145/3394171.3413528

[3] P. K. Athira, C. J. Sruthi, and A. Lijiya, “A signer independent sign language recognition with co-articulation elimination from live videos: An Indian scenario,” Journal of King Saud University—Computer and Information Sciences, vol. 34, no. 3, pp. 771–781, 2022. Available: https://doi.org/10.1016/j.jksuci.2019.05.002

[4] B. Subramanian, B. Olimov, S. M. Naik, S. Kim, K. H. Park, and J. Kim, “An integrated MediaPipe-optimized GRU model for Indian sign language recognition,” Scientific Reports, vol. 12, Art. no. 11964, 2022. Available: https://doi.org/10.1038/s41598-022-15998-7

[5] S. Katoch, V. Singh, and U. S. Tiwary, “Indian sign language recognition system using SURF with SVM and CNN,” Array, vol. 14, Art. no. 100141, 2022. Available: https://doi.org/10.1016/j.array.2022.100141

[6] D. Kothadiya, C. Bhatt, K. Sapariya, K. Patel, A. B. Gil-González, and J. M. Corchado, “DeepSign: Sign language detection and recognition using deep learning,” Electronics, vol. 11, no. 11, Art. no. 1780, 2022. Available: https://doi.org/10.3390/electronics11111780

[7] S. Bhamare and S. Bhamare, “Translating the unspoken: Deep learning approaches to Indian sign language recognition using CNN and LSTM networks,” in Proc. Annu. Int. Conf. Emerging Research Areas: Int. Conf. Intelligent Systems (AICERA/ICIS), 2023, pp. 1–6. Available: https://doi.org/10.1109/AICERA/ICIS59538.2023.10420379

[8] Joshi, S. Agrawal, and A. Modi, “ISLTranslate: Dataset for translating Indian sign language,” in Findings of the Association for Computational Linguistics: ACL 2023, 2023, pp. 10466–10475. Available: https://doi.org/10.18653/v1/2023.findings-acl.665

[9] N. C. Camgoz, S. Hadfield, O. Koller, H. Ney, and R. Bowden, “Neural sign language translation,” in Proc. IEEE/CVF Conf. Computer Vision and Pattern Recognition (CVPR), 2018, pp. 7784–7793. Available: https://doi.org/10.1109/CVPR.2018.00812

[10] N. C. Camgoz, O. Koller, S. Hadfield, and R. Bowden, “Sign language transformers: Joint end-to-end sign language recognition and translation,” in Proc. IEEE/CVF Conf. Computer Vision and Pattern Recognition (CVPR), 2020, pp. 10023–10033. Available: https://doi.org/10.1109/CVPR42600.2020.01004

[11] A. Duarte, S. Palaskar, L. Ventura, D. Ghadiyaram, K. DeHaan, F. Metze, et al., “How2Sign: A large-scale multimodal dataset for continuous American sign language,” in Proc. IEEE/CVF Conf. Computer Vision and Pattern Recognition (CVPR), 2021, pp. 2734–2743. Available: https://doi.org/10.1109/CVPR46437.2021.00276

[12] D. Li, C. Rodriguez Opazo, X. Yu, and H. Li, “Word-level deep sign language recognition from video: A new large-scale dataset and methods comparison,” in Proc. IEEE/CVF Winter Conf. Applications of Computer Vision (WACV), 2020, pp. 1459–1469. Available: https://doi.org/10.1109/WACV45572.2020.9093512

[13] S. Jiang, B. Sun, L. Wang, Y. Bai, K. Li, and Y. Fu, “Skeleton aware multi-modal sign language recognition,” in Proc. IEEE/CVF Conf. Computer Vision and Pattern Recognition Workshops (CVPRW), 2021, pp. 3413–3423. Available: https://doi.org/10.1109/CVPRW53098.2021.00380

[14] B. Zhou, Z. Chen, A. Clapés, J. Wan, Y. Liang, S. Escalera, et al., “Gloss-free sign language translation: Improving from visual-language pretraining,” in Proc. IEEE/CVF Int. Conf. Computer Vision (ICCV), 2023, pp. 20871–20881. Available: https://doi.org/10.1109/ICCV51070.2023.01908

[15] K. Lin, X. Wang, L. Zhu, K. Sun, B. Zhang, and Y. Yang, “Gloss-free end-to-end sign language translation,” in Proc. 61st Annu. Meeting Association for Computational Linguistics (ACL), vol. 1: Long Papers, 2023, pp. 12904–12916. Available: https://doi.org/10.18653/v1/2023.acl-long.722

[16] R. Wong, N. C. Camgoz, and R. Bowden, “Sign2GPT: Leveraging large language models for gloss-free sign language translation,” in Proc. Int. Conf. Learning Representations (ICLR), 2024. Available: https://proceedings.iclr.cc/paper_files/paper/2024/hash/4ebba705ffdee81e0a638c99fe066ce2-Abstract-Conference.html

Cites this article as

M. Bhatt, N. Dave, H. Patel, "Artificial Intelligence-Based Indian Sign Language to Speech Conversion Using Computer Vision and Deep Learning: A Review", International Journal of Innovative Research in Computer Science and Technology (IJIRCST), vol. 14, no. 5, pp. 63-70, 2026. doi: https://doi.org/10.55524/ijircst.2026.14.5.8

Corresponding Author

Meshwa Bhatt

Research Scholar, Department of Computer Science, Vanita Vishram Women’s University, Surat, Gujrat, India

Download Full Paper

Download PDF

No. of Downloads: 6 | No. of Views: 35

Artificial Intelligence-Based Indian Sign Language to Speech Conversion Using Computer Vision and Deep Learning: A Review

Citations

Total View 35

Total Download 6