EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Sign Language Recognition with Multimodal Sensors and Deep Learning Methods

Chenghong Lu, Misaki Kozakai, Lei Jing

Electronics · 29.11.2023 · 12 (23) · 4827

DOI: 10.3390/electronics12234827 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Electronics, online erschienen am 29.11.2023. Verfasst von Chenghong Lu und 2 weiteren Autor:innen. Die Arbeit wurde bisher 26-mal zitiert (Crossref) und nennt 30 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: JSPS KAKENHI, JKA Foundation, NEDO Intensive Support for Young Promising Researchers.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Sign language recognition is essential in hearing-impaired people’s communication. Wearable data gloves and computer vision are partially complementary solutions. However, sign language recognition using a general monocular camera suffers from occlusion and recognition accuracy issues. In this research, we aim to improve accuracy through data fusion of 2-axis bending sensors and computer vision. We obtain the hand key point information of sign language movements captured by a monocular RGB camera and use key points to calculate hand joint angles. The system achieves higher recognition accuracy by fusing multimodal data of the skeleton, joint angles, and finger curvature. In order to effectively fuse data, we spliced multimodal data and used CNN-BiLSTM to extract effective features for sign language recognition. CNN is a method that can learn spatial information, and BiLSTM can learn time series data. We built a data collection system with bending sensor data gloves and cameras. A dataset was collected that contains 32 Japanese sign language movements of seven people, including 27 static movements and 5 dynamic movements. Each movement is repeated 10 times, totaling about 112 min. In particular, we obtained data containing occlusions. Experimental results show that our system can fuse multimodal information and perform better than using only skeletal information, with the accuracy increasing from 68.34% to 84.13%.

Förderung

  • JSPS KAKENHI
  • JKA Foundation
  • NEDO Intensive Support for Young Promising Researchers

Zitieren

APA

Lu, C., Kozakai, M., & Jing, L. (2023). Sign Language Recognition with Multimodal Sensors and Deep Learning Methods. Electronics, 12(23), 4827. https://doi.org/10.3390/electronics12234827

Vancouver

Lu C, Kozakai M, Jing L. Sign Language Recognition with Multimodal Sensors and Deep Learning Methods. Electronics. 2023;12(23):4827. doi:10.3390/electronics12234827

BibTeX
@article{Lu2023,
  author = {Lu, Chenghong and Kozakai, Misaki and Jing, Lei},
  title = {Sign Language Recognition with Multimodal Sensors and Deep Learning Methods},
  journal = {Electronics},
  year = {2023},
  volume = {12},
  number = {23},
  pages = {4827},
  doi = {10.3390/electronics12234827}
}
Kostenlos starten →