EUVIMEDEUVIMED entdecken
Crossref

The Trumpet of the Swan: Decoding Lip Language for Speech-impaired Users through COTS Earbuds

Ming Gao, Yichen Dai, Zhanbo Zhou, Zhengxin Guo, Jinsong Han, Fu Xiao

Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · 02.12.2025 · 9 (4) · 1-32

DOI: 10.1145/3770648 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, online erschienen am 02.12.2025. Verfasst von Ming Gao und 5 weiteren Autor:innen. Die Arbeit wurde bisher 3-mal zitiert (Crossref) und nennt 91 Literaturangaben. Gefördert von: the National Science Fund for Distinguished Young Scholars of China, National Natural Science Foundation of China, Natural Science Foundation of Jiangsu Province of China.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Lip language is a straightforward method of non-verbal communication for people with laryngeal and lingual injuries, dysphonia, or vocal cord lesions. It is convenient for these speech-impaired individuals, especially when their hands are occupied. However, it depends mainly on line-of-sight-required sensors (e.g., cameras for vision-based and speaker/microphone for acoustic-based solutions) or specialized peripherals to collect and understand the lip language. Therefore, the usage of lip language is limited. Recently, inertial measurement units (IMUs) have been increasingly embedded among commercial off-the-shelf (COTS) earbuds (e.g., Apple AirPods). The built-in IMUs have the potential to sense silent articulatory gestures. However, these sensors present low sampling rates (of merely 25 Hz) and signal-to-noise ratios, blocking error-free lip reading. As a countermeasure, we comprehensively analyze the articulations of speech-impaired persons. Accordingly, we map inertial signals to lip language with effective noise cancellation. We propose an inertia-based lip language decoding system, named SwanTrumpet. It supports lip reading, unvoiced interaction, and user identification for speech-impaired users. In addition, we address the issue of user diversity in lip language translation, which is significant among the speech-impaired. We implement the prototype connected with two COTS earbuds. Extensive evaluations on 11 speech-impaired participants demonstrate its capability to improve the interaction experience for people lacking a voice.

Förderung

  • the National Science Fund for Distinguished Young Scholars of China
  • National Natural Science Foundation of China
  • Natural Science Foundation of Jiangsu Province of China
  • the Natural Science Foundation of the Jiangsu Higher Education Institutions of China
  • Natural Science Research Start-up Foundation of Recruiting Talents of Nanjing University of Posts and Telecommunications

Zitieren

APA

Gao, M., Dai, Y., Zhou, Z., Guo, Z., Han, J., & Xiao, F. (2025). The Trumpet of the Swan: Decoding Lip Language for Speech-impaired Users through COTS Earbuds. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 9(4), 1-32. https://doi.org/10.1145/3770648

Vancouver

Gao M, Dai Y, Zhou Z, Guo Z, Han J, Xiao F. The Trumpet of the Swan: Decoding Lip Language for Speech-impaired Users through COTS Earbuds. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies. 2025;9(4):1-32. doi:10.1145/3770648

BibTeX
@article{Gao2025,
  author = {Gao, Ming and Dai, Yichen and Zhou, Zhanbo and Guo, Zhengxin and Han, Jinsong and Xiao, Fu},
  title = {The Trumpet of the Swan: Decoding Lip Language for Speech-impaired Users through COTS Earbuds},
  journal = {Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies},
  year = {2025},
  volume = {9},
  number = {4},
  pages = {1--32},
  doi = {10.1145/3770648}
}
Kostenlos starten →