Silent Speech Recognition with Strain Sensors and Deep Learning Analysis of Directional Facial Muscle Movement.

Journal: ACS applied materials & interfaces
PMID: 36413961

Abstract

Silent communication based on biosignals from facial muscle requires accurate detection of its directional movement and thus optimally positioning minimum numbers of sensors for higher accuracy of speech recognition with a minimal person-to-person variation. So far, previous approaches based on electromyogram or pressure sensors are ineffective in detecting the directional movement of facial muscles. Therefore, in this study, high-performance strain sensors are used for separately detecting - and -axis strain. Directional strain distribution data of facial muscle is obtained by applying three-dimensional digital image correlation. Deep learning analysis is utilized for identifying optimal positions of directional strain sensors. The recognition system with four directional strain sensors conformably attached to the face shows silent vowel recognition with 85.24% accuracy and even 76.95% for completely nonobserved subjects. These results show that detection of the directional strain distribution at the optimal facial points will be the key enabling technology for highly accurate silent speech recognition.

Authors

  • Hyunjun Yoo
    Department of Electrical and Computer Engineering, Inter-University Semiconductor Research Center (ISRC), Seoul National University, Seoul08826, Korea.
  • Eunji Kim
    Samsung Advanced Institute of Technology, Samsung Electronics Co., Ltd., 130 Samsung-ro, Yeongtong-gu, Suwon 16678, Republic of Korea.
  • Jong Won Chung
    Organic Material Lab., Samsung Advanced Institute of Technology (SAIT), Samsung Electronics, Suwon16678, Korea.
  • Hyeon Cho
    Department of Electrical and Computer Engineering, Inter-University Semiconductor Research Center (ISRC), Seoul National University, Seoul08826, Korea.
  • Sujin Jeong
    Department of Electrical and Computer Engineering, Inter-University Semiconductor Research Center (ISRC), Seoul National University, Seoul08826, Korea.
  • Heeseung Kim
    Department of Electrical and Computer Engineering, Seoul National University, Seoul08826, Korea.
  • Dongju Jang
    Department of Electrical and Computer Engineering, Inter-University Semiconductor Research Center (ISRC), Seoul National University, Seoul08826, Korea.
  • Hayun Kim
    Department of Electrical and Computer Engineering, Inter-University Semiconductor Research Center (ISRC), Seoul National University, Seoul08826, Korea.
  • Jinsu Yoon
    Department of Electrical and Computer Engineering, Inter-University Semiconductor Research Center (ISRC), Seoul National University, Seoul08826, Korea.
  • Gae Hwang Lee
    Organic Material Lab., Samsung Advanced Institute of Technology (SAIT), Samsung Electronics, Suwon16678, Korea.
  • Hyunbum Kang
    Organic Material Lab., Samsung Advanced Institute of Technology (SAIT), Samsung Electronics, Suwon16678, Korea.
  • Joo-Young Kim
    Organic Material Lab., Samsung Advanced Institute of Technology (SAIT), Samsung Electronics, Suwon16678, Korea.
  • Youngjun Yun
    Organic Material Lab., Samsung Advanced Institute of Technology (SAIT), Samsung Electronics, Suwon16678, Korea.
  • Sungroh Yoon
    4 Department of Electrical and Computer Engineering and Interdisciplinary Program in Bioinformatics, Seoul National University, Seoul, Korea.
  • Yongtaek Hong
    Department of Electrical and Computer Engineering, Inter-University Semiconductor Research Center (ISRC), Seoul National University, Seoul08826, Korea.

Keywords

External Resources

Popular Topics
Recent Journals