Skip to content

This project will focus on sign language translation using wearable devices, which is able to help people having troubles with hearing and speaking in real world scenarios 😄

License

Notifications You must be signed in to change notification settings

manhph2211/Sign-Translator

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

4 Commits
 
 
 
 
 
 

Repository files navigation

Sign-Translator-Using-Wearables

Introduction

  • Sign Language is a form of communication by people who are deaf, hard of hearing, and non-verbal. This mainly employs signs made by moving the hands. However, not only hands, but also palm orientation, movement, location, and expression/non-manual signals, ... lead to the good performance of the Sign Language Recognition System.

  • Over 466 million people(5%, might reach 900M in 2050) are speech or hearing impaired, and 80% of them are semi-illiterate or illiterate(WHO).

    • In US, ASL is a native language for around 2,50,000-5,00,000 people.
    • In China, Chinese Sign Language is being used in China by approximately 1M to 20M deaf people.
    • In UK, Approximately 1,50,000 people use British Sign Language (BSL).
    • In India, approximately 1.5 million signers use Indo-Pakistani Sign Language.
  • One approach supporting phone call: A relay service (text and video based)​

    • Takes extra time, complex settings and inconsistent call ​
    • Only person should talk or type at a time​
    • Sensitive and private information or data & image might be leaked
  • SLR is really necessary but not easy to create a tool that meets daily conversation (including online) and there are gaps to fill in such as real-time and natural output decoding, sensor selection and placement optimization, and robustness with non-uniform background environments ...

  • Proposed pipeline for making phone call:

image

Related Works

  • Wearable System for Recognizing American Sign Language in Real-Time Using IMU and Surface EMG Sensors (2016):

    • Fusion IMU and sEMG signals and give suitable feature extraction and selection
    • Adaptive auto-segmentation to extract periods during which signs are performed using sEMG
    • Proved sEMG is useful to combine by experiments
  • DeepASL: Enabling Ubiquitous and Non-Intrusive Word and Sentence-Level Sign Language Translation (2017):

    • Uses Leap Motion – an infrared light-based sensing device extracting the skeleton joints information of fingers, palms and forearms
    • Propose 2 branch models for corresponding hands and fusion
  • SignSpeaker: A Real-time, High-Precision SmartWatch-based Sign Language Translator (2019)

    • Using portable and lightweight smartwatch and mobile phone
    • Supporting both fingerspelling and sentence-level using bilstm given extracted spectrogram
    • Additional TTS engine
  • SonicASL: An Acoustic-based Sign Language Gesture Recognizer Using Earphones (2021)

    • First method for using Sonic wav(inaudible) + earphones
    • Propose CNN-LSTM CTC
    • Support TTS engine
  • SignFi: Sign Language Recognition Using WiFi (2018 - IMWUT)

    • Collects CSI measurements to capture wireless signal characteristics of sign gestures.
    • Raw CSI measurements are pre-processed to remove noises and recover CSI changes over sub-carriers and sampling time
    • Only word-level recognition and relatively low accuracy in new-user
  • MyoSign: Enabling End-to-End Sign Language Recognition with Wearables (IUI 2019)​

    • Emphasize the application of EMG in SLR
    • Use CNN for feature extraction and then BiLSTM + CTC
  • GASLA: Enhancing the Applicability of Sign Language Translation (Infocom 2022)​

    • Introduce a method for collecting suitable word and sentence data (synthesized by words)
    • Also use BiLSTM and CTC and use spectrogram as input
    • Focus on the translation task, which is a higher version of the recognition task when outputting spoken text instead of sign text.
  • WearSign: Pushing the Limit of Sign Language Translation Using Inertial and EMG Wearables (IMWUT 2022)

    • Leverages a smartwatch and an armband of ElectroMyoGraphy (EMG) sensors to capture the sophisticated sign gestures​ to do translation task
    • Introduce a training method with 2 CTC layers and encoder-decoder architecture together with an attention mechanism.
    • Borrow the idea of back-translation and leverage the much more available spoken language data to synthesize the paired sign language data

About

This project will focus on sign language translation using wearable devices, which is able to help people having troubles with hearing and speaking in real world scenarios 😄

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published