ETRI Successfully Delivers Sound to the Hearing Impaired Through Tactile Sensation

Senior Researchers Shin Seung-yong and Jung Seung-eun of the Electronics and Telecommunications Research Institute (ETRI) are demonstrating a training process that identifies precise pitches through vibrations using a tactile pitch system.

[Asia Economy Reporter Seulgina Cho] Domestic researchers have developed a technology that allows hearing-impaired individuals to perceive sound through tactile sensations. This advancement enables hearing-impaired people to produce more accurate tones and sing or speak with their desired voice.

The Electronics and Telecommunications Research Institute (ETRI) announced on the 19th that it has developed a ‘Tactile Pitch System’ that analyzes the pitch of surrounding sounds and the user's own voice and converts it into tactile patterns.

Although medical and ICT advancements have made it possible for hearing-impaired individuals who have undergone cochlear implant surgery to engage in smooth spoken conversations with the general public, there have still been limitations in distinguishing pitch. To address this, ETRI developed a technology that allows sound to be perceived through tactile nerves rather than auditory ones. This user interface (UI) technology extracts frequency signals from auditory information such as music or sounds, recognizes the pitch, and converts it into tactile patterns delivered to the wearer’s skin.

ETRI explained, "Hearing-impaired individuals have faced many restrictions in enjoying music or singing. By utilizing this technology, they can feel the pitch of sounds from their surroundings or their own voice through their skin."

The method devised by the research team involves delivering vibrations to the first joint of the index finger on the left hand via a glove when the ‘Do’ note of a 4-octave scale is heard nearby. Considering the structure of the hand and ease of perception, 36 notes corresponding to 3 octaves are expressed as tactile patterns on one hand. However, since pitch is identified by the vibration location on different parts of the hand, about a month of training is required to learn to recognize the pitch of surrounding sounds and one’s own voice through tactile sensations.

After undergoing the developed learning method and training process, users can also adjust their voice to match the desired pitch. This technology can be used not only by hearing-impaired individuals but also by the elderly as an aid for language and sound learning.

The research team conducted commissioned research with Kangnam University to observe the effects of the Tactile Pitch System. Two hearing-impaired individuals who had cochlear implant surgery participated in the clinical study. After approximately 15 hours of training over a month, participants improved their ability to understand pitch through tactile sensations and produce desired tones with their own voice by about three times.

Although institutions such as Stanford University, Rice University, and Facebook in the United States have conducted research on conveying voice and text information through tactile means, commercializing a system that can deliver all information solely through tactile feedback still requires significant follow-up research. The ETRI Tactile Pitch System was developed based on actual user demands that cochlear implant and hearing aid users want to perceive pitch, even if not all information, and has the advantage of immediate applicability to user environments. It is also the first research case proven through clinical trials.

The research team expects this technology to greatly assist not only in music activities for the hearing-impaired but also in improving communication. Previous studies have shown that combining music activities during language rehabilitation training helps improve understanding of spoken language and sounds. The team plans to collaborate with related associations and organizations to improve the system’s wearability and completeness and to establish more effective special education methods and training standards.

Shin Hyung-cheol, head of ETRI’s Human Augmentation Research Lab and the lead researcher, stated, “We have strived to develop appropriate technology that truly meets the needs of minorities in our society, not technology for technology’s sake. We hope this technology will be widely used as a warm welfare ICT that can practically help many people.” Professor Lee Jun-woo of Kangnam University’s Department of Social Welfare also said, “Generally, hearing-impaired individuals have a limited range of sounds they can produce. It is revolutionary that through this training, they can produce sounds in ranges that were previously difficult to achieve.”

The research team also plans to develop wearable devices such as wristbands and arm bands to facilitate easier training and comfortable wear. In addition, ETRI is developing technology that detects the direction and location of sounds signaling dangerous situations, such as fire alarms or traffic signals, and conveys this information through tactile feedback via wearable devices. Field tests targeting hearing-impaired individuals are currently being prepared.

This content was produced with the assistance of AI translation services.