For a deaf or hard-of-hearing person, even an ordinary conversation in the classroom can become a barrier: the teacher does not know sign language, and an interpreter is not always available. In recent years artificial intelligence has started to fill this gap — systems are emerging that recognise signs through a camera and turn them into text and live speech.
How does the technology work?
Such a system usually consists of several stages:
- Computer vision: key points of the hands, fingers, facial expression and body posture are extracted from the camera image;
- Sequence analysis: a sign is a movement rather than a single frame, so a neural network (for example, an LSTM or a Transformer) learns sequences of movements;
- Translation into text: recognised signs are converted into words and sentences;
- Speech synthesis (TTS): the resulting text is played back as spoken audio.
Main challenges
Sign language is not spoken language “shown with the hands”; it is an independent language with its own grammar. Facial expression, the speed and direction of movement also carry meaning. Training a model requires large amounts of high-quality annotated video, and open datasets for the sign language used in our country are still very scarce. Lighting, background, camera angle and each person’s individual signing style also affect accuracy.
Why it matters for education
If such a system works in the classroom, a deaf student can more easily ask the teacher and classmates questions and take part in discussions. In the other direction, speech-to-text technology can display the teacher’s words on screen as subtitles. Together, the two technologies help create a truly inclusive environment.
A research field for students
This area is also an excellent hands-on project for computer science and AI students: collecting and labelling a dataset, training computer vision models and testing the result in real conditions are all essential skills for a modern specialist. Most importantly, such projects should be carried out in partnership with sign language users themselves.
AI cannot fully replace a human interpreter, but it can be a powerful assistant in reducing everyday communication barriers.