A real-time Indian Sign Language interpreter that bridges gesture and voice — in hospitals, restaurants, airports, classrooms, and everywhere in between.
What's great
Silent Interpreter tackles a real, underserved problem — only 1 in 6 of India's 70M+ deaf and hard-of-hearing people has access to a trained sign language interpreter. Rather than just another captioning tool, it's designed around a full pipeline (camera → gesture recognition → language generation → speech output) built on real ISL research datasets, aiming for two-way communication instead of one-directional captioning. It's early — a hackathon MVP with the UI and simulation working and the AI backend still to come — but the architecture and research grounding are solid, and the vision is genuinely needed.
What needs improvement
Connect the live AI backend — currently the demo runs as a UI simulation, not live inference
Expand gesture vocabulary beyond the current ~25 demo signs toward full ISL coverage
Add support for regional ISL dialect variations
Add offline/low-bandwidth support for real-world use in hospitals and rural areas
Improve accuracy for continuous signing (not just isolated gestures)
vs Alternatives
Most existing tools focus on one side of the problem — either speech-to-text captioning (Otter.ai, Google Meet captions) or isolated sign-recognition demos. Few attempt full two-way interpretation grounded in actual ISL datasets (INCLUDE, ISLTranslate, ISL-CSLTR, CISLR, ISL-FS) rather than generic ASL models repurposed for India. We chose to build for ISL specifically because it's underserved compared to ASL-focused tools, even though that means starting with a smaller vocabulary and building up.
Ratings
Ease of use
Reliability
Value for money
Customization
Report
25 views
What's great
The accuracy, smooth interface, and real-time responsiveness make cross-language communication effortless. It feels completely natural and non-intrusive.
vs Alternatives
I chose Silent Interpreter because of its superior integration, low latency, and non-disruptive user experience compared to clunky alternative tools.
Ratings
Ease of use
Reliability
Value for money
Customization
Report
30 viewsWhat's great
I just showed this to 2 of my frnds and now we all 3 are in awe of this really amazing thing developed by you folks....SO PROUD
What needs improvement
More sign languages could be added
vs Alternatives
i wandered over some tools like....otter.Ai, Ava but this one stood out to me as it was just too good and so docile compared to rest out there
Ratings
Ease of use
Reliability
Value for money
Customization
Report
6 views
The problem
India has millions of Deaf and hard-of-hearing signers, but Indian Sign Language (ISL) is one of the most under-resourced sign languages in the world when it comes to AI tooling. Most sign-language tech you've heard of targets ASL. ISL has had almost none of that attention. fewer datasets, fewer models, fewer products.
We set out to build something narrower but real: a live isolated-sign interpreter, point a camera at someone signing a single ISL word, and see (and hear) the translation instantly, entirely on-device.
Why "isolated," not "continuous"
Fluent, continuous sign language: full sentences, signed at natural speed is still a genuinely unsolved research problem, for ISL or any sign language. It requires modeling grammar, facial non-manual markers, and knowing where one sign ends and the next begins, all with almost no continuous-sentence training data available for ISL.
So we scoped down to something achievable: one sign at a time, recognized reliably, rather than a fluent conversation handled badly. That's a real, useful building block and honestly, most sign-language products that actually ship work this way too.
How it works
Training (Google Colab):
- Pulled ISL video data from INCLUDE, the largest available isolated-sign ISL dataset (AI4Bharat), covering 263 distinct signs.
- Extracted hand and body landmarks from every video using MediaPipe pose (33 points) plus both hands (21 points each) rather than training on raw pixels. This is dramatically more data-efficient and is the standard approach across current ISL research.
- Trained a Bidirectional LSTM classifier on these landmark sequences, with regularization tuned to avoid overfitting on a modest dataset.
- Exported the trained model to TensorFlow.js, so it can run directly in a browser no server, no API calls, no sending anyone's camera feed anywhere.
Live inference (the web app):
- MediaPipe runs live in the browser, extracting the same landmarks from your camera feed in real time.
- A motion-based segmentation system watches for when your hands start moving (a sign beginning) and go still again (a sign ending) rather than blindly chopping the video into arbitrary time windows.
- The captured sign gets resampled to match the exact format the model was trained on, then classified.
- The result appears as text and gets spoken aloud instantly, so a hearing person in the room doesn't even need to be looking at the screen.
What we learned the hard way
The model itself was, honestly, the easy part. The real war was fought in dependency hell: MediaPipe deprecated its entire Python API mid-project, TensorFlow's newest releases broke the model-conversion tooling in about five different, increasingly obscure ways, and we eventually had to spin up a whole separate, older Python environment just to get a trained model out of Colab in a usable format.
Where it stands, and where it's honestly limited
- Trained on 261 of INCLUDE's 263 sign classes.
- Runs fully client-side camera in, prediction + speech out, nothing leaves the device.
- Like any ISL model trained on a modest, non-crowdsourced dataset, accuracy will vary by sign, by signer, and by how closely someone's signing style matches the training videos. This is a prototype demonstrating a real, working pipeline not a clinically validated interpreting tool.
- Real-world deployment of anything like this should involve Deaf and ISL-fluent reviewers from the start, not just after the fact.
What's next
Expanding vocabulary coverage, tightening segmentation for faster/slower signers, and the much harder frontier eventually exploring continuous, sentence-level ISL recognition, which remains an open problem the research community (including groups like RKMVERI) is actively working on.
@hardi_jain1 Such an important topic! What is the main setting or situation the user should use it in? Is there any chance you are planning to add other sign langauges?
@maria_telegina thank you so much, maria! 🤍
main setting: we built it for everyday situations where deaf/HoH individuals in India face communication gaps hospitals, schools, and public spaces are our biggest focus, since access to trained ISL interpreters is really limited (only 1 in 6 deaf Indians has access to one)
right now it converts ISL gestures to english and hindi text + speech in real time
other sign languages: not on the immediate roadmap well we're deeply focused on ISL first since it's under-resourced compared to ASL/BSL in existing research
but our long-term vision (continuous ISL → transformer models → offline edge AI → eventually universal communication) leaves room for expanding beyond ISL down the line
for now we want to get ISL right!
@hardi_jain1 That's an amazing mission! Especially for the hospitals or bureaucratic organisations could be a real change in accessibility.
Possibly you could give a little more of the explanation of this important context here or in one of the threads in the forum of the product so that we all are more familiar with the history and the scale of the problem.
Looking forward to following the further journey of your product!
@maria_telegina thank you so much maria, that means a lot 💙
happy to share more context
right now in india there are over 70 million deaf and hard of hearing people, making it the largest such community in the world, but only about 1 in 6 has access to a trained isl interpreter
hospitals are actually one of the most urgent cases for us, imagine trying to describe symptoms or understand a diagnosis without a shared language, it can genuinely be a matter of safety, not just convenience bureaucratic spaces like courts, banks, and government offices carry the same weight since a lot of important decisions happen there and people get left out simply because there's no interpreter present