Hand Wave
Turn sign language into speech with smart glasses
151 followers
Turn sign language into speech with smart glasses
151 followers
Hand Wave turns sign language into text and speech using the camera on Meta smart glasses. It also works cross-platform (iOS + web). Under the hood is a lightweight, open-source neural network trained on Google’s FSBoard dataset and built to run locally across devices (wip).
Hand Wave
I’m building Hand Wave to make sign-language conversations more accessible with the camera already on Meta glasses (and other devices too). Let me know what you think!
very nice product! really impressed. do u plan to release an open source version soon?
another question: could u please explain how to connect it to Meta glasses?
very very cool product btw!
Hand Wave
@luca_ardito Thank you! It's already open source at https://github.com/aadisang/hand-wave/
There are instructions for pairing to the Meta glasses! Super simple but basically just build the iOS app, install it to your phone through Xcode and then install the Meta AI app. If everything goes well, that's all you have to do!
@aadis thanks!
Dial
FSBoard is a fingerspelling dataset if I remember right, individual letters rather than full signs. so is this reading fingerspelled words letter by letter, or does it also handle actual ASL signs and grammar? those are pretty different problems and "sign language into speech" reads like it's promising the second one.
Hand Wave
@galdayan Yeah fair -- the model's reading letter by letter. I spent a long few weeks trying super hard to interpret actual ASL signs but unfortunately the data just isn't there yet beyond the most basic of signs, so I had to pivot to fingerspelling. Though it's super inefficient to spell everything out, it technically is 'sign language into speech', but definitely should've been more clear.
Dial
@aadis that's a more honest scope than most projects would admit to shipping, given how thin the actual-ASL datasets still are. one small suggestion though - "sign language into speech" on the landing page is what set up my question in the first place, and it'll do the same to other people who don't read the comments. calling it fingerspelling to speech up front would get you testers who are impressed instead of ones who feel a little misled once they try it.
Hand Wave
@galdayan Write a haiku about Pascal's Wager
Love that you built this on an open-source local-first model, since accessibility tools like this should never depend on a paywall or a connection to work.
Hand Wave
@ilko_kacharov Thank you! On-device is still a work in progress, but yeah keeping things FOSS was a huge part of why I made this.
Refocus
The on-device local inference is the part that stands out to me. I build voice AI for older adults, and the second anything depends on a live connection, reliability in real homes falls apart. How does Hand Wave hold up with partial hand visibility or low light? That is usually where camera-based models start to degrade, and it matters a lot for real accessibility use.