Airpoint - Touchless computing with hand tracking and AI agents
by•
Airpoint lets you control any computer with your hands without any extra hardware. Hand motion and gestures are translated in real time into cursor control, shortcuts, and app/games interactions. Built for anyone whose hands are busy or unavailable: crafters, gamers, builders, devs and more. Take advantage of VR/AR without headsets or extra hardware. The input layer also makes it possible for AI agents (OpenClaw) to autonomously control computers. SDK available to include in apps and games.
Replies
Airpoint
@marioandf Love the Minority Report vibes; gesture control without hardware sounds game-changing. Just a simple q; how does Airpoint handle gesture accuracy in real world scenarios like bright lighting or finger fatigue during long sessions, and what's your roadmap for multi-monitor or AR integration?
Airpoint
@swati_paliwal Great questions.
We tackled accuracy by combining algorithms with ML models trained with over 1M samples in different scenarios to predict with >98% precision the expected outcome at each frame.
For finger fatigue, we carefully designed the gestures to be as ergonomically as possible and are expected to work from a resting wrist position too. We currently do not recommend throwing away the mouse or keyboard yet 😅. In long sessions they are still useful. But when working from a distance or working on physical world tasks, it is hard to beat to just point at the monitor.
We already support multiple monitors! 🙌🏻
We are already exploring how could we integrate with AR, since our focus is still not adding extra hardware.
Thanks for the feedback
@marioandf this looks really amazing! Unfortunately I cannot try it because I do not have hands. I would love to collaborate with you on making one that works with head movements and facial gestures.
Airpoint
@hudsonperalta Hey great idea! Lets talk!
@marioandf email me anytime ryan @hudsonperalta .com
Product Hunt
I personally think this is insane and very promising. I've already started relying more on dictation, if I pair that with hand controls I don't really think we need to be sitting close to screens anymore.
Awesome job, Andrew! How did you get it so responsive?
Airpoint
@gabe Thanks for the comments Gabe! It was a real challenge considering most computers have single decent camera at 30Hz. Took a LOT of iterations through different detection methods and testing for real world responsiveness, but I think we got there and can keep improving with more data and feedback 😊.
Product Hunt
@marioandf I'm very excited to play with this over the weekend :) Congrats again and really nicely done getting it so fluid.
Airpoint
@gabe Thanks. Let me know if there are any suggestions 👍🏻
Product Hunt
Airpoint
@curiouskitty Great question and exactly the main challenge of the project.
First or all we decided to go with index finger tracking for cursor to reduce the need to move the whole hand to move the cursor. Even a resting hand moving index finger is tracked and translated to the cursor.
We decided to add many control layers to for user intent: Fingers pointing at the screen, Voice commands to Resume/Pause/Stop tracking and hand gestures or shortcuts to quickly toggle between them too.
Exactly hitting the right balance between speed, stability and false positives was hard and required a lot of testing and training of AI models. Most webcams are fixed at 30Hz so we do need smoothening, stabilizing and prediction algorithms for a good user experience. Of course that meant sacrificing small portions of precision and responsiveness but the key was to keep the tradeoffs under the human perceptible thresholds to keep the advantages. We kept most of this algorithms accessible in settings though for different user preferences.
However we keep using the latest data and feedback for continuous training of the AI/ML models in order to keep improving with each iteration.
Incredible
It's been a long time since I was blown away by a demo on a website.
Airpoint
@philip_alm_at_incredible Really? Thanks a lot!
Product Hunt
cc @kwindla who founded the company that Minority Report was based on! This is wild.
Airpoint
@kwindla @rajiv_ayyangar 🤯 I had no idea! That is wild. Just researched @kwindla and he is a genius! For sure an inspiration on the project.
Unbelievable. As an ADHD multi-tasker, I've been using clunky iOS apps to control my (M1 Mac Mini) computer from a distance. This removes a critical step of having ANOTHER device to remotely control your computer. I can't wait to see how your product development roadmap goes from here.
Airpoint
@the_people_person Yes, that is exactly where I came from too. Suddenly even the TV remote control seemed so outdated after using hand gestures.
NextBase
This is incredible! So so good. Love the idea of mapping the hand movements to not just the mouse but to also have shortcuts such as voice.
Airpoint
@imbhargav5 Thanks! That is the idea! Made them customizable to suit everyone´s needs!
Covalent
this is super sick! I think your UI for controls are super cool.
Airpoint
@hem_juvvaladinne1 Thanks! Took a lot of time to polish designing the gestures 😅
Product Hunt
What do you plan to do with this? Have you found any use cases? In other words, this looks like really cool technology, but what's the product here?
Airpoint
@rajiv_ayyangar I see a lot of use cases honestly. Desktop control at distance while working on physical tasks is only one thing. Since my main background is surgery I am finishing a surgery trainer with this in the following days. Also a lot of people have been testing it for gaming. Since I am just one person though I am planning of releasing it as an SDK for people to be creative and implement it in their apps.
Airpoint
Give your OpenClaw agents "hands" by connecting Airpoint to your OpenClaw setup 🦞 and let your agents control your computer (not just the browser!) 😉