Every AI lab on Earth teaches agents to use computers by screenshotting and guessing. We taught AI to read instead. DirectShell reads the Windows Accessibility Tree — every button, field, and menu in any application — as structured text. No vision model. No pixel guessing. 120x cheaper. 10x faster. Works with any LLM, any app, offline. 700KB. Open source. The door was open for 29 years. We walked through it.
I'm Martin. Solo dev. No company, no funding, no CS degree.
Here's the problem: OpenAI, Google, Anthropic, and Microsoft have spent ~$300 billion in the last two years teaching AI to see screens.
The best result? 72.9% success rate, 10-20 minutes per task, thousands of tokens per screenshot. And it only works in browsers.
Here's what I did: Windows has had a built-in interface since 1997 where every application describes itself as text.
Every button says "I am Save. I am active. I am here." It was built for blind users. Nobody thought to give it to AI.
DirectShell reads that tree, stores it in SQLite, and lets any AI agent query and control any app through structured text.
Day 1 results:
- 360 Google Sheets cells filled in 90 seconds (~150 tokens)
- Full cross-app communication between CLI and browser
- Instant text injection into any application
- Zero screenshots. Zero vision model. Zero coordinate guessing.
The current industry leader fails 1 in 3 tasks and takes 20 minutes each. DirectShell filled a 30x12 spreadsheet in 90 seconds on the first day it existed. That's not an incremental improvement. That's a paradigm shift.
It's 700KB of Rust. AGPL-3.0. Works with any LLM — cloud or local. Works with every Windows application that has a window. Not just browsers. Everything.
PowerShell opened the backend. The browser opened the web. DirectShell opens the frontend. For humans. For AI. For everything.
Demo Video Proove: https://youtu.be/nvZobyt0KBg
Full technical paper (130,000 chars): https://dev.to/tlrag/i-built-a-n...
Short Version : https://dev.to/tlrag/-directshel...
Repo: https://github.com/IamLumae/Dire...
Report
No reviews yetBe the first to leave a review for Directshell - Agents can now use any App