loomcode: any-model AI agent for your terminal, now experimenting with screen control

Hey PH 👋 I'm Toshal, and I just open-sourced loomcode — an MIT-licensed AI coding agent that lives in your terminal instead of a browser tab.

The idea started from being tired of switching between chat windows and my editor. So loomcode plugs into whatever model you already pay for — Anthropic, OpenAI, Gemini, OpenRouter, NVIDIA NIM (free tier), or local Ollama — and you can switch models mid-session with /models. No lock-in, your own keys.

The part I'm most unsure about (and would love feedback on) is Loom Computer, a beta feature where the agent can actually see your screen and control it — clicking, typing, switching windows — through one floating pill. I added a panic hotkey, password-manager blocking, and action caps, but I know giving an agent "hands" on a real desktop is a trust question, not just a technical one.

Repo: https://github.com/toshalkumbhar8979-design/loomcode
Install: npm install -g loom-agent
Beta: https://github.com/toshalkumbhar8979-design/loomcode-beta

Curious what this community thinks: for terminal-based coding agents, What's the first thing you'd try to break if an AI agent had full control of your desktop?, Non-vision models get a text map of the screen instead of a screenshot — has anyone here worked with that kind of 'screen as text' approach?