Launching today
UI-Atlas
Map desktop interfaces for computer-use agents
15 followers
Map desktop interfaces for computer-use agents
15 followers
UI-Atlas is an open-source, offline-first toolkit for building evidence-backed maps of desktop interfaces. It captures surfaces, controls, states, actions, transitions, and lineage to observed evidence. The maps can support agents, testing, accessibility, automation, and security or governance systems that need to understand what is being acted on. UI-Atlas maps the interface; it does not enforce policy or execute workflows.
App in the Air 5.0
neat idea - how does this compare to computer use on codex/cc?
@bayka computer use as a product in codex/claude code is an end product packed into a harness. It can gather some of the information about the app like we do in runtime, but it lacks structured UI maps and lacks ability to ground target apps to such maps before doing any kind of actions on them.
UI Atlas is not a computer use agent harness, it's a tool for collecting and building UI maps for building more reliable computer use harnesses.
The easiest way to use such maps to improve computer use is to build your own computer use harness, collect UI maps using UI atlas, then build a grounding tech that grounds current observations to the UI map, and then feeds that data back to the computer use model inside your harness.
You can also use the recorded map data to train computer vision models like YOLO (in fact, we've experimented with such models back in January-February) to better identify controls of the target apps. Think of such model as UI Grounding Model.
Wrote an intro to the problem here:
https://x.com/danielko/status/2097594696193413448