Launched this week
Cursor Crane
Control your Mac at the pace of your keyboard
93 followers
Control your Mac at the pace of your keyboard
93 followers
Most hint-based Mac tools focus on UI elements. Cursor Crane extends the same interaction model to windows, multi-window workflows, and text editing. Its dedicated modes use predictable rather than arbitrary hints. Enter command sequences to switch modes, perform actions, or run scripts. It also makes awkward keyboard-only tasks practical: selecting text across lines, editing focused fields in an external editor with Text Area Portal, and copying text or capturing UI through Element Menu.
Abnormal Mouse
Predictable rather than arbitrary hints is the whole thing, and I am glad it is in the first line instead of buried in a feature list.
Arbitrary hints mean you read the screen every single time. Predictable ones mean the sequence moves into your fingers after about a week, and that is the difference between a tool people demo and a tool people keep. Most of this category optimises the first press. You have optimised the hundredth.
Text Area Portal is the one I did not know I wanted. I write long text into browser textareas constantly and it is miserable, and every workaround I have tried means composing elsewhere and pasting back, which loses the field state and half the time the formatting.
Question rather than a critique: does the hinting read the accessibility tree, or something else? Apps with missing or lazy accessibility labels are where everything in this category quietly falls over, and I would rather know upfront which ones you already know are rough than discover it myself on a Tuesday.
Abnormal Mouse
@rabnoor_s Thanks for the thoughtful comment, and sorry for the late reply — I just woke up.
The hint system is primarily generated from the macOS Accessibility Tree. It uses information from the element itself and its children to generate hints.
When an app does not expose a usable Accessibility Tree, Cursor Crane falls back to an OCR-based mode instead.
Abnormal Mouse
the Text Area Portal idea is genuinely clever, basically turning any input field into something you can edit in your actual editor instead of fighting with cursor placement. feels like a really thoughtful gap to fill.
Abnormal Mouse
@observability Thanks! Editing text fields in VS Code is indeed much more comfortable, especially for longer or more complex text.
Besides Text Area Portal, Input Mode also provides a set of keyboard shortcuts to avoid fighting with cursor placement. You can navigate and select text by searching for content, making it much easier to edit text even in fields that normally have limited keyboard support.
This would be quite useful for humans as well as computer use agents. Computer use relies on taking a screenshot and sending it to an LLM. In this case, an agent does not have to rely on mouse movement to take an action on the screen, as it can simply use the keybindings shown in the screenshot.
@devin_ersoy Devin, this is the sharpest point in the thread and it matches exactly what breaks in practice.
The screenshot loop fails in a specific way. The model reads an image, picks a pixel, clicks. Everything after that is a guess. The layout shifted between the screenshot and the click, or the page re-rendered, or the coordinate space did not match the device pixel ratio. You get a click that lands somewhere plausible and wrong, and the agent has no way to know it missed. It just carries on.
Hint labels remove the guess. The target stops being a coordinate and becomes a token, and a token either exists or it does not. That is verifiable in a way a pixel never is.
The version I would want on top of what you described is the hint map being queryable rather than only visible, so an agent can ask what is on screen instead of reading labels back out of an image. Same idea, one less lossy step.
ShX, is there any scripting hook that exposes the current hint map?
Abnormal Mouse
@devin_ersoy Thanks for the thoughtful comment, and sorry for the late reply — I just woke up.
From what I’ve observed, ChatGPT’s computer use already combines screenshots with accessibility information. As for whether Cursor Crane’s interaction model can be integrated into AI agents, I’m still exploring the possibilities.
If you have any ideas or use cases in mind, I’d love to hear them.
Abnormal Mouse