agent-desktop: Let AI Agents Use Your Computer's Apps, Without Screenshots

agent-desktop is a computer-use CLI and C library from Vercel Labs that lets AI agents read and operate desktop apps through the operating system's accessibility tree. Instead of taking a screenshot and guessing where the buttons are, the agent gets a structured snapshot: every element it can act on, with a role, a name, and a ref it can click, type into, or scroll. The agent acts on a ref, takes a new snapshot, and checks that the action worked, with no pixel matching anywhere in the loop.

It runs on macOS, Windows and Linux, is written in Rust, and is licensed under Apache 2.0. Install it from npm and it ships native binaries and platform helpers for x64 and ARM64. When we checked on November 3, 2026 it had 11,728 stars and 796 forks, and the repository notes it gained about 3,800 stars in a single day, making it the day's fastest-growing repo.

The design keeps the loop inside your agent. Every CLI call runs in its own process, and the agent's own code decides what to do next, which means agent-desktop works with any harness rather than locking you into one vendor's agent. By default it is headless: ref actions go through the accessibility API without moving the pointer or taking focus, and headless calls prohibit foreground activation and synthesized input. The global --headed flag allows physical input when you actually want it.

The core loop is three commands: observe, act, observe again. Any app name from agent-desktop list-apps works; Finder is the macOS example in the README. Each command prints one JSON envelope with ok and either data or error, which makes the output easy to parse in an agent loop, and pasting a snapshot into a JSON formatter makes the tree readable while you are learning the format.

agent-desktop snapshot --app Finder -i      # interactive elements, each with a ref
agent-desktop click @s8f3k2p9:e8            # act on a ref from that snapshot
agent-desktop snapshot --app Finder -i      # observe again to verify

A snapshot returns JSON with a ref for every element the agent can act on, plus the app, window, snapshot id and ref count. Refs belong to their snapshot: a bare @e8 is rejected, and when the UI changes an action returns STALE_REF or AMBIGUOUS_TARGET, which is the signal to take a fresh snapshot and use its refs. When several windows match, the snapshot lists the candidate window ids and you pass one with --window-id.

Dense apps need a map first. snapshot --skeleton --app Slack -i --compact gives the outline, then snapshot --root @s8f3k2p9:e3 -i --compact drills into one region, so the agent never has to swallow a whole application's tree in one context window.

The README lists 60 command names, 56 of them operational. The observe group covers snapshot, find, get, is, screenshot and list-surfaces. The act group works on refs: click, double-click, triple-click, right-click, type, set-value, clear, focus, select, toggle, check, uncheck, expand, collapse, scroll and scroll-to.

Beyond that there are keyboard and mouse commands (press, hover, drag, mouse-move, mouse-click, mouse-wheel), app and window management (launch, close-app, list-apps, list-windows, list-displays, focus-window, resize-window, move-window, minimize, maximize, restore, open-system-surface), notifications (list-notifications, dismiss-notification, dismiss-all-notifications, notification-action), clipboard (clipboard-get, clipboard-set, clipboard-clear), plus wait, session commands (session, trace, cursor-overlay) and system commands (status, permissions, version, batch, skills).

Four commands refuse to run until a daemon can release a held key or button: key-down, key-up, mouse-down and mouse-up. That is a safety rail, not a missing feature: a held key with no release path would leave the machine in a stuck state.

agent-desktop ships its own documentation, matched to its version, so an agent never has to guess at flags or semantics. agent-desktop skills get desktop returns the loop, refs, verification and retry rules, and agent-desktop skills get platform prints what differs on the OS you are running on. Every command also has --help.

Install the skill into an agent harness from skills.sh, or add a short block to your AGENTS.md telling the agent to fetch the desktop and platform skills before its first use and follow them. The error contract is part of the same story: an error carries a code, a disposition and usually a suggestion, and you retry only when error.disposition.retry is safe. That single rule stops the classic agent failure mode of retrying an action that already partially happened.

For programs rather than CLIs, each release also ships libagent_desktop_ffi for macOS, Linux and Windows, so you can call the tool in process instead of starting the CLI for each command. agent-desktop skills get ffi covers the entry points, memory ownership and threading rules.

All three platforms get the accessibility tree, click and type, screenshots, clipboard and app and window management, but the details differ. macOS needs Accessibility permission, granted through agent-desktop permissions --guide. Windows needs no grant for apps at your terminal's integrity level. Linux needs the user's graphical session with at-spi2-core.

The edges are where the README is honest. On Linux, mouse input works on X11 with --headed, screenshots use X11 windows or the Wayland portal, clipboard on Wayland needs wl-clipboard, and window control is X11 only. Notifications work on macOS, Windows and GNOME Shell. Shell surfaces such as the Dock, menu bar extras, GNOME tray and Action Center are reachable through open-system-surface.

Official Homebrew and Scoop packages are being submitted but are not yet in the default catalogs, and the repository no longer maintains a custom Homebrew tap or Scoop bucket. For direct execution without a JavaScript launcher, download the matching native archive from GitHub Releases and keep its helper beside the executable.

The README's privacy warning deserves to be read before you hand this to an agent. Snapshots, clipboard reads, screenshots, command output and debug artifacts can contain private desktop content, so keep them out of public logs and uploads. That is not a hypothetical: a snapshot of a mail client or a password manager is a dump of your screen in structured form.

One optional helper is separate from the native CLI and easy to confuse with it. The JEV helper sends screen descriptions to a configured model provider; it is not part of the CLI's local loop, so if you care about keeping screen content local, do not enable it without reading its docs. The privacy and data handling notes live in the repository's FAQ.

Two smaller gotchas: in PowerShell, quote every ref (a bare @token is the splatting operator), and remember that refs go stale. An agent that caches refs across snapshots will spend its budget on STALE_REF errors instead of doing work.

Install it globally with npm, which also works under Bun, Yarn and pnpm:

npm install -g agent-desktop
agent-desktop version

On macOS, run agent-desktop permissions --guide first and grant Accessibility access, then list what is running and take your first snapshot:

agent-desktop list-apps
agent-desktop snapshot --app Finder -i

Pick a ref from that snapshot, act on it, and snapshot again to verify, exactly as the quick start shows. If the action fails, read the error envelope: the code and suggestion tell you whether to retry, re-snapshot, or disambiguate a window. The repository's docs folder covers installation problems, the JSON output contract, concepts such as refs and sessions, and a solutions directory of past bugs and lessons.