Loading...
The accessibility tree is the labeled hierarchy macOS publishes for screen readers, and it gives an AI agent exact element coordinates instead of pixel guesses. A look at the token math against screenshots, the cases where the tree returns nothing, and how OpenOwl queries it through AXUIElement.
Give Claude, Codex, or any AI assistant the ability to see your screen and automate any task. Runs locally on your Mac or PC.
An accessibility tree is the structured description of a user interface that macOS publishes for assistive software, and an AI agent reads it the same way VoiceOver does: as a labeled hierarchy of buttons, text fields, menus, and rows, where each node carries a name, a role, and screen coordinates. An AI agent that queries the accessibility tree asks the operating system where a control sits rather than guessing pixel positions from a capture. Instead of estimating where the Send button sits inside a flat capture of a window, the agent requests an element named "Send" with role Button and gets back the same rect the OS uses for its own hit testing.
macOS exposes this through the AXUIElement API, where every role carries an AX prefix: AXButton, AXTextField, AXMenuItem. Windows has UI Automation and Linux has AT-SPI. The APIs differ in shape and share one premise, which is that each app describes its own controls for software outside the app to read.
A single node contains attributes like these:
| Attribute | Example value | Purpose |
|---|---|---|
| AXRole | AXButton | control type |
| AXTitle | "Send" | visible label |
| AXValue | "hello@example.com" | current contents |
| AXPosition | (412, 880) | top-left in screen coordinates |
| AXSize | 72 x 28 | width and height |
| AXChildren | [...] | nested elements |
Position and size matter most here. A node with coordinates attached lets your agent click a control straight from tree data, skipping both the OCR pass and the vision model.
VoiceOver walks this tree. When a blind user tabs through a form, VoiceOver calls the same AXUIElement functions an MCP server calls. It reads AXTitle out loud, announces AXRole so the user knows whether focus landed on a button or a checkbox, and reports AXValue so they hear what a field contains.
Two consequences follow for anyone building agents. Tree coverage tracks accessibility investment. Apps built with compliance in mind, including Apple's own apps and anything audited under Section 508 or the European Accessibility Act, carry dense trees with good labels. Apps whose developers skipped that work return a window frame and little else.
The labels exist for humans to read. AXTitle contains "Send" rather than btn_submit_primary_v2. A model handed a plain-language instruction can match against those labels with no selector map to maintain, which is why computer use agents that drive desktop apps target the tree first.
A screenshot-driven agent sends a capture to a vision model, receives an x/y guess, and clicks there. Four failure modes recur.
Downscaling. OpenOwl shrinks every screenshot to 1280px wide at JPEG quality 80 before the model sees it. Any coordinate read off that capture belongs to the shrunken frame rather than your display, so a click at those numbers lands in the wrong place. Use the element coordinates the tools return, never numbers you eyeball from a capture.
Occlusion. A tooltip or notification banner sitting over the target still looks clickable in a flat capture.
Ambiguity. Three buttons labeled "Delete" in a list look identical to a vision model. Each one has a different parent in the tree, so your agent can pick the row it wants.
Missing state. A disabled button drawn in low contrast looks the same as an enabled one, and the tree carries the enabled flag as data.
Reading coordinates from the tree removes all four problems, because the OS hands back the rect it already uses for hit testing. A click at the center of that rect lands inside the control.
One caveat about tree accuracy: a tree contains what the developer declared. A developer can label a control "Save" and wire it to something else, and web apps with stale ARIA attributes drift out of sync with what a user sees on screen.
Payload size separates the two approaches. OpenOwl's downscale step (1280px wide, JPEG quality 80) takes a capture from about 2MB of PNG down to about 119KB of JPEG, a 17x reduction, and that is the compressed payload before any tokenizer touches it. Token counts vary by provider and tiling scheme, so measure your own at your display resolution before you budget for a long run.
A tree query costs whatever its text costs. A query filtered to buttons in the frontmost window returns lines like:
[0] Button "Compose" (24,140) 96x32
[1] Button "Send" (412,880) 72x28
[2] Button "Discard" (496,880) 80x28
Three lines of text carry the same content you can act on that a vision model has to infer from a megapixel of pixel data.
The gap compounds inside an agent loop. A twenty-step task that captures a screenshot per step drags twenty captures through context by the end, while text output leaves room for the full history of what the agent did. That history matters when step 18 fails and the model needs to reason about step 3.
OpenOwl applies the same logic to clicking. Both click and click_element skip the post-action screenshot unless the action failed or moved the app to a new screen. A comment in OpenOwl's source estimates this saves around $0.02 per call in vision tokens, an estimate rather than a measured figure.
Three levers control what a tree query costs you:
role="Button" returns the subset you can act on.list_elements also truncates at 100 elements and drops unnamed Pane, Group, and Custom nodes.ui_fingerprint hashes the top elements and their positions into a short string. Compare it between actions to catch a modal opening or a tab switching without pulling the whole tree again.Knowing the failure modes up front saves you a debugging session later.
This is the largest gap. On macOS the accessibility tree covers browser chrome, meaning tabs, the address bar, and the toolbar. Page content inside Chrome or Safari stays invisible to find_element and list_elements. Gmail, Notion, Linear, and every other web app sit outside tree-based targeting, so you drive them with OCR, keyboard navigation, or the browser's own tooling.
Electron disables renderer accessibility by default. Relaunch the app with --force-renderer-accessibility and its ARIA-mapped elements show up in the tree. Without the flag you get a window and nothing inside it.
Swing stays invisible to the platform accessibility layer. Enable the Java Access Bridge with jabswitch -enable, or fall back to OCR.
Figma's canvas, the Monaco editor inside VS Code, charts, maps, and games paint pixels without publishing nodes. No tree exists for your agent to walk.
A node can exist with correct coordinates and an empty title, which is the normal state for icon-only toolbar buttons. OpenOwl drops unnamed Pane, Group, and Custom nodes from list_elements output because they add bulk without adding targets. An unnamed button still leaves you with no way to identify it by name.
Web text fields often ignore synthetic mouse clicks. Tab into them instead of clicking, then write text with clipboard(action="write") followed by send_keys("cmd+v").
A tree query is synchronous IPC into another process. When the target app blocks its main thread on a long render or a modal spinner, the AXUIElement call blocks along with it. OpenOwl puts a timeout on every tool for this reason. find_element and list_elements stop at 10 seconds, click_element at 15 because it may run OCR first, and ui_fingerprint at 5. Each one returns a message about an unresponsive tree rather than hanging your session.
Working agents cascade through methods. OpenOwl's click_element runs three tiers in order:
VNRecognizeTextRequest at accurate recognition level), with normal and inverted passes on a two-thread pool so dark backgrounds still read. This tier is what makes web apps clickable at all.The screenshot tool blends both sources into one response. Every capture appends on-screen text with coordinates (capped at 50 phrases) plus an accessibility element summary (capped at 30), so a single call hands the model OCR text and tree data together.
click_text adds a correction loop of its own. When a center click produces no visual change, it retries at six offsets: below, below far, above, above far, right, and left. That handles labels sitting next to their control rather than on top of it.
Keyboard navigation sits underneath everything else, and Tab plus Enter work when every visual method fails. Build one operational habit: log which tier fired. A screen that resolved on tier 1 last month and needs tier 3 today points at an app update that dropped its labels.
OpenOwl is an MCP server for macOS. Six of its 42 tools work on the accessibility tree:
| Tool | Behavior |
|---|---|
find_element | name and role lookup, returns role, name, x, y, width, height |
click_element | primary click tool, runs the three-tier cascade |
list_elements | enumerates a window, filters by role, caps at 100 elements |
get_focused_element | reads AXFocusedUIElement, returns what holds keyboard focus |
smart_find | tree first, OCR fallback, reports which method matched |
ui_fingerprint | hash of top elements and positions for change detection |
A few details change how you write prompts against these.
Role names use a Windows-style vocabulary at the surface and AX underneath. Pass role="Button" and OpenOwl maps it to AXButton before walking. Edit maps to AXTextField, ComboBox maps to AXPopUpButton, and Tree maps to AXOutline. That vocabulary is what most models produce when you ask them to name a control type.
Depth defaults to 5. Setting a role filter makes the walker ignore the cap and traverse the full tree, since a Button nine levels deep is still a Button and the filter keeps the output small.
Names fall back from AXTitle to AXValue. Text fields often carry no title and a value, so a field containing "hello@example.com" is findable by its contents when it has no label.
click_element returns the focused element after the click. Read that before you type. A click landing on a container rather than the field inside it is the most common silent failure in form automation, and every keystroke after it goes nowhere.
Two macOS permissions gate all of this: Accessibility, which covers input and every AXUIElement query, and Screen Recording, which covers screenshots and window titles. OpenOwl runs a preflight at startup so both dialogs appear at launch rather than mid-task, and you grant them in System Settings > Privacy & Security. Restart the server afterward, because macOS does not re-evaluate grants for a process that is already running. Screenshots that come back black point at a missing or stale Screen Recording grant.
Next step: install OpenOwl with the quick setup guide for Claude Code and Codex, then run list_elements against one app you plan to automate and read the raw output. The free tier gives you 50 tool calls a day, enough to walk several windows and learn which of your apps expose a usable tree and which ones need an OCR workflow instead.
No. The DOM is a browser's internal model of one page, while the accessibility tree is an OS-level structure spanning every running app. A browser builds its own accessibility tree from the DOM, mapping ARIA roles and semantic HTML into platform roles, then exposes part of that to the OS. On macOS the part OpenOwl reaches covers browser chrome, so page content stays outside tree queries.
On macOS, yes. You grant Accessibility permission in System Settings > Privacy & Security, and AXUIElement calls return errors until you do. Screen Recording is a separate grant covering screenshots and window titles. Restart the server after granting either one, because macOS does not re-evaluate permissions for a running process.
It can do both. The AX API supports actions such as AXPress on a button and attribute writes such as setting AXMinimized on a window. OpenOwl uses AX actions for window management and synthetic clicks at tree coordinates for controls, since a real click reproduces the hover and focus behavior an app may depend on.
The ratio depends on your display resolution, your provider's image tiling, and how much you filter the query. The direction holds across every setup. A role-filtered query returns a few lines of text where a screenshot returns a full capture, and OpenOwl's downscale step alone cuts payload about 17x, from about 2MB of PNG to about 119KB of JPEG. Measure the token side yourself against your own provider and display.
Fall through the cascade. Try a fuzzy match, then OCR through click_text or find_text, then keyboard navigation with Tab and Enter. When list_elements returns a window frame with no children, treat that app as opaque and build the workflow around OCR and keyboard from the start.