Loading...
RPA replays selectors recorded at design time. AI desktop automation reads the accessibility tree and runs OCR at runtime. A look at how each one fails, what maintenance costs, and which workloads should stay on UiPath or Automation Anywhere.
Give Claude, Codex, or any AI assistant the ability to see your screen and automate any task. Runs locally on your Mac or PC.
AI desktop automation and RPA both drive applications that expose no API, and they differ in how each one finds a target on screen. RPA replays selectors captured at design time (control IDs, XPath-style paths, image templates), while an AI agent queries the accessibility tree and runs OCR at runtime, then picks the target from what it reads. That difference decides which approach throws an exception the morning after a vendor ships a UI update, and which one costs you more per year to keep running.
If you run UiPath or Automation Anywhere today, ask a narrow question rather than a philosophical one. Which of your bots break most often, what changed on screen when they broke, and would a runtime-vision approach have survived that specific change? Part of your fleet will answer yes. A large part will answer no.
An RPA recorder watches you perform a task once and freezes what it saw into a selector. The selector is a path into the application's UI object model, and it pins several attributes at once:
<wnd app='invoiceapp.exe' cls='WindowsForms10.Window.8.app.0.141b42a_r6_ad1' title='Invoice Entry' />
<ctrl automationid='grdLines' />
<ctrl role='data item' name='Row 3' idx='3' />
The vendor's next release can invalidate any attribute in that block. The cls value carries a build-specific hash that regenerates when the vendor recompiles. The title breaks when someone renames the screen, and idx='3' breaks the moment a row gets inserted above it. Pin fewer attributes and the selector matches the wrong control. Pin more and it stops matching at all.
The failure modes fall into a short list:
idx, row number, tab order) breaks when the data changes shape.That last one matters for triage. From a selector-not-found error you learn that the bot stopped and nothing else. You open the app and reproduce the run by hand to find out whether the UI changed, the network lagged, or a modal stole focus.
An AI desktop automation agent skips the recording step. At runtime it asks the operating system what is on screen, gets back a tree of elements with roles and names and coordinates, and hands that to the model along with a screenshot. OpenOwl ships 42 tools across 14 modules for this work, and the primary click path (click_element) runs a three-tier cascade: accessibility tree first, then fuzzy name matching, then OCR.
On macOS the OCR layer uses the Apple Vision framework (VNRecognizeTextRequest at accurate recognition level), with normal and inverted passes running in parallel on a two-thread pool so dark backgrounds still resolve. Screenshots downscale to 1280px wide at JPEG quality 80. The OpenOwl authors put that near a 17x payload reduction in a source comment (119KB JPEG against about 2MB PNG).
The two approaches fail in different ways.
| Change on screen | Selector-based bot | Accessibility + OCR agent |
|---|---|---|
| Button label changes from "Submit" to "Send" | Hard fail if the selector pins the name | Finds it when the model accepts the new label |
| Button moves to another toolbar | Hard fail on index or path | Finds it by name at the new coordinates |
| New build regenerates class names | Hard fail | Unaffected, since names and roles hold |
| Theme switches to dark mode | Image-template steps fail | Inverted OCR pass handles it |
| Display DPI or resolution changes | Image templates and fixed coordinates fail | Coordinates come from the tree at runtime |
| An unexpected modal appears first | Hard fail, or a click lands on the modal | Agent reads the modal and can dismiss it |
| Form field order changes | Hard fail on tab-order steps | Finds each field by label |
Read the right column with suspicion. A selector-based bot fails hard and stops. An agent that reads the screen can pick the wrong element and keep going, which turns a caught exception into a silent data error. OpenOwl mitigates this at the tool level: click captures a 400x400 region before and after, reports whether pixels changed, returns the element that took focus, and warns when the foreground window changed. click_text retries at six offsets when the center click produces no visual change. Those checks catch a good share of mistargets, and they do not catch all of them.
Neither vendor publishes a number you can use here, so reason about where the hours go instead.
Selector maintenance is reactive and it clusters. A vendor ships an update, every bot that touches that app breaks in the same week, and one team rebuilds selectors under a deadline. The work is cheap per incident and it arrives in spikes you cannot schedule.
You spend agent maintenance hours somewhere else. Prompts and workflow definitions need tuning when a screen changes shape enough to confuse the model, and you carry a per-run inference cost that a licensed bot does not have. A single OpenOwl click includes a forced 0.5s sleep, two 400x400 region captures, and an accessibility query, so it is not a cheap call. The click and click_element tools omit the screenshot from the response unless the action failed or caused navigation, which the authors estimate saves about $0.02 per call in vision tokens. Treat that as their estimate rather than a measurement.
For a like-for-like cost comparison on your own workload, run one rebuilt bot and one selector bot side by side over a full release cycle and count the incidents in each. Anyone quoting you a percentage without running your screens is guessing.
Plenty of workloads should stay on UiPath or Automation Anywhere, and pretending otherwise wastes your time.
The workloads worth moving are the brittle tail: the bots that break several times a quarter, run at low volume, and consume a disproportionate share of your maintenance budget.
Both approaches exist because the app has no API. Neither one escapes the limits of what the desktop exposes, and OpenOwl has specific gaps worth knowing before you plan a migration:
--force-renderer-accessibility or the tree comes back empty.jabswitch -enable or fall back to OCR.find_element and list_elements see browser chrome (tabs, address bar, toolbar) and nothing inside the page. Use OCR or the browser's own tooling for page content.find_text matches text in both. Scope with window_title or capture a region.An RPA platform with a native connector for that same legacy app often exposes more structure than the accessibility tree does. Check the connector list before you assume the agent approach wins.
RPA platforms carry a decade of enterprise audit tooling: signed run logs, video capture, approval workflows, separation of duties. An MCP server does not replace that, and you should not tell your risk team it does.
What OpenOwl records is narrower, and worth stating without inflation. The server logs validation calls and daily usage per API key, and it identifies machines by a SHA-256 hash of MAC address plus hostname, truncated to 32 hex characters. It keeps a rolling window of 50 screenshot files and evicts the oldest first. The license cache holds for 24 hours so the server keeps working offline, and usage checks fail open on a network error, which is a deliberate availability tradeoff your security reviewer will ask about.
For repeatability, record_workflow and replay_workflow_tool capture a session and play it back, which gives you a deterministic path for the steps that need one. Use the model for the parts that change and a recorded workflow for the parts an auditor will ask to see.
Nobody should rip out a working RPA estate. Run both and let the failure data decide.
list_elements to see what the app exposes before you write any automation. If the tree comes back empty, you fall back to OCR, so plan for lower reliability there.Getting started on macOS takes a few commands:
npm install -g openowl
mkdir -p ~/.openowl
echo "owl-xxxx-xxxx-xxxx" > ~/.openowl/api.key
claude mcp add owl --transport stdio -s user -- owl
owl --version
Grant Accessibility and Screen Recording in System Settings > Privacy & Security, then restart the server. macOS does not re-evaluate permission grants for a process that is already running, so the restart is mandatory. If screenshots come back black, check for a missing or stale Screen Recording grant.
The free tier covers 50 tool calls per day, which is enough to rebuild one bot and watch it run. The OpenOwl pricing tiers list daily limits and rate caps per plan. To hand the first migration to someone who has done it before, the done-for-you setup service builds and delivers a working workflow.
No. It replaces the brittle tail of an RPA estate, meaning the low-volume bots that break whenever a vendor ships a UI update. High-volume unattended transaction processing stays on RPA, and so does anything running on Windows, Citrix, or mainframe emulators.
OpenOwl's click_element falls through fuzzy matching to OCR, so the agent keeps working from what it reads on screen. Reliability drops on that path. Before you commit, run list_elements against the target app and see what comes back. Electron apps need --force-renderer-accessibility and Java Swing needs jabswitch -enable.
It runs as a stdio MCP server driven by an MCP client, so a client session has to be running. macOS also requires Accessibility and Screen Recording permissions on an active session, which rules out a locked or headless machine. For unattended overnight batches, RPA orchestration remains the better fit.
No. OpenOwl ships for macOS only, on both Apple Silicon and Intel Macs. Windows code exists in the Python source, but the project ships no Windows binary.
Rebuild one bot on the free tier (50 tool calls per day) and count the calls a single run consumes. Multiply by your run frequency, then compare against the maintenance hours that bot costs you today. The pricing page lists per-tier daily call limits.
Pull your RPA failure log, find the bot that broke most times last quarter, and check whether those breaks were selector changes. If they were, rebuild that one bot against the accessibility tree on the free tier, then run both versions through the vendor's next release. You will trust that comparison more than any vendor benchmark.