Loading...
No model watches your display on its own. A screenshot reaches an AI only when software you installed captures one and sends it. Here is the full pipeline, what leaks into a capture, and which parts never leave your Mac.
Give Claude, Codex, or any AI assistant the ability to see your screen and automate any task. Runs locally on your Mac or PC.
No AI model watches your screen on its own. A model sees your screen when a program running on your machine captures an image and sends it, which happens because you installed that program and granted it permission.
That rules out most of what people picture when they ask the question. A chatbot in a browser tab has no route to your desktop. The page it runs in sits inside the browser sandbox, which blocks it from reading other windows, other tabs, or the framebuffer. Change the setup and the answer changes. Screen sharing tools, remote support software, and computer use agents exist to move pixels from your display into another process. Install one of those and the answer becomes yes, under conditions you set.
The pipeline decides the rest: what a capture contains, which parts stay on your Mac, which parts cross the network, and where the arrangement breaks in ways that catch people out.
A model cannot reach into your machine. It emits a tool call, and a process on your side decides what to do with it. The sequence below describes how OpenOwl handles a screenshot call on macOS.
| Stage | Where it runs | What exists at this point |
|---|---|---|
| Model emits a tool call | Model provider | A JSON request naming a tool and its arguments |
| Local server receives it | Your Mac | A function call, no pixels yet |
| Capture via mss | Your Mac | Raw framebuffer contents in memory |
| Crop, if you asked for one | Your Mac | Only the rectangle you specified |
| Downscale | Your Mac | An image capped at 1280 pixels wide |
| Encode as JPEG | Your Mac | Compressed bytes at quality 80 |
| Write to disk | Your Mac | A JPEG file under a rolling cap of 50 |
| Base64 encode | Your Mac | A text payload inside the tool result |
| Transport | Network | The payload travels to your model provider |
| Inference | Model provider | The image becomes tokens the model reads |
Two gates sit in that chain, and both belong to you. macOS blocks framebuffer access behind the Screen Recording permission, and it blocks synthetic input behind the Accessibility permission. OpenOwl requests both at startup, so the system dialogs appear before an agent begins work instead of interrupting it halfway through. A denied permission means the capture stage returns nothing.
Pay attention to the downscale step. OpenOwl resizes anything wider than 1280 pixels before transport, which means a 5K display arrives at the model at a fraction of its native resolution. The model can still read UI labels at that size. Fine print inside a document often does not survive the resize.
A screenshot captures the whole display. Everything visible on it goes into the frame, and people underestimate how much that is.
Consider a normal working screen. The menu bar shows your wifi network name, your battery state, and the icon of every background app you run. Tab titles in a browser window carry client names, document names, and internal project codenames. Your Slack sidebar lists unread direct messages by sender, and a terminal keeps scrollback that may hold environment variables, connection strings, and tokens you pasted while debugging. An email client with a preview pane renders message bodies before anyone opens a message.
Then there is the timing hazard. A notification banner that arrives between the model deciding to take a screenshot and the capture executing lands in the image, and message previews include sender and content by default. This failure mode is the worst of the set because it does not reproduce. Run the same automation ten times and the banner appears once, which makes it the kind of leak that survives testing and shows up in production.
Give password managers their own rule. An unlocked vault window on screen at capture time puts credentials into an image that then travels to a model provider. Lock it before an agent starts.
Screen capture and screen understanding are separate operations, and conflating them makes the privacy picture worse than it is. Several ways of reading the screen never produce an image at all.
| Operation | Where it executes | Does an image leave your Mac |
|---|---|---|
| Accessibility tree query | Your Mac, via AXUIElement | No |
| OCR via the Vision framework | Your Mac, on device | No |
| OCR via RapidOCR | Your Mac, via onnxruntime | No |
| Screenshot to a hosted model | Your Mac, then the network | Yes |
Of these, the accessibility tree covers the most ground. macOS exposes UI elements with names, roles, and bounding boxes through AXUIElement, so a tool can locate the Send button in an application and click it without encoding a single pixel. OCR fills the gap for applications with poor accessibility support, and it also runs on device. The Vision framework ships with macOS, and RapidOCR executes through onnxruntime in the same local process.
Do not read that as a clean win. The result of a local operation still travels back to the model as text. OCR that reads a field label reads whatever text sits in that region, and that text goes into the conversation. Local processing shrinks what leaves your machine without sealing it.
Anthropic's computer use documentation is direct about the risk, and the guidance is worth repeating. Run the agent inside a dedicated virtual machine or container with the fewest privileges it needs. Keep sensitive accounts and data out of that environment. Limit internet access to a domain allowlist. Require human confirmation before anything consequential.
The reasoning behind that guidance comes down to blast radius. A desktop agent operates against your logged-in sessions, your file system, and every application you left authenticated. Your browser cookies are its browser cookies. Your SSH keys sit where it can reach them.
Prompt injection adds a second problem. Content on screen is input to the model, and a model reading a web page cannot always separate page content from instructions aimed at it. A page carrying text that looks like a command becomes an attack surface once an agent can see it and click. The defense is structural. Run agents under a dedicated macOS user account, keep credentials out of that account, and treat any session where the agent browses untrusted pages as compromised.
Read the trade-offs in more depth in our breakdown of how computer use agents work and where they fail.
You have three levers, and they operate at different layers.
The first is the capture call itself. screenshot accepts region_x, region_y, region_w, and region_h to crop to a rectangle, plus a monitor argument that picks a display. Pass monitor=0 to grab every display combined, or monitor=1 for the primary one.
screenshot(
monitor = 1,
region_x = 0,
region_y = 0,
region_w = 900,
region_h = 600
)
Be precise about what this does. The capture grabs the monitor first, then crops. The full frame exists in local memory for the duration of one function call, and OpenOwl encodes and transports only the cropped result. Nothing outside the rectangle reaches the network. If your threat model concerns what a model provider receives, cropping works. If it concerns what touches memory on your own machine, cropping is not the control you want.
Display layout gives you the second lever. Set monitor to one screen, keep sensitive applications on another, and you get a physical separation that survives mistakes in tool arguments.
That leaves the operating system. macOS Screen Recording permission is all or nothing per application, so you cannot tell it to show one app and hide another. What you can do is revoke the grant in System Settings under Privacy & Security, and inspect exactly which applications hold it. For anything sensitive, a separate macOS user account dedicated to automation is the strongest available boundary, because a different account means a different keychain, a different browser profile, and a different set of logged-in sessions.
OpenOwl runs as a local process on macOS. The binary sits on your disk, captures happen in that process, and OpenOwl writes screenshots to local storage under a rolling cap of 50 files, deleting the oldest as new ones arrive. There are 42 MCP tools in total, and the ones that read the accessibility tree or run OCR on device complete work without producing an image at all.
OpenOwl makes two network calls, and both belong in the open in a post about screen privacy. It validates licenses against a remote endpoint, and that call carries an API key, a machine ID, and a usage count, with no screen content. Separately, when a hosted model such as Claude requests a screenshot, that image travels to the model provider as part of your conversation, under whatever data policy governs your account with them. OpenOwl captures the screen. What happens after that depends on the AI client you point it at.
The free tier runs 50 tool calls per day, which is enough to watch the pipeline behave on your own machine before you decide whether the trade-offs suit your work. More on who builds this and why on our about page.
No. A chatbot in a browser tab runs inside the browser sandbox, which blocks it from reading other tabs, other windows, or the framebuffer. Screen access requires a separate program running outside the browser with an operating system permission you granted. On macOS that means an entry under Privacy & Security > Screen Recording, which you can inspect and revoke at any time.
OpenOwl does not send screen content to OpenOwl servers. It writes captures to local disk and returns them to whichever AI client called the tool. If that client is a hosted model such as Claude, the image travels to that model provider as part of your existing conversation. License validation is a separate call that carries an API key, a machine ID, and a usage count, and no pixels.
You can narrow what gets sent. Hiding one app from a capture sits outside what macOS offers. The region_x, region_y, region_w, and region_h parameters crop to a rectangle before transport, and monitor limits capture to one display, so sensitive work can live on a second screen. Screen Recording permission is all or nothing per application, which makes a separate macOS user account for automation your real boundary.
On your Mac. OpenOwl prefers the built-in Vision framework on macOS and falls back to RapidOCR through onnxruntime. Both execute in the local process. Finding and clicking text on screen involves no image upload, though the text the tool returns does go back to the model as part of the result.
The image includes it. A banner that lands between the model deciding to take a screenshot and the capture executing appears in the frame, and message previews often carry sender names and content. This failure depends on timing, so it will not reproduce on demand. Turning on Do Not Disturb before an automation run removes the whole class of problem.