The Mac Mini in my office runs an agent that writes code, publishes articles, and generates images while I'm asleep. Twice in the last month it stopped doing any of that because a small gray box appeared on a screen nobody was looking at.
macOS permission dialogs. "node" would like to control this computer using accessibility features. Allow or Don't Allow. The dialog is modal, it blocks the thing that triggered it, and it sits there patiently forever. My agent has full control of the machine, right up until the operating system asks it a question. Then it waits, and I find out six hours later when I notice a job never finished.
So last week I built a watcher for it. The build took an evening and taught me more about the boundary between an automated system and its OS than anything else I've done this year.
Phase one: just tell me
The first version did nothing but look and report. Every five minutes the worker takes a picture of the console display, scans for anything that looks like a permission prompt, and if it finds one, sends a push to my phone with a screenshot link and writes an audit row. Detect and notify. No clicking.
I wanted to use the accessibility tree for detection, because that's the structured, correct way to read a UI: real elements with real labels and real coordinates, no guessing. It returned zero elements. Not an error, not a permission failure, just an empty array.
That turns out to be the rule, not a bug. Screen-mode accessibility queries return nothing for other applications' windows. The AX tree is scoped to what you own, and my worker owns none of the windows it cares about. The one thing on the machine I most needed to read was structurally invisible to the API designed for reading things.
OCR reads it fine. The Vision framework doesn't care who owns the window, because it isn't reading a window, it's reading pixels. So the watcher works the way a person does: it looks at the screen and reads the words. It matches on strict signatures ("Don't Allow", "Always Allow", "Allow Once", "Open System Settings", or the Allow and Deny pair together). A lone "Allow" or "OK" never fires, because those strings show up in ordinary app chrome all day.
The first real bug was the deduplication. I fingerprint each dialog so the same one doesn't re-notify every five minutes, and I built the fingerprint from the button labels plus the surrounding text. The surrounding text was the whole screen's OCR, which includes the menu bar clock. The clock changes every minute. Every tick produced a new fingerprint, so every tick was a brand new dialog, so my phone buzzed every five minutes with the same popup. The fix is dumb and works: only count text within 700 pixels of a trigger button as part of the dialog. The fingerprint went stable, and as a bonus the notification body now quotes what the dialog actually says instead of the clock and my notification center.
Phase two: what if it just clicked it
I showed it to myself working, and immediately asked the obvious question: we can't just have it click Allow?
The honest answer is that this is the part where you should slow down. macOS permission prompts exist for exactly one reason, which is to put a human in the loop before software gets a capability it didn't already have. Writing a robot that clicks them is not a clever automation. It's deliberately dismantling a security control, on purpose, with my own hands, on a machine that runs unattended.
I built it anyway, with the boundary drawn in code rather than in my intentions.
The click is gated on an allowlist that lives in the settings table as a JSON array of substrings. Empty or missing means notify only, which is the default. Nothing self-approves unless I have written down, in advance, the specific text of the prompts that are allowed to self-approve. Right now that list has exactly one entry: "node" would like. My own runtime asking for its own grants back, usually because a Homebrew upgrade moved node's path and silently dropped its permissions. Everything else on the machine still buzzes my phone and waits for me.
Three things fought me on the way to a working click.
SSH clicking is a dead end. My instinct was to have the click come from my remote session, but a synthetic click from an SSH context has no GUI session attribution. On that path the prompts don't even appear, let alone accept input. The click has to originate from the worker's own logged-in GUI context, which means the thing being granted permission is also the thing pressing the button. That is uncomfortable, and it is the only thing that works.
The Allow button doesn't OCR. Every single detection came back with "Don't Allow" and nothing else. The default button is white text on the accent color, and Vision reads it as no text at all. The button I needed most was the one button on screen my detector could never see. macOS lays the default out immediately right of Don't Allow, so I derive the click point geometrically from the box I can read. It's a guess, backed by measurement, and the verification step catches it when the guess is wrong.
Verification lied to me once. I confirmed a click by re-reading the screen and checking that no dialog signature remained. Then a second, unrelated prompt was open during a test, so the check reported failure on a click that had genuinely landed. Now I verify that this dialog is gone, matched by its own pattern text, instead of asserting that zero dialogs exist anywhere.
What the OS still wins
The proof it works isn't the screenshot. It's a line from the dialog's own process: button returned:Allow, gave up:false. That's the prompt itself reporting a real registered press, not my code claiming success at a coordinate.
The part I like most is the part that fails. macOS blocks synthetic input on some prompt classes outright, and no amount of cleverness gets past that. So the watcher tries once per fingerprint, and if the dialog is still there, it stops and falls back to buzzing my phone. The prompts guarded hardest are the ones that matter most, and on those, the operating system beats my automation. Correctly.
That's roughly the shape I want for anything unattended I build now. Automate the boring recurring thing, write the exception list down in a file instead of carrying it in your head, make it default to asking, and let the parts you can't automate stay un-automated instead of forcing them. I have an agent that clicks Allow for exactly one prompt. Everything else still gets a human.