A guide that points at the control you need.
Ask about the window in front of you. The companion reads the controls that window publishes, talks you through the next step in plain sentences, and moves the real pointer onto the control it just named, while it is still talking. It points. It never clicks.
Built for macOS 14 and later on Apple Silicon, and running today. The Windows build comes from the same codebase and has not been run on real hardware yet. Both are pre-release, so there is no download link anywhere on this page: the form at the foot of it asks for your address, and the build comes by email.
Watch it point
The pointer arrives before the sentence ends.
The answer streams back a step at a time, and the instruction to move the pointer arrives in the middle of the sentence rather than after it. So the cursor is already travelling while the rest of that sentence is still arriving. On the one full run measured on hardware, the pointer crossed the screen about two seconds into the answer, and again at eleven.
The ring is drawn on the control’s real bounds, taken from the window itself. When there are no real bounds to take, the ring is left off on purpose, and the pointer is dimmed to match.
How it works
Three things happen when you ask.
There is no always-on capture, no background recording, and no model call until you ask for one. Between questions the only thing running is a note of which application is in front of you.
It reads the window
One read per question, not a running watch of your screen. It asks the app in front of you what it is showing: the name, the role and the exact rectangle of each control, filtered down to the ones a person could actually use and capped at a hundred and twenty. The only thing it watches between questions is which application is in front, checked four times a second, so that clicking into the companion to type does not make the companion the subject of its own answer.
It talks you through it
The answer comes back as one to six short steps in plain sentences. No headings, no bullet lists, nothing to translate into clicks. With speaking left on, each step is read out as it arrives by the voice your system already has, which needs no key and makes no network call.
It points at the control
It warps the pointer onto the control it just named. Warping rather than posting a mouse event is deliberate: the companion points at things and never clicks them, so it cannot synthesise a button press that an app would mistake for you.
Why it lands on target
Exact where the window says so. Visibly unsure where it does not.
When the window publishes its controls
A window can hand the accessibility layer a list of what it is showing: names, roles, and the exact rectangle of each control. That list is how a screen reader works, and the companion reads the same one. It points at the rectangle the window reported, so nothing is rescaled and nothing drifts, and the ring lands on the control’s real bounds. Across a sweep of native windows that list came back rich enough to point with, and some browsers pass a whole web page through it, so the same exact pointing reaches things inside a page.
When it publishes nothing useful
Some windows return their menu bar and nothing else. When fewer than three real controls come back, the companion takes a single picture of the screen instead, scales it down before it goes anywhere, and works from coordinates. The pointer is then drawn dimmed and no ring is drawn at all, because a guess should not look as confident as a hit. Apps that ship their own browser engine are the case this was built for: several of them published their menu bar and an empty window shell, which is a list long enough to look healthy and useless to point with.
Measured on hardware, the coordinate maths is accurate to within one point across four display modes, from the default Retina mode through a scaled one to a non-Retina one, all of them on a single built-in display. A second monitor has not been through it yet. When it does fall back to a picture, that picture is of the primary display, which is the display the window is on in the common case but not in every case.
Permissions and privacy
What macOS asks for, and why.
- Accessibility
- The control list. This is the whole difference between pointing at a button and pointing near it. Without it the app still opens, reads zero controls, and tells you that is what happened.
- Screen Recording
- Only for the fallback picture, on the questions where the window in front of you published nothing useful. On every other question no image is captured at all.
- Microphone
- Only for the hold to talk key. The key cannot leave the microphone open: a hold under two tenths of a second is discarded as noise, and a watchdog force ends any capture at thirty seconds. That watchdog has been triggered on hardware to prove it fires.
macOS applies each grant to the next launch rather than to the process already running, so the app asks you to restart it after every one. That is the operating system, not a choice we made. It also runs outside the sandbox deliberately, because a sandboxed process gets nothing back from the accessibility API.
What leaves your machine
- Your question, and the list of control names the window published.
- The downscaled picture, on the questions that needed one. On the questions that did not, no image is sent.
- Put your own model key in Settings and the only server in the path is your model provider, called directly. Leave the key empty and the same question goes to our hosted relay instead, which holds the key for you and forwards it. Which of the two you are on is the difference worth knowing, so the key field is on the first settings screen.
- Speaking is local by default: the answer is read out by the voice your system already has, with no key and no network call. Adding your own voice key swaps that for a hosted voice, and then the text of each step goes to that service too.
- Both of the companion windows are excluded from screen capture, so the companion never lands in a recording. That cuts both ways: it will not appear in your own screen share either.
Where the build is today
What is running, and what is not.
All of this came out of one bring-up session on one machine. It is written down here the same way it is written down in the repository.
- macOSRunning today
- Apple Silicon. The build targets macOS 14 and later, though the one machine it has actually been run on was newer than that. It is signed but not yet notarized, so it is handed out directly rather than through a public download link.
- WindowsCompiles, not yet run
- The same codebase, compiled on every change in CI. It has never been launched on real Windows hardware, and we will keep saying so until it has.
- Voice inputKey verified, prompt missing
- The hold to talk key and its watchdog are verified on hardware. The microphone usage string is still missing from the build, so typing your question into the companion window is the reliable way to ask today.
- More than one displayUnit tests only
- The coordinate maths is covered by tests, but no second monitor has been through it on hardware. That is the largest gap we know about, and a negative origin secondary display is the case we most want to try.
Point it at the window you are stuck in.
The macOS build runs today and goes out by email while it is pre-release: one message, when there is a build for you to run. The Windows build is on the same list, and it is not going out to anyone until it has been launched on real hardware.