Doing

Driving your Mac

Moe can see your screen and use your Mac, open an app, click a button, fill a field, through the same accessibility machinery a screen reader uses. Clicks are delivered straight to the target window in the background, so your cursor does not move, your keyboard focus does not change, and you can keep typing while Moe works in another window.

It tries not to touch the screen at all. Driving a window is the last resort, not the first. Asked to open a meeting link, Moe reads the join address off the calendar event and opens it: no screenshot, no clicking, nothing that can miss. The order it works through is: the data it already has, then the app's own scripting, then a menu item or a keyboard shortcut, then its own browser for anything on the web, and only then a click.

It knows your apps' menus. Ask Moe to do something in an app and it can read that app's menu bar directly, the items and their shortcuts, and invoke the one it needs by name rather than hunting for it on screen. That is faster and far more reliable than clicking, and if the item does not exist Moe is told so plainly. It also works where looking does not: music software, CAD, 3D tools and games draw their main window as one canvas Moe cannot read, but their menus are ordinary. Items that are greyed out are reported as greyed out, which is the difference between "not available right now" and "does not exist".

Moe's own browser#

When the job is a web page, look something up, fill a form, work through a site, Moe drives a browser of its own rather than yours. It reads the page as text, so it knows what the buttons and fields actually are instead of guessing from a picture, and it clicks the exact one. Nothing appears on your screen while it happens, nothing takes focus, and your own tabs are untouched.

It is signed in as you. If your default browser is Chrome, Edge or Brave, Moe browses on a private copy of your profile: your logins and cookies come along, so a booking site with your account, or a cart you started, are places Moe can work without you signing it in. The copy lives in Moe's own folder, is refreshed each time Moe opens its browser — and again when you sign in to something in your own browser while Moe's is open, once Chrome has saved it — and never touches the browser you have open. Google is the one sign-in that does not come along: Google ties it to your own Chrome. So when a site offers "Continue with Google", Moe presses it and picks your account in Google's list itself. It only brings the page up for you if Google then asks for something only you can give — your passkey, a security key, a verification step or your password — and carries on once you have done it. Moe's browser is its own application on your Mac, separate from your Chrome: opening or quitting yours never reaches Moe's, and Moe's never shows up as yours. To carry your logins across, your own Chrome runs for a fraction of a second, hidden, on the copy, to read its cookies and hand them over, and it is gone before Moe's browser starts; Moe never asks for your Keychain and never keeps its key. Moe's browser closes itself a couple of minutes after the last job, and when Moe quits. Moe never starts a second copy of your browser. If a site still blocks Moe's own browser, Moe can do it in your Chrome instead, once you have added the Memoe extension (see Moe in your Chrome); without it, Moe says so and offers to do it on your screen with you watching. With Safari, Firefox or Arc as your default, Moe browses signed out, because it has no profile it can copy. Change your default browser and Moe follows at its next launch. Away from your Mac, Moe has the same browser but not your profile. Its browser there keeps its own sign-ins from one day to the next, on your cloud computer and nowhere else: a site it has been signed in to stays signed in, and everything else it browses signed out.

Signing in on your cloud computer. When a site there needs you (a password Moe doesn't have, a code, "is this you?"), Memoe sends a link to your own chat: Sign in to x.com on your cloud computer. It only ever goes to your own chat, never to anyone Moe is talking to, and Moe itself never sees it. Open it on your phone and tap Open the browser; you see the page as it is on your cloud computer, and you tap and type into it as you would on any site. Your password goes to the site, not to Moe. Tap Done, tell Moe, and it carries on. The sign-in stays there for every later job. The link works for ten minutes, only on the first phone that opens it (a forwarded copy won't open), and a newer link ends the older one. Treat it like a key: don't share it.

"Are you human?" checks. When a site puts up a check that it is a person visiting (a "Just a moment" page, an "I'm not a robot" box, a press-and-hold button), Moe deals with it itself where it can: it waits out a page that clears on its own, ticks the box, or holds the button, just as you would, and carries on. It only does this in its own browser on your Mac, never from your cloud computer. When a check needs more than that, or keeps coming back, Moe brings the page up in front of you itself, without you having to ask, tells you in one line what to do, and carries on when you say you are done. It never tries on ticket-selling sites, where getting past these checks automatically is against the law in the US, and a page saying your connection has been blocked goes straight to you.

When no page will load. If your Mac is set to send web traffic through a proxy that is not running (System Settings › Network › your connection › Details › Proxies; scutil --proxy in Terminal shows it), every page in Moe's browser fails with "There is something wrong with the proxy server". Moe now says that it is your Mac's proxy setting, not the website, instead of retrying the site. It never switches the proxy off for you; turn the proxy off there, or start whatever it points at.

For the picture puzzles ("click every square with a bus"), Moe can use a solving service on your own account: set browser.captcha.solver to nopecha or capsolver in Moe's Hermes settings and put that service's key in ~/.hermes/.env (NOPECHA_API_KEY or CAPSOLVER_API_KEY). Only the puzzle's picture is sent; Moe clicks the squares itself, in its own browser, and tries at most three rounds on one puzzle. hCaptcha's puzzles are read straight from the picture on the page; a puzzle that moves or asks you to drag something is swapped for a new one (a few times, free) rather than sent to the service. Without a key, those puzzles come to you. Cloudflare's checks are never sent to a service.

When a check comes to you. Moe shows you its own browser window with that page open, and anything it had already typed into the form is still there. Solve the check, tell Moe you are done, and its window goes back out of sight while it carries on in the background. A sign-in page, or any page in the middle of something Moe was doing, is never sent to your own Chrome, where it would mean nothing; if Moe cannot show you its window, it says so. Only a plain public page may open in your browser, and Moe then says plainly that it is your browser and that you will need to finish that step there yourself.

A passkey usually cannot be used from Moe's browser, because your passkeys live with your own Chrome and Touch ID. When a site asks for one, Moe first tries the page's "Try another way" for something that works anywhere, such as a prompt on your phone or a code.

When a page is slow. A page that is busy, or waiting on a pop-up, gets half a minute to answer before Moe gives up on that step, and Moe is told which page it was rather than that its browser broke. A pop-up nobody can answer is dismissed instead of holding the page. Moe closes the older tabs it opened itself, keeping at most six, so a long job does not slow down as it goes.

Logins and passwords. When a site wants you signed in, Moe first checks whether it already has a login saved for that site, and uses it if so. If it has none, there are two ways to give it one, and in neither does the password pass through the conversation or through Moe's own thinking:

You type it, in Moe's password window. A small Moe window comes up in front of whatever you are doing, headed with the site's address — "Save your login for fazier.com". Type your username or email and your password (the password box shows dots) and press Save or Return; Esc or Cancel leaves without saving anything. Moe stores the login without ever seeing the password, and only ever fills it in on that site. Check the address in the heading before you type: if it uses letters from another alphabet, the window warns you. A window nobody answers closes by itself after five minutes, and nothing is saved.
Moe makes one, for a new account. When Moe is signing you up somewhere new, it can make a strong password itself and keep it, so you never have to think one up. It asks you first, every time, with a card headed Create an account on fazier.com? that says Moe will make a strong password and keep it for you, and shows the email or username the account will use. Press Create account, or Not now and nothing is made. A yes covers that one site only — never the next one, and not for an hour the way some browser steps are — and no "always allow" can switch this question off. Moe only asks once it has what a signup needs, the page's address and the email to use; if either is missing, you are not asked to approve something that cannot work: nothing is made, and Moe is told what it left out.

Codes a site emails you. When a site signs you in by emailing a code ("we sent a 6-digit code to your email"), Moe finds the box for it, reads the fresh email from that site in your connected mail itself, and puts the code in. It waits up to a minute for the email to arrive, and it looks in every Gmail account you connected. It only uses mail that arrived in the last ten minutes and that comes from, or is about, the site it is signing in to, and the code never appears in your conversation. It does this only while you are there, in a conversation you started. A scheduled job never reads a code out of your mail — a code in your inbox can sign someone in, and a job runs on words that are not always yours — so the job reports that the sign-in is waiting for you instead. If no email turns up, Moe asks you for the code in its password window — "Sign-in code for fazier.com" — where you type it and press Enter code, and Moe puts it in the page without it appearing in your conversation. If you read a code out to Moe, or type it in the conversation, Moe does not type it for you yet: it brings the page up in front of you and asks you to type the code in yourself.

Passcodes and PINs you know. When a site asks for a passcode or PIN that only you know (a chat app's passcode, a PIN to unlock something), Moe does not stop and tell you it cannot: it asks for it in the same password window, headed with what the site wants, and you type it there and press Enter code. Moe puts it into the page without ever seeing it, and it never appears in your conversation. Read it out or type it in the chat instead and Moe will not use it; it asks you to type it into the window, or into the page yourself.

When a page needs you. Some steps only you can take: a passkey or a security key, a "confirm it's you" check, an "are you human?" check Moe could not get past itself. Moe watches for these by what the page says and offers: a sign-in or verification page that tells you to do one thing ("Use your passkey to confirm it's you"), with nothing else to fill in — not by the word alone: a sign-in page that merely offers "Sign in with a passkey" beside a password box, or an article about passkeys, is not one. It reads English first, and a few common lines in French, German, Spanish and Hebrew; in other languages it relies on seeing that the page is stuck, and you can always ask to see the page. Moe's browser is out of sight, so when it meets one — or when you ask to see the page ("open that so I can approve it") — and you are at your Mac, Moe brings the page up in front of you in its own browser window, on the same page and still signed in, tells you in one line what to do, and waits. Say "done" and it carries on from where you left the page; the window stays open until you do. If Moe's browser had to restart to show you the page, every tab comes back, but anything typed on a page that the site had not saved is gone, and Moe says so. If another conversation is using Moe's browser at that moment, Moe does not restart it under that work: it opens the page in its tab group in your Chrome (when the Memoe extension is connected), or else in your default browser — and then says that its own sign-in did not come with it.

"At your Mac" means you asked from the Mac itself and it is unlocked, awake and its display on. If you asked a moment ago and are now waiting without touching anything, that still counts for the first three minutes. If you are messaging Moe from your phone, or a scheduled job meets the step, nothing pops up on an empty screen: Moe tells you what is waiting and where, and picks it up when you are back. Moe only tells you that something was sent, or is waiting on your phone or another device, when the site itself said so.

Which of your profiles. Chrome keeps one set of logins per profile — the faces in its profile menu. Moe carries one of them. Left alone, it is whichever profile you used last, and if you have several that changes as you switch windows: Moe can be signed in to X in one profile and signed out in another. Settings › Permissions › Browse as lists your profiles by name and account and lets you pick the one Moe should always use; "Whichever I used last" is the other choice. It takes effect the next time Moe opens its browser. The row appears only when there is something to choose — one profile, or a browser Moe cannot copy, and it stays out of the way.

If Moe meets a sign-in page where you should already be signed in, it stops and says which profile it is browsing as, and that Browse as picks another — it does not guess a password, and it does not report the job as done.

For a tab you have open on screen, or a browser's own menus and dialogs, Moe looks at the screen the way it looks at any other app.

Moe does not ask before every click. Clicking a date, typing a search, scrolling a list, opening a page: none of that is a question. What Moe asks about, in a browser, is the moment that matters: paying, buying or booking; sending a message, an email or a post; deleting or cancelling something that does not come back; entering a password or a card number; letting an app into one of your accounts. One yes covers that kind of moment for the rest of the conversation, up to an hour, so a booking is one question, not one per page. See Before anything goes out.

Moe in your Chrome#

Some sites refuse any browser but the one you use yourself. For those, and whenever you ask, Moe can work in your own Chrome through the Memoe extension. Moe works in its own tab group in your Chrome, only on turns you start; Chrome shows a bar while it does.

Its own tab group. Moe's tabs sit in a group called Moe, in a window of their own behind yours. Moe never reads, moves, closes or types into a tab outside that group — unless you ask, or drag a tab into it — and it leaves your focus where it is.
Only on turns you start. A scheduled job, a background task or a message from your phone uses Moe's own browser, never yours.
The bar. While Moe is working in a tab, Chrome shows "Memoe — Moe in your Chrome" started debugging this browser across the top. That is Chrome saying an extension is at work; it goes when Moe is done.

Setting it up. Open Connections › Moe in your Chrome and click Set up. Moe puts the extension in its own folder, tells Chrome how to reach it, opens Chrome's Extensions page and shows the folder in Finder. In Chrome:

Turn on Developer mode, at the top right of the Extensions page.
Click Load unpacked.
Pick the chrome-extension folder Moe showed you.

The row turns to Connected within a few seconds. If the Extensions page did not open, type chrome://extensions into Chrome's address bar. (When the extension is in the Chrome Web Store, Set up opens its page there instead.)

The row reads Not installed, Installed, not connected (the extension is not loaded, or Chrome is closed) or Connected. If it also says Chrome lets the extension open files on your Mac, turn off Allow access to file URLs under Chrome › Extensions › Memoe; Moe never uses it. After Memoe updates, the row can say Update available: click Update, then Reload on Memoe in Chrome's Extensions page, so the extension matches the new Memoe. Without Google Chrome on the Mac, Set up says to install it first and changes nothing.

Turning it off. Turn off on the same row stops Moe using your Chrome within a few seconds. The extension stays in Chrome until you remove it there (Chrome › Extensions › Memoe › Remove). Turn on brings it back.

If Moe needs your Chrome and the extension is not connected, a Fix card says so with a Set up Chrome button that opens this row.

Windows on another desktop#

If you use several Spaces, an app on a desktop you are not looking at cannot be clicked; macOS does not allow it. Moe can still read such a window, and it will tell you where the window is rather than pretending the app is missing. Let Moe bring a window to the front when it has to, in Settings › Permissions under Moe and your screen, is on: for the few things that cannot be done in the background, a window on another desktop, a minimised one, or a menu in an app that is not in front, Moe brings that window forward, which on a full-screen app means moving you to another desktop. Off, it leaves your screen alone and tells you what it could not reach. Reading the screen is never affected either way.

It stops instead of looping#

Every action comes back with a verdict saying whether it actually landed, and Moe checks that rather than assuming. If the same action returns the same result several times, Moe changes approach; if it happens again, the action is refused outright and Moe tells you what it is seeing and asks. Actions whose whole point is repetition, scrolling, arrow keys, undo, are never counted against that.

And it knows when to stop. A screen task has an allowance: 150 actions, and 30 looks in a row with nothing done between them. Past those, Moe stops and tells you where it got to and what is in the way, rather than clicking on for another ten minutes; the allowance refills once you have answered.

You can always stop it#

While Moe has your screen the notch says so in words beside it, Using Safari with a Stop you can click, and a thin magenta line glows around the edge of the screen, so you can tell at a glance, wherever you are looking, that it is clicking and typing rather than just thinking. The line never gets in the way of a click and never shows in a screenshot. When Moe is in its own browser, working through a page off-screen, the notch says Browsing. Hover the notch and the word in the open panel names the same thing; the stop icon behind the ⋯ turns magenta and its label says what it ends: End computer use or End browser use. Pressing it gives up the whole run: no further click or keystroke is sent, and whatever Moe was part way through is abandoned rather than finished behind your back. Stop beside the folded notch does the same. The words and the line go when the run ends. Stopping is something you do, not something Moe guesses: it cannot tell your hand on the mouse from its own typing, so it does not try.

Moe never types a password, a passcode, a PIN, a 2FA code, a card number or a recovery phrase you read out to it or that it sees on screen. That does not stop the job: in a web page, it asks you for it in its password window and enters it without seeing it; anywhere else, it tells you which field needs you, waits while you type it, and carries on.

Screen access needs two macOS permissions, Accessibility and Screen Recording, and they belong to the small driver Moe installs for this rather than to Moe itself; Settings › Permissions shows which are missing and takes you to the right place.

Settings › Permissions#

What Moe asks macOS for, and whether macOS has said yes. None of these is required: Moe works without every one of them. A permission you don't give just switches that one thing off, and Moe says so instead of pretending it looked. Nine rows, in the order they tend to come up:

Microphone — hearing you while you hold the talk key.
Accessibility — noticing the talk key while you are in another app.
Input Monitoring — the same job; some Macs ask here instead.
Automation — asking Notes, Calendar and Messages for what is in them, and clicking and typing on your behalf when you ask Moe to drive the Mac.
Calendar — what you have on, every calendar in Calendar.app including Google ones, and adding events when you ask.
Reminders — what you still owe people, and adding one when you ask.
Contacts — who you mean when you say a name.
Full Disk Access — the things macOS keeps locked away even from you: Messages and SMS, call history, Chrome history, and the Google Drive folder on this Mac.
Screen & System Audio Recording — seeing your screen when you ask "what do I do here", and recording the other side of a call in a meeting.

The word beside each name is the state: Allowed, Not allowed, Not asked yet (nothing is broken; Moe has not needed it yet) or Can't tell. Input Monitoring and Automation always say Can't tell: macOS gives no way to read those two without either prompting you or guessing, and an honest blank beats a wrong guess. Re-check at the top reads them all again.

How do I? on a row (Details, on one that is allowed) opens what it is for, what happens if you say no, the click path in System Settings, and Open the settings pane, which takes you to the right place. The four permissions macOS only reads when an app launches, Accessibility, Input Monitoring, Full Disk Access and Screen Recording, also offer Quit and reopen Moe. On Accessibility alone there is one more: if System Settings shows Moe switched on and this still says Not allowed, that switch belongs to an older copy of Memoe; Reset and ask again clears it, and you switch Moe on once more.

Moe and your screen, under the rows, holds three switches about what Moe may see and touch on the screen: Moe sees my screen when I ask (on), Move my mouse to what Moe points at (off) and Let Moe bring a window to the front when it has to (on). Talking to Moe and Driving your Mac say what each does.

What Moe is allowed to do, below them, is about Moe's own hands rather than macOS's. Moe can do anything read-only on this Mac without asking; sending mail, sending messages, sharing files and deleting are denied to start with, because a voice assistant has nowhere to show you an approval dialog mid-sentence, so the safe default is that nothing leaves your machine. Edit the permission rules opens the file those rules live in, where you can move a line from deny to allow when you decide you want it; restart Moe afterwards. It is a guard rail, not a sandbox: the real protection is that outbound actions are off until you say otherwise, and what stands in front of a send when you do is the next page.

When something needs a switch turned on#

Sometimes Moe can't do what you asked because macOS, or one of your connections, is keeping it out: Memoe isn't allowed to read Chrome's data, or into your Desktop folder, or to control Notes. When that happens a Fix card appears on the panel under the notch and in Chat: one line saying what is in the way, in plain words, and a button that fixes it.

Open Settings opens the exact page of System Settings where the switch is. Turn it on and Moe notices within a couple of seconds, then carries on with what you asked — you don't have to ask again. It keeps looking for five minutes; after that, just ask again.
Moe also tells you in its reply what is in the way, in one sentence, and offers to open the setting. Say yes and it opens the page for you, the same way the button does. It only does that while you are there, in a conversation you started: never from a scheduled job, a background task or a message from your phone.
The card always names whose switch it is. Most are Memoe's own (Memoe Dev's, if you are running that). Seeing and using your screen is done by a helper called CuaDriver, which has its own rows under Accessibility and Screen & System Audio Recording; when the card says CuaDriver (the screen helper), that is the one to switch on, not Memoe.
Some fixes aren't a switch. A connection that needs you to sign in again offers Open Connections; an expired sign-in to Claude offers Sign in; Moe's Chrome extension not being connected offers Set up Chrome, which opens Moe in your Chrome; a screen helper that has stopped offers Quit and reopen Memoe, which starts it again. A few only explain what happened, with Dismiss.
A window on another desktop gets no card: Moe switches to it, or asks you to.
The card only ever names things it recognises — a browser, a folder like Desktop or Downloads, an app like Notes, a connection like GitHub. Anything else is just "another app", "that folder" or "a connection", so nothing a web page or a program says can put its own words on the card.
A card about controlling another app (Automation) appears only after Memoe has checked with macOS that the switch really is off for that app.

A card shows once for each cause while Moe works on one request, however many times the same thing failed, and goes when you press a button, Dismiss it, or ask Moe something new. Moe never turns a permission on for you (macOS doesn't allow it); it tells you which one and takes you there.