Teach workflows: edge-case gap analysis
This lists the cases the plan (strict inputs, model-free runner, desktop steps, files, business flow) does not cover yet. Each row has a severity and a fix. P0 blocks real business use. P1 breaks common workflows. P2 is polish or scale. "Today" is checked against the code where marked ✓.
1. What the recorder misses
| Edge case | Today | Sev | Fix |
|---|---|---|---|
| Keyboard: Enter to submit, Tab, shortcuts (⌘C/⌘V, ⌘S) | not recorded: only click and change |
P0 | Record keydown for Enter/Tab/Escape and modifier shortcuts. Never record plain typed characters (the value comes from input). |
| Rich editors (Gmail, Notion, Docs): contenteditable | missed: change doesn't fire there |
P0 | Record input on contenteditable and log the final text once per field. |
| Embedded frames (payment forms, embedded apps, Salesforce) | main frame only | P0 | Inject into every frame. Name the frame in the step. |
| Shadow DOM (modern web components) | Labels come from the outer element | P1 | Use composedPath()[0] and build selectors that pierce shadow roots. |
| Canvas apps (Google Sheets cells, Figma, maps) | Clicks have no element | P1 | Record as a desktop-style step (coordinates relative to a found anchor) plus a check. Prefer the app's API when one is captured. |
| Custom dropdowns, date pickers, autocomplete | Recorded as raw clicks on list items | P1 | Detect role=listbox/option/combobox and record "choose {value}". |
| Drag and drop, file drop zones | Not recorded | P1 | Record drop with the file name, then make the file an input. |
| File upload (OS file picker) | The picker is outside the page | P0 | Record the chosen file as a file input. Replay sets it directly; the picker is never opened. |
| Downloads and save dialogs | Not recorded | P0 | Record will-download: file name pattern and folder become a file.expect. |
| New windows and popups (OAuth, print, "open in new tab") | Partly | P1 | Follow the new window as a new step target. |
| Hover menus, right-click menus | Hover not recorded | P2 | Record the click that follows. Replay opens the menu by its trigger. |
| Copy here, paste there (between apps) | Not recorded | P0 | Record a copy as "extract value → variable" and a paste as using that variable (section 3). |
| Mistakes, undo, backtracking during recording | All recorded | P1 | Collapse undone actions. Let you strike steps in Review (exists). Offer "trim to the successful path". |
| Waiting (spinners, slow loads) | Timing not used | P1 | Record which element appeared before the next action, and replay waits for it rather than a fixed time. |
2. Desktop apps (the planned desktop recorder)
| Edge case | Sev | Fix |
|---|---|---|
| Apps with no accessibility tree (games, Citrix/RDP, VMs, some Java apps) | P0 | Fall back to image anchors (a cropped picture of the control). Mark these steps fragile. Show "this app can't be automated reliably" when recording. |
| Different screen size, DPI, multiple monitors, window position | P1 | Never store raw coordinates. Store the accessibility path, else coordinates relative to the window. |
| Different app version or language on another machine | P1 | Store role + label in the recorded language, plus the order in the tree as a fallback. Flag skills used across languages. |
| macOS permission prompts mid-run (Accessibility, Screen Recording, Automation) | P0 | Check permissions before a run starts. Never discover them halfway. |
| System dialogs and notifications stealing focus | P1 | Known-overlay allowlist: dismiss or wait. Bring the target window back to the front before each step. |
| Screen locked or Mac asleep during a scheduled run | P0 | UI automation can't run on a locked screen. Run on a cloud computer or VM, or stop with "screen locked" instead of failing silently. |
| A person using the computer at the same time | P0 | Use the existing control lease (423). Queue the run, or run on a separate computer or session. |
| Windows UAC / admin prompts | P1 | Elevated steps are a separate "needs admin" category, approved up front. |
| Mac skill run on Windows (or the reverse) | P1 | Desktop steps are tied to one OS. Show which OS a skill needs; browser/API/file steps stay portable. |
3. Data flow (missing from the step format)
| Edge case | Sev | Fix |
|---|---|---|
| Using a value produced earlier (an order number, a generated ID) | P0 | extract steps that save to a variable, then {{var.order_id}}. Validation treats variables like inputs. |
| Repeat for each row (CSV, inbox, list on a page) | P0 | for_each over an input list or an extracted list, with a per-item result and a "stop vs skip on error" setting. |
| Branches (record exists → update, else create) | P1 | if on a check result, recorded from two recordings of the same task, or added in the editor. |
| Pagination ("next page" until done) | P1 | A loop that stops when an element disappears. |
| Dates, time zones, locale formats (10/11 vs 11/10, 1.000,50) | P0 | Typed inputs carry a format per target field, taken from what was typed in the recording. |
| Optional inputs left empty | P1 | Steps marked "skip if empty". |
| Special characters, very long text, emoji | P2 | Escape per target (URL, CSS, shell). Set length limits per field. |
| Outputs: what the run returns (a file, a number, a link) | P1 | Declared outputs, shown in the run result and passed to the next skill or schedule. |
4. Accounts, secrets and sign-in
| Edge case | Sev | Fix |
|---|---|---|
| Passwords typed during recording | P0 | Already not recorded. The step becomes use_credential (exists in the computer toolset) pointing at the vault. |
| 2FA / one-time codes (SMS, email, authenticator) | P0 | request_human step (exists) or an approved code source (bot email inbox). Never stored in the skill. |
| CAPTCHA | P1 | Always request_human. Never solved automatically. |
| Session expired mid-run, SSO redirect | P0 | Detect the sign-in page as a known state and run the skill's sign-in sub-skill, then resume. |
| Runs on another machine or browser profile without the session | P1 | The business profile says which browser profile or account. Check sign-in before step 1. |
| Two runs on the same account at once | P1 | Lock per account, or queue. |
5. Safety and governance
| Edge case | Sev | Fix |
|---|---|---|
| Unattended scheduled run reaches a pay/send/delete step | P0 | Approve the specific action ahead of time per skill version ("may submit invoices under $X"), else pause and notify. |
| Retry after a partial failure submits twice (double payment, duplicate email) | P0 | Steps marked "not safe to repeat" never auto-retry. Resume after a crash starts after the last confirmed step. |
| Repair mode reads page text that tries to give it instructions | P0 | Repair may only propose new locators and waits. It can't add steps or change values. Page text is data. A person approves the new version. |
| Who may record, edit, publish or run a skill | P1 | Roles per business, and an audit log of edits and runs. |
| Recordings capture customer PII (names, emails, account numbers) | P0 | Today: secret scan and screenshots kept local. Add PII redaction for example values, and a "regulated data" flag that blocks cloud model use. |
| Network log holds cookies and tokens | P1 | Strip auth headers and cookies at capture, not only at the server. |
| Regulated work (finance, health) under the audit-first rule | P1 | Regulated skills need certification (a recorded test run) before scheduling. |
6. Things that go stale
| Edge case | Sev | Fix |
|---|---|---|
| Site or app redesign | P0 | Fallback locators → repair → proposed version (the plan). A drift report lists steps whose main locator keeps failing. |
| A/B tests, feature flags, per-account layouts | P1 | Several observed variants per step, from several recordings or repairs. |
| Cookie banners, surveys, chat widgets, "what's new" popups | P0 | Known-overlay allowlist per site, dismissed before a step when present. |
| Maintenance pages, rate limits, 5xx errors | P1 | Recognise them as known states: wait and retry later, not repair. |
| Data-dependent screens (empty states, zero results) | P1 | The check result decides: "no results" is a valid outcome, not a broken locator. |
| Daylight saving and time zones in schedules | P2 | Schedules in the business's zone (routines support timezone). |
| App and OS updates on the runner | P1 | Record app and OS versions per run. A version change plus a failure points to the cause. |
7. Running and operations
| Edge case | Sev | Fix |
|---|---|---|
| Which computer runs it (laptop, cloud computer, Windows VM) | P0 | Each skill names its runner, checked against the OS and apps it needs. |
| Crash or reboot mid-run | P1 | Checkpoint after each confirmed step, then resume or roll back. |
| Concurrency and queueing | P1 | One run per computer session; others queue. |
| Evidence of what happened | P1 | Per-step screenshot (local) + check result + locator used, in one run record. |
| Failure alerts | P1 | Notify the owner (push / email / Slack) with the failing step and screenshot. |
| Speed | P1 | Fast path with no model and no fixed sleeps. Measure time per step. |
| Testing without side effects | P1 | Dry run: stop before "not safe to repeat" steps, or run against a staging profile. |
8. Skill lifecycle
| Edge case | Sev | Fix |
|---|---|---|
| Several recordings of the same task (variations) | P1 | Merge into one skill with branches and per-step variants. |
| Waiting for something outside (an email arrives, a file lands, approval from a manager) | P1 | wait_for steps on events: mailbox, folder, webhook, with a timeout. |
| Skills calling skills (sign-in, export, upload) | P1 | Sub-skills with their own inputs and versions. |
| SKILL.md for agents vs the runner's script | P2 | Generate both from one source. Agents call the runner and don't re-read the steps. |
| Old skills with wrong data (e.g. the pizza skill's site name) | P2 | A migration pass that re-derives what it can and flags the rest. |
9. Surfaces
| Edge case | Sev | Fix |
|---|---|---|
| Recording on the web or phone (no Desktop) | P1 | The Chrome extension records browser steps. Phones run and monitor skills but don't record them. |
| Running a desktop skill from the phone | P1 | The phone starts the run on the chosen computer and shows live progress and approvals. |
| Cloud Linux computers | P2 | Browser, API, file and shell steps only. Desktop steps need a Mac or Windows runner. |
The P0 list
- Keyboard, rich editors, frames, uploads, downloads and copy/paste in the recorder.
- Variables (
extract),for_each, and typed formats for dates and numbers. - Credentials, 2FA and sign-in recovery as built-in steps.
- Pre-approved risky actions for unattended runs, and "not safe to repeat" steps.
- Repair limited to locators (protection against page-text injection), plus PII redaction.
- Known-overlay allowlist and drift report.
- Desktop: permission preflight, locked-screen handling, control lease, apps with no accessibility tree.
- Choosing a runner per skill (OS and apps).