The 5-Question Filter: What Should You Actually Automate?

Most automation queues are sorted by demo appeal. The flashy task goes first, the tedious one goes last, and six weeks later the flashy one is an unmaintained script nobody trusts while the tedious one is still being done by hand.

That isn't a model problem. It's a sorting problem.

A demo is one run, watched by a forgiving human. Production is the thousandth run with nobody watching.

"Can AI do this?" is the wrong question

It's the question everyone asks, and it has stopped being useful. For a wide and faintly insulting range of tasks the answer is now yes-ish, which tells you nothing about whether you should build it.

It's a capability question. The outcome is decided by specification. Capability is whether a model can produce something once, with a forgiving audience. Specification is whether you can state the job precisely enough that a system can do it a thousand times with nobody in the room.

That gap is demo versus production — on a day when the ledger export has a blank column and the client contact has changed jobs.

So filter on the shape of the task. Five questions, yes or no, in the order the worksheet uses them.

1. Is it frequent?

Why it matters. Automation cost is front-loaded: mapping the process, collecting examples, writing the spec, building the check, maintaining it while the tools change. The return arrives per run, so frequency is the only input that compounds — and a weekly task hands you a test set for free.

The failure mode. Automating the quarterly board pack: three weeks of work to save two hours a quarter, and the template changes before it has run twice. Rare tasks also mean nobody remembers what good looks like.

What a good answer looks like. Yes, weekly or more often, and the runs resemble each other. Monthly is a checklist and a saved prompt, not a build. Weekly but messy? Automate the stable core, leave the tail to a person.

2. Is the cost of error low?

Why it matters. This is about blast radius: how far a wrong answer travels, and who gets hurt. A wrong internal draft reaches one colleague and dies there. A wrong email to a client gets forwarded and becomes a conversation about whether you can add up. A wrong payment instruction reaches the bank. How far it travels decides what the system may touch.

The failure mode. Treating "the model is usually right" as a risk assessment. It isn't. Usually right means occasionally wrong, spread across the client list at volume. The expensive failure is rarely a garbled answer; it's a confident, well-formatted wrong number that goes out because it looked like the others.

What a good answer looks like. A clean yes: if it's wrong, the worst case is a wasted hour and nobody outside the team sees it. A no is not a rejection. High blast radius doesn't mean never; it means guardrails, built first — a ladder you climb in order: dry run, where it proposes and changes nothing; human review, where a person reads every output; approval, where a named person signs the action; spend cap, a hard ceiling it cannot exceed. None of it is expensive before the build, and all of it is embarrassing after — it's also the difference between an agent you supervise and a control plane that can stop it.

3. Is it verifiable?

Why it matters. Verifiable means you can write down what correct looks like, then test it. Without it you can't distinguish improvement from change, and neither can the reviewer — they aren't checking quality, they're forming an impression.

The failure mode. Output that's cheap to produce and expensive to trust: long memos, 400-line diffs, decks of numbers nobody checked.

What a good answer looks like. Yes, and you can name the check: a diff against a known-good file, a reconciliation to the ledger, a schema, a five-field checklist. Then the nuance people skip — verification has to be cheap. If checking takes longer than doing, it isn't verifiable in practice; it's doing the task with extra steps and a machine's confidence attached. Seconds per item works. Minutes per item is a hobby.

4. Is it well-specified?

Why it matters. An agent optimises to your specification, including the holes. Ambiguity doesn't get resolved by good judgement; it gets resolved arbitrarily and then repeated consistently. Consistency looks like reliability, which is what makes it dangerous.

The failure mode. "Summarise the competitor news." Forty bullets one week, three hedged paragraphs the next. Every output is defensible, none is what you wanted, and you're maintaining a nine-hundred-word prompt that says "be concise".

What a good answer looks like. The stranger test: could a stranger do this and know when they're done? That needs a scope, a shape and a test. "One page, five items maximum, each with a link and one line on why it matters, from the last seven days, excluding anything we covered last week." A stranger could accept or reject that, and the sentence contains its own check.

5. Is the middle substantial?

Why it matters. Picture a sandwich: thin inputs at one end, thin outputs at the other, a fat routine middle in between — reading, transforming, drafting, checking, routing. The middle is where the hours go, so the middle is what you're buying. If it's thin, one lookup and one format change, you need a formula, not a system.

The failure mode. Automating the ends. Bolting a chat window onto a task whose middle is two clicks.

What a good answer looks like. Yes: the middle is substantial and routine, repetitive enough to describe, varied enough to need judgement. Then run the flowchart test. If you can draw the whole thing as a flowchart with no decisions in it, it's a workflow, not an agent; build the workflow, which is cheaper and more reliable. If the flowchart has a box where a person has to weigh something, that box is the agent-shaped part. Either way you've learned what to build, not just whether to.

Worked example: chasing overdue invoices

A composite example, not a specific firm. A thirty-person services business sends fifty invoices a month, and somebody spends a few hours a week chasing the late ones.

Frequent — yes. Chasing happens most days, each invoice follows a reminder cadence, and a year of sent emails is a ready-made example set.

Cost of error — not low. Client-facing, and about money. A wrong contact, a wrong amount, a wrong tone, or a reminder to a client who paid yesterday all cost trust. So the guardrail ladder comes first: read-only ledger access, draft-only output, a human presses send, an amount check before every draft. Dry run, review, approval, then a spend cap.

Verifiable — yes, and cheap. The reviewer compares two numbers and a date against a ledger row, then scans the tone: seconds per draft, against minutes to write one.

Well-specified — mostly. "For every invoice more than seven days overdue with no payment recorded, one draft to the billing contact quoting the invoice number, amount and days overdue, in house tone, queued for review, never sent." That passes the stranger test, with named exceptions: disputed invoices, part-payment arrangements, and the handful of clients where late payment is a relationship problem, not an admin one. Those stay on a human's list.

Substantial middle — yes. Read the ledger, match payments, compute days overdue, pick the contact, draft, route for review. Thin ends: a list of overdue invoices in, a queued draft out. The middle is the work.

Verdict: agent-shaped. Build it in this order — guardrail, spec, check, agent — because reversing that order is how you end up apologising to a client in week two.

The score isn't the point

Five yeses is a green light; five nos is a no. The interesting cases are mixed. A weighted total averages away a fatal flaw: four strong answers and one "we'd never really know if it's right" is not an 80% project. It's three months of quiet waste and a small dent in your credibility.

Read the hesitations instead. The question you had to think about is the work item. Hesitated on specification? You're not ready to build; you're ready to write a spec. Hesitated on verification? You're about to build something nobody can supervise.

Ordering beats scoring. The boring task that repeats weekly, has a crisp definition and a check you can run in ten seconds will beat the impressive one — not because the model is better at it, but because you can tell when it's wrong.

Start here

Or read the long version above, score four tasks from your own week, and pick the one that scored cleanly. Build that one first.