How long should an AI pilot take?

Long enough to run on real data with real users, short enough that the client can absorb the cost of stopping. Most useful pilots cover a single decision for one team, with a decision date and pass criteria agreed before work starts. If a pilot cannot show working output on real inputs, the slice was scoped too wide.

What is an eval in an AI project?

An eval is a test set for a system whose output varies between runs. Instead of checking exact equality, it checks properties: correct format, a real cited source, an appropriate refusal, or a grader score above an agreed bar. Built from real historical examples, it gives you a repeatable quality number before and after every change.

Do AI agents need human approval before acting?

Wherever a mistake is expensive or hard to reverse, yes. Client-facing messages, financial actions and writes to a system of record should pass an approval queue where a person reviews and confirms. Low-cost internal suggestions can run without one. The rule follows from what a wrong answer costs, which should be settled during scoping.

How do I price an AI project when the effort is uncertain?

Price the pilot as a fixed, narrow piece of work with defined output and a decision date, then price the production build once the pilot has shown you the real data and the real failure modes. That keeps the client's initial risk small and stops you fixed-pricing work whose difficulty nobody can see yet.

All field notes

June 3, 20267 min read

How to Scope an AI Project So It Actually Ships

AI projects fail at the scoping stage far more often than at the build stage. The model is rarely the problem. The problem is that nobody wrote down which decision the system is allowed to make, what happens when it gets that decision wrong, and how anyone would know. Here is a scoping process you can run in a single call, plus the two traps that consume projects skipping it.

AI Systems
Process
Agency Ops

Start With the Decision, Not the Model

Every useful AI system makes or supports a decision. Which reply this enquiry gets. Which invoice is flagged for review. Where this ticket routes. Which contract clause answers the question asked. Before anything else, write that decision down in one sentence, in the client's own vocabulary, and get them to agree it is correct.

If you cannot name the decision, you do not have a project. You have an interest in AI, which should be billed differently. The usual brief is some version of we want to use AI in the business. Turn it into: the first-draft reply to an inbound enquiry is written automatically, for a named person to approve before it sends.

Naming the decision gives you scope boundaries for free. Everything that is not that decision is out of scope for the first build, and you have a sentence to point at when a new idea arrives in week three. Ideas are welcome; they go on the version two list.

The Five Questions That Decide Scope

Run these in order. Each changes the shape of the build, and any one left unanswered is a project risk you carry without having priced it.

Question two is where most projects quietly die. The data is in the CRM usually means it is in the CRM, plus three spreadsheets and one person's memory. Confirm working access before you quote, not after. Question four sets the approval design: cheap reversible errors can be applied automatically, while errors that reach a client or move money need a human gate. Question five is why systems degrade for months without anyone noticing.

What decision does this system make, in one sentence?
What does it read to make that decision, and do we have working access to all of it today?
Who is accountable for the output, and at what point do they see it?
What does a wrong answer cost, and is the damage recoverable?
How will we know next month whether it is still working?

Approval Queues Are a Scoping Decision

An approval queue is a hold. The system produces output, a human reviews it, and the output ships only on a yes. It sounds like a compromise. In practice it is what makes an AI project shippable, because it converts an accuracy problem, which is hard to bound, into a throughput problem, which is easy to measure.

Use one wherever output reaches a client, moves money, writes to a system of record, or touches a compliance surface. Skip it where the cost of being wrong is a mildly worse internal suggestion. That judgement comes out of question four, which is why approval design belongs in scope rather than in a later feature conversation.

The queue has to be faster than doing the task by hand, or people route around it and you have built expensive shelfware. Show the reasoning and the source beside the output, and allow edit-and-approve rather than reject-and-redo. Track the approval rate: that number is your quality signal, and eventually your evidence for widening the system's autonomy.

Evals, in Plain Language

An eval is a test set for a system that does not produce identical output every time. Traditional tests assert exact equality. Evals assert properties instead: did the answer cite a source that exists, did it stay inside the required format, did it decline when declining was correct, did a grader score it above an agreed bar.

Building one is cheaper than clients expect. Collect a few dozen real examples from the client's history, covering the normal cases and the awkward ones. Write down what a good answer looks like for each, then run the set before every change you ship. That is a morning of work, and it separates a system you can improve deliberately from one you can only hope about.

Evals also change the commercial relationship. When the client asks whether the thing is working, you answer with the eval pass rate and the approval rate from the queue. Two numbers, both defensible, both trending, instead of an opinion in month four while they decide whether to renew.

Structure the Work as a Pilot

A pilot is the narrowest slice that produces the real output, on real data, end to end, for one team or one queue. It is not a prototype and not a slide deck: it runs, and someone whose job it affects uses it. This is the structure we work in at Kronx, with a working staging link before any full contract.

Choose the slice by frequency and pain rather than by how impressive it looks in a meeting. The highest-volume repetitive decision teaches you more in two weeks than the flagship use case teaches in two months, because volume gives you enough examples to see the failure modes.

Set the decision date and the criteria before the pilot starts: what pass rate on the eval set, what approval rate from reviewers, what time saved, and who decides. Kill criteria agreed in advance are what make a cautious client comfortable starting, because the downside is bounded.

The Trap: Automating an Unmapped Workflow

Before automating anything, write down how the work happens today. Who does it, in what order, what they check, where they deviate from the official process and why. This takes a day or two and consistently finds steps that exist only to compensate for a bug elsewhere, or approvals nobody has needed since a staff change.

Automating an unmapped workflow encodes the confusion and runs it at machine speed. Worse, when output is wrong, nobody can tell whether the system misbehaved or faithfully reproduced a process that was already broken. That ambiguity erodes trust while you resolve it.

A useful rule: if the client cannot produce a written version of the current process within a week, the first deliverable is the map. That is legitimate paid work, and it often reveals that fixing the process removes half the reason for the automation.

The Other Trap: Demo-Grade Shipped as Production

A demo needs to work once, on inputs the builder chose, with the builder watching. Production must work on inputs nobody anticipated, at an inconvenient hour, with nobody watching. That distance is where AI budgets go, and it is almost never discussed at proposal stage.

The gap is made of unglamorous things: error handling and retries, a cost ceiling so a loop cannot spend the client's month in an afternoon, logging you can search when someone asks what happened on Tuesday, a documented way to turn it off, and a defined behaviour for I do not know.

Prompt-only systems are the classic case. They demo well, because the inputs were tidy. Then a document arrives in a different format, or a field is blank, or a source returns an error instead of data, and the output degrades quietly rather than failing loudly. Quiet degradation is the worst outcome, because the client finds it before you do.

What the Scope Document Should Contain

Keep it short enough that the client will read it. Everything below fits on two pages, and every line is there because its absence has sent a project sideways somewhere. Send it before the pilot starts and have both sides sign.

The decision statement, in one sentence, in the client's words.
Every data source, with access confirmed rather than assumed.
Where the human gate sits, and who staffs it.
The eval set, the pass bar, and who wrote the reference answers.
An explicit out-of-scope list, including the good ideas parked for version two.
Kill criteria, and the date the decision gets made.
Ownership of code, accounts and keys, and what handover contains.

Questions We Get Asked

How long should an AI pilot take?: Long enough to run on real data with real users, short enough that the client can absorb the cost of stopping. Most useful pilots cover a single decision for one team, with a decision date and pass criteria agreed before work starts. If a pilot cannot show working output on real inputs, the slice was scoped too wide.
What is an eval in an AI project?: An eval is a test set for a system whose output varies between runs. Instead of checking exact equality, it checks properties: correct format, a real cited source, an appropriate refusal, or a grader score above an agreed bar. Built from real historical examples, it gives you a repeatable quality number before and after every change.
Do AI agents need human approval before acting?: Wherever a mistake is expensive or hard to reverse, yes. Client-facing messages, financial actions and writes to a system of record should pass an approval queue where a person reviews and confirms. Low-cost internal suggestions can run without one. The rule follows from what a wrong answer costs, which should be settled during scoping.
How do I price an AI project when the effort is uncertain?: Price the pilot as a fixed, narrow piece of work with defined output and a decision date, then price the production build once the pilot has shown you the real data and the real failure modes. That keeps the client's initial risk small and stops you fixed-pricing work whose difficulty nobody can see yet.