The Build Framework

The framework

Seven gates between an idea and a system that earns its keep

Most AI programmes do not fail at the model. They fail because nobody wrote down what would have to be true, and nobody checked. Each stage below has one question, one decision tool, and one gate you either pass or do not.

Qualify

The most expensive sentence in enterprise AI is "let's just try it and see." It sounds like pragmatism. It is usually a way of avoiding the two questions that decide the outcome: what would have to be true for this to work, and how would we know early if it is not. This is a working framework for teams who have to answer both before they spend anything material.

It assumes you are not a research lab. You have a process that costs money, a small number of people who understand it, and a board that will ask what happened to the budget. Every stage here is built to be finished in weeks, not quarters, and every stage can end in a decision to stop.

00 / QUALIFY

Is this an AI problem at all?

Gate: you can state, in one sentence, why a rules engine or a spreadsheet will not do this job.

A meaningful share of what gets scoped as AI is a decision table with better marketing. If a domain expert can write the logic down as if-then rules in a day, the rules engine wins on every dimension that matters in production: it costs almost nothing per call, it returns in milliseconds, it is deterministic, it can be unit tested, and when it is wrong you can point at the exact line and fix it in an afternoon.

An LLM cannot do any of those things. What it can do is handle inputs nobody could enumerate in advance. That is the entire trade, and it is worth paying for only when the input space is genuinely open.

Figure 1 · Approach profile

Three tools, three shapes

Relative strength on each axis. Higher is better. Note that rules and classical ML dominate everywhere except the one axis that usually forces the decision.

Directional. Absolute values vary by workload; the shape of the trade-off does not.

The practical rule: use the cheapest mechanism that clears the accuracy bar. Most real systems end up as a stack, where rules handle the 70% of cases that are unambiguous, a classifier handles the next 20%, and a language model handles the residual that needs judgement. That design is far cheaper than sending everything to a frontier model, and far easier to explain to a regulator.

Decision tool 01

What should actually build this?

Five questions about the work itself. The recommendation updates as you answer.

Q1Could an expert write the decision rules down in under a day?

Q2What does the input look like?

Q3Do you have labelled historical examples?

Q4Does the task need judgement over ambiguous language?

Q5What happens when it is wrong?

Rules engine or decision table0

Classical ML model0

LLM with retrieval0

Recommendation

Answer the questions above

The recommendation and its reasoning appear here.

01 / FALSIFY

Write the hypothesis so it can be killed

Gate: the hypothesis has a number, a date, and a condition under which you stop.

A business case is a claim about the world. Most AI business cases are written so that no observation could ever contradict them, which is why they survive for eighteen months and then quietly disappear. The fix is to write them the way a scientist would: a specific prediction, a measurement method agreed in advance, and a threshold below which you shut it down.

The format that works is one sentence with four slots. We believe that [intervention] will change [metric] from [baseline] to [target] by [date]. We will know we were wrong if [observable condition]. If you cannot fill the baseline slot, stop and go measure it, because a baseline captured after launch is not a baseline, it is a negotiation.

The claim you will hear	What usually turns out to be true	The test that settles it in two weeks
It will save 20 hours a week	Hours are saved but never leave the cost base, so finance sees nothing	Name in advance where the capacity goes: reduced outside spend, more volume, or a role not hired
Accuracy will improve with a better model	Most errors trace to retrieval and data quality, not model reasoning	Take 100 failures and label the root cause. If under a third are reasoning errors, the model is not your problem
We need to fine-tune on our data	The data is not labelled, not consistent, and not enough of it	Try to assemble 500 clean input and output pairs. Most teams cannot inside a fortnight
The pilot proved it works	The pilot ran on curated inputs with volunteers who wanted it to work	Replay it against a random sample of last month's real traffic, including the messy cases
Users will adopt it once it is available	Adoption requires the surrounding process to change, which nobody funded	Count who used it unprompted in week four, not week one
We should build it ourselves for control	Internal builds reach production at roughly a third the rate of bought solutions	Ask what happens to this system when the two people who built it leave

If no realistic observation could kill your business case, you do not have a business case. You have a preference.

Figure 2 · Cost of changing your mind

Kill it early or pay for it later

Relative cost of abandoning or reversing a decision, indexed to the qualify stage.

Directional model. The shape is consistent across software delivery research; AI adds data and retraining cost to the later stages.

02 / DECIDE

Choose the architecture, and write down why

Gate: every material decision has an ADR with the options you rejected and the trigger that would make you revisit it.

There is a default ladder, and the discipline is to climb it one rung at a time rather than jumping to the top because it sounds more serious. Prompt engineering first. Then retrieval. Then a small amount of fine-tuning, if and only if retrieval failed for a reason fine-tuning actually fixes. Agents last, and only where the task genuinely branches.

The long context question

This is the decision most often got wrong right now. Context windows have grown to the point where you can paste an entire policy manual into a prompt, and the obvious conclusion is that retrieval and fine-tuning are obsolete. That conclusion is half right in a way that costs money.

Long context solves the knowledge problem beautifully in a prototype. In production it has three properties that matter. You pay for those tokens on every single call, so what looks free at 200 requests a day becomes the largest line on the invoice at 200,000. Retrieval quality degrades as the window fills, because relevant facts buried in the middle of a very long context are recalled less reliably than facts near the edges. And latency scales with what you send.

So the split is cleaner than the debate suggests. Retrieval is for facts. It handles knowledge that changes, that must be cited, and that is too large or too sensitive to sit in a prompt. Fine-tuning is for form. It handles behaviour you cannot reliably instruct: an exact output structure every time, a house tone, a domain vocabulary the base model does not have, a narrow classification task where you want frontier accuracy at small model cost and latency. Long context is for the session. It handles the document you are working on right now.

The clearest surviving case for fine-tuning

Distillation. You have a task running at high volume, a frontier model does it well, and the unit cost is unsustainable. You use the large model to generate a few thousand high quality examples, fine-tune a small model on them, and run the small one in production at a fraction of the cost and latency. This is an economic decision, not a capability one, and larger context windows do not touch it.

Symptom	Usual cause	What actually fixes it
Makes up facts	No grounding, or retrieval returning the wrong passages	Retrieval, with citation required in the output contract
Answers are stale	Knowledge baked into a prompt or a tuned model	Retrieval over a source of truth you already maintain
Output format drifts	Instruction following limits under long prompts	Structured output constraints first, fine-tuning if it persists
Does not know our vocabulary	Domain language absent from pretraining	Glossary in retrieval first. Fine-tune only if that fails
Too slow or too expensive at volume	Frontier model on a repetitive narrow task	Distil to a smaller fine-tuned model, plus caching
Fails on multi-step work	Single prompt asked to plan and execute at once	Decompose into steps with checks. Agents only if branching is real

Decision tool 02

Prompt, retrieval, fine-tune, long context or agent?

Select every constraint that applies to your workload. Ranking updates live.

Prompt engineering only0

Retrieval augmented generation0

Long context, no retrieval0

Fine-tuning or distillation0

Agentic workflow0

Recommendation

Select your constraints

Start with the cheapest approach that clears your accuracy bar. Nothing here rewards sophistication for its own sake.

Architecture decision records

An ADR is one page, written when the decision is made rather than after. Its value is not documentation. It is that six months later, when the model you chose is deprecated or the cost profile changes, you can tell the difference between a decision that should be revisited and one that only feels uncomfortable. The revisit trigger is the field most teams omit and the one that does the work.

Decision tool 03

Draft an ADR

Fill this in for each material choice: model, retrieval strategy, human oversight, hosting, data boundary.

Decision

Context and constraints

Options considered and why rejected

Consequences we accept

Revisit trigger

03 / EVALUATE

Build the test before you build the system

Gate: a frozen golden set exists, someone who is not the builder owns the score, and the pass mark was set before the first result came in.

Teams that ship reliable AI have one habit in common. They write the evaluation set first, from real historical cases, including the ugly ones, and they freeze it. Then any change to prompt, model, retrieval or chunking is judged against the same fixed target. Without that, you are tuning against yesterday's anecdote and calling it progress.

Four layers, and you need all of them. Each catches what the one before it misses.

Layer	What it is	Catches	Cadence
Golden set	100 to 500 frozen real cases with agreed correct answers	Regressions from any change	Every pull request
Model as judge	A separate model scoring outputs against a rubric	Quality drift at volumes humans cannot review	Nightly on a sample
Human review	Experts scoring a random sample, blind to version	What the rubric missed and the judge rewarded	Weekly, then monthly
Online signals	Override rate, escalation rate, task completion, rework	The gap between scoring well and being useful	Continuous

Two rules make this cheap. Never let the person who wrote the prompt own the score. And keep a small held-out set that never appears in any prompt or example, because the fastest way to a great evaluation number is to accidentally teach the system the test.

04 / SHIP

Deploy in a way that lets you be wrong safely

Gate: there is a defined rollback, a decided autonomy level, and a named person who can switch it off.

The rollout pattern that works is boring and sequential. Run in shadow first, where the system produces an answer that nobody sees and you compare it to what the human did. Then canary on a small slice of live traffic with an easy path to a human. Then default, with opt-out. Then remove the human, if the numbers earned it. Teams that skip to step three explain themselves to an executive committee later.

Autonomy is not one setting. It is a function of how bad a wrong answer is and how easily it can be undone. Those two axes give you a grid, and every workflow belongs in exactly one cell.

Decision tool 04

How much human oversight does this need?

Select the cell that matches your workflow.

Easily undone

Costly to undo

Irreversible

Low impact

Material impact

Severe or regulated

Oversight design

Select a cell

Each cell carries a different operating cost. That cost belongs in the business case, not in a later surprise.

One warning on the human in the loop. A reviewer who approves 500 items a day is not providing oversight, they are providing a signature. If you need genuine review, you need to sample rather than gate everything, or you need to route only low-confidence cases to a person. Design for the reviewer's attention as carefully as you design the model.

05 / OPERATE

Assume it degrades, because it will

Gate: drift monitors are live, an owner is on call, and a retraining or re-prompting path is rehearsed rather than theoretical.

Traditional software fails loudly. AI systems fail quietly, keep returning confident answers, and the degradation shows up in a business metric weeks before anyone connects it to the model. Four things move underneath you.

Data drift. The inputs change. New product lines, new document templates, a new customer segment writing differently.
Concept drift. The right answer changes. A policy is updated, a threshold moves, a regulation lands. The model is now confidently correct about last year.
Provider drift. The vendor updates or deprecates the model underneath you. Behaviour shifts without a deployment on your side.
Prompt and index rot. Someone adds one more instruction, the corpus grows, chunking that worked at 10,000 documents stops working at 100,000.

Figure 3 · Quality over time

Unmonitored systems decay. Monitored ones sawtooth.

Golden set score across a year. The lower line is a system nobody watched. The upper line is the same system with drift alerts and scheduled intervention.

Illustrative. The pattern, not the values, is the point: intervention is cheap and frequent, or expensive and late.

What to instrument, in rough priority order: the golden set score on a schedule, input distribution against the training or design baseline, retrieval precision, refusal and fallback rate, human override rate, latency at the 95th percentile, cost per successful task, and the business metric from your stage 01 hypothesis. That last one is the only one that tells you whether any of it matters.

Cost, and how to actually reduce it

Unit prices keep falling while invoices keep rising, because consumption grows faster than price drops. The levers below attack different parts of the bill and stack reasonably well. Toggle them to see the compounding effect.

Decision tool 05

Cost optimisation, stacked

Enter your monthly inference spend and the value the workflow returns. Then apply the levers you can realistically implement.

Monthly inference spend ($)

Monthly value delivered ($)

Annual spend today

Annual spend with levers applied

0.0x

Value returned per dollar of opex

Illustrative planning model. Savings apply to the remaining spend after each prior lever, so they compound rather than add. Validate against your own token telemetry before committing to a number.

06 / PROVE

Make it legible to the people who fund it

Gate: a named executive can state, without help, what this system costs, what it returns, and what would make them stop it.

Initiatives do not usually get cancelled because they failed. They get cancelled because nobody could explain them at the moment the budget got tight. Leadership visibility is not a status deck. It is a small number of figures that stay in the same place every month, including the ones that look bad.

Business metric from the hypothesis

Baseline → now

The single number this was funded to move, against the pre-launch baseline

Cost per successful task

$ / task

Not total spend. Total spend rising while unit cost falls is a healthy system

Value per dollar of opex

Ratio

Benchmark is roughly $3.70 returned per dollar across studied adopters

Human override rate

% and trend

Rising override means quality drift or scope creep. It should fall over time

Golden set score

% vs pass mark

The quality number that does not move just because usage moved

Adoption in week four

Active users

Week one measures curiosity. Week four measures whether the work changed

The cadence that holds up: weekly for the delivery team on quality and cost, monthly for the sponsor on the six figures above, and quarterly for the executive committee on the portfolio, where each initiative is explicitly labelled as efficiency, growth or capability building. Those three are judged differently, and merging them is how a learning project gets killed for failing an ROI test it was never given.

One habit worth borrowing: publish the kill list. Say out loud which initiatives were stopped and why. Organisations that do this can start things quickly, because stopping is normal rather than an admission of failure. Organisations that cannot stop anything eventually cannot start anything either.

07 / THE GATE

Is this actually done?

Twenty-one items. Anything unchecked is a decision you have deferred, not avoided.

Decision tool 06

Production readiness

Tick what is genuinely true today, not what is planned.

0 of 21 complete

Start ticking to see where the gaps are.

Selections are not stored anywhere.

The shortest version

Use the cheapest mechanism that clears the bar. Write the hypothesis so it can be killed, and capture the baseline before you start. Climb the architecture ladder one rung at a time and record why. Freeze the evaluation set before you build. Ship into shadow, then canary, then default. Assume it degrades and instrument accordingly. Report the same six numbers every month, including the ones you would rather not.

None of this is about the model. It never was. Roughly 70% of the value in these programmes comes from people and process, about 20% from data and technology, and about 10% from the algorithm itself. Most budgets are allocated in almost exactly the reverse order, which is a reasonable explanation for most of what has gone wrong so far.

Sources and benchmarks referenced: MIT Project NANDA, The GenAI Divide (2025) on build versus buy success rates and pilot-to-production cycle time; BCG on the 10-20-70 allocation and the AI value gap; IDC and Microsoft, The Business Opportunity of AI, for the $3.70 per dollar benchmark; McKinsey State of AI 2025 on adoption against EBIT impact; Gartner on agentic project cancellation drivers; FinOps Foundation State of FinOps 2026 on cost overruns. Figures marked illustrative or directional are modelled for planning purposes and should be replaced with your own telemetry before any commitment.