Eleven pilots is not one production system. We start only where there is an operations team willing to own the result on a Tuesday — and if there is not, we say so in the first meeting rather than the ninth.
What is in it
Five capabilities, and one precondition.
Lineage and ownership
Fixed first. We will not build on a source that has no owner, because a model reading unowned data is an incident waiting for a date.
Evaluation on live data
Against your real distribution before anything is built, not against a curated sample that flatters the result.
Serving and rollback
A route back that has been used, not described. Models are reverted more often than anybody plans for.
Decision logging
Every production decision logged, replayable, and explainable to somebody who has complained about it.
The operations handover
A named team, an on-call rota, and a runbook they wrote. Continuous, not an event at the end.
What we decline
Three of them, stated as terms rather than as principles.
We check the machine on paper
We do not start where there is no owner for the data, no operations team, or a success criterion that cannot be measured after the fact. Those are not preferences. They are the three conditions under which this work reliably reaches nobody, and we have stopped taking it.
Pilot versus production
| In a pilot | In production |
|---|---|
| A curated sample | Your live distribution, including the ugly tail |
| Failure is a worse number | Failure is a wrong answer given to a person |
| Nobody is on call | A named team, with a rota |
| Explainability is optional | There is a complaints procedure attached to it |
| Rollback is restarting the notebook | Rollback is a route back inside the window |
| Done is a result | Done is a Tuesday |
Questions we are always asked
Four of them, answered the way we answer them on the phone.
Do you build models or buy them?
Whichever survives contact with your operations team. We sell no model and own none, so there is nothing for the recommendation to be steered toward.
What if nobody owns the data?
Then we fix that first or we do not start. Lineage and ownership come before anything is built, because a model reading a source with no owner is a production incident with a date on it.
Can you explain a decision after the fact?
Every production decision is logged, replayable and explainable to a reviewer who does not write code. That is a build requirement here, not a feature request.
What if the answer is that you should not use a model at all?
It sometimes is, and we have written it. A rule nobody has to monitor beats a model nobody will own.
Case studies
Programs in this area, described by what they were rather than by who paid for us.