How we stopped hoping our AI was right and started knowing it

Waldo VanderhaeghenProduct-driven Business development
#retail-media#ai#product-management#ads#engineering

The first version of our campaign recommendation pipeline was fast, impressive, and wrong in ways we did not notice for weeks. My team built it for an ads product: every night it writes recommendations for hundreds of ad campaigns, and account managers act on them. We gave the model everything we had, from campaign data to performance history and budget state, and asked it to decide. It decided with confidence, and 42 times it decided wrong: inverted logic, overridden human choices, recommendations that looked plausible and were not. Those were no hallucinations; they were what unbounded AI reasoning looks like in production.

Account managers trust the pipeline now, and at the start they did not. The difference is a harness around the model: explicit gates where we used to trust it, and outputs we evaluate where we used to hope they were right. Most of that harness was built after launch, one failure at a time, by the people who act on the recommendations.

What went wrong

At first we treated the model like a smart analyst, gave it all the data and let it figure things out. The failures were subtle. The model would read a performance metric that stood for a floor, the minimum acceptable threshold, and reason about it as a ceiling. It then recommended the opposite of what the campaign needed. It saw underspend and suggested removing targeting restrictions that an account manager had set on purpose. It recommended adding keywords to a campaign that was already burning through its entire daily budget before noon.

None of this looked broken from the outside. The language was fluent, and the logic held together on its own terms. The model was reasoning in a space where we had never told it where the boundaries were, which made unbounded reasoning a design problem for us to solve, whatever model we used.

Before and after. Before: campaign data went straight to the LLM, which made the call unchecked, and 42 recommendations came out wrong. After: code classifies the campaign first, and the LLM only writes up the result as the recommendation.Campaign dataLLM decidesRecommendationCampaign dataCode classifiesLLM writes it upRecommendationBeforeunchecked42 wrong callsAfterBefore and after. Before: campaign data went straight to the LLM, which made the call unchecked, and 42 recommendations came out wrong. After: code classifies the campaign first, and the LLM only writes up the result as the recommendation.Campaign dataLLM decidesRecommendationCampaign dataCode classifiesLLM writes it upRecommendationBeforeAfterunchecked42 wrong calls
For the diagnosis, code now classifies the campaign first, and the model only writes up a result that is already decided.

What the harness means in practice

A harness, in the sense we use it here, is the set of constraints, gates and checks that make AI output trustworthy instead of hopeful. It leaves the model’s reasoning in place and defines exactly where that reasoning happens and what happens to its output. In our pipeline that means four AI tasks, each with one defined job.

Four scoped tasks

The first task judges whether a campaign’s ads chase one buyer intent or several, then builds a keyword list from real search demand. The second scores keyword relevance: does this term fit what this advertiser sells? The third takes a diagnosis that is already solved, with budget state, performance verdict and historical data, and puts it into plain language. The fourth looks at a dashboard screenshot and answers with a single word: OK, LOADING or EMPTY.

Every output lands in the CVR Optimizer dashboard in front of an account manager, who can read it, question it or override it. The tasks differ in how much the model contributes, and the pattern stays the same for all four: scope defined in advance, output visible, a human override always available. The advertiser summary under the chart in this screenshot is the kind of text the third task writes.

The CVR Optimizer dashboard for one advertiser: a 28-day chart of CTR, average daily spend and CVR against the benchmark and daily budget, tiles for budget, CTR, at-risk campaigns and CPA, and below them a plain-language advertiser summary with recommended actions.

Evals before and after the model

Where the model kept wandering, we replaced it with deterministic code. Where its judgment adds value, we kept it and added evals. These are checks that run before and after the model and verify its output against criteria we defined before looking at any specific result. Writing the criteria first and reading the output second is the whole shift from hoping to knowing.

The checks come in two tiers. The first tier is automated and strict, with structural and threshold-based checks, and a recommendation that fails it is never shipped. Whatever passes goes to manual review against the failure taxonomy.

Every new recommendation passes strict automated checks first. One that fails is not shipped; one that passes goes on to manual review against the failure taxonomy.New recommendationAutomated checksManual reviewNot shippedPassFailEvery new recommendation passes strict automated checks first. One that fails is not shipped; one that passes goes on to manual review.New recommendationAutomated checksManual reviewNotshippedPassFail
The automated tier catches the failures we already know how to describe; manual review is there for the ones we have not named yet.

The failure taxonomy holds five failure categories, fifteen invariants and forty-two catalogued incidents. It is the ledger of every time we learned what “wrong” looks like specifically enough to check for it automatically. Each entry started as a failure somebody noticed, which makes the harness accumulated scar tissue.

Who built it

The most important thing we learned was about people, and it came from the account managers who act on the recommendations. They are the ones who catch the subtle failures. One message read: “This rec told me to add keywords to a campaign that’s already maxing its daily budget.” That message landed in Slack and a new incident was opened. The incident became a taxonomy entry, the entry became an invariant, and the invariant was encoded into the system prompt. The account manager never wrote a line of code and shaped the system as much as any engineer did.

The loop between night and day: the pipeline runs overnight and produces a recommendation, an account manager acts on it and spots a failure, the incident is logged, becomes a taxonomy entry and an invariant, and returns as a new rule in the system prompt for the next run.New rule in promptOvernight runRecommendationAccount manager actsSpots a failureIncident loggedTaxonomy → invariantThe loop, stacked: the pipeline runs overnight and produces a recommendation, an account manager acts on it and spots a failure, the incident is logged, becomes an invariant and returns as a new rule in the system prompt for the next run.Overnight runRecommendationAccount manager actsSpots a failureIncident loggedNew rule in promptInvariant
The pipeline runs overnight; the rules it follows come from the people who use its output during the day.

The pipeline runs overnight, and the discipline that makes it trustworthy is built in daylight. It happens in Slack threads, in incident reviews and in “wait, that doesn’t look right” moments. The people behind those moments have no formal obligation to investigate, and they do it anyway because they want to do their jobs well. The system prompt is 150 lines long and growing, and every line has a story behind it. Most of those stories started outside the engineering team.

Shipped means instrumented

That changed what “shipped” means for us. The first version that ran in production was instrumented rather than finished. The real build happened afterwards, in the loop between the people generating recommendations and the people acting on them. The harness is what accumulated after launch, and no document written before launch could have held it. At the system level this is what co-development looks like: a failure surfaces, it gets a name, the name becomes a rule and the rule gets enforced. The people closest to the output drive that loop, and the engineers encode what they catch.

The horse

There is a reason “harness” is the right word for this. A harness channels the force of a horse while the horse still runs flat out. What it adds is direction, a structure that carries the load, and a way to use what the animal can do.

AI reasoning works the same way. The model produces keyword lists that would take a human analyst hours, spots cross-campaign patterns across hundreds of advertisers at once, and classifies at a scale no team could match. The harness keeps all of that and adds what makes it usable: scoped tasks, checked outputs and visible decisions.

In a previous post about the 5 levels of AI integration I described L4, “Harness-Driven”. At that level AI operates in a closed feedback loop, and humans define the acceptance criteria. The unsolved problem I flagged there was legibility: real L4 has to make sense to someone other than its author. The failure taxonomy is our answer. It is a plain ledger of incidents, categories and invariants, far from elegant, and people who did not write it can read it.

Limiting the model was never the goal. The first version went wrong in 42 ways that took us weeks to notice, and today every one of those incidents has a name in the taxonomy. That is the difference between hoping the model is right and knowing when it is wrong.