The quality gate: 27 checks between the model and the prospect

Asked to pick the best image for a storm-damage page, the model returned another flood. It was the best flood in the batch. It was still wrong.

That is the whole problem with asking a model whether its own output is good. A ranker answers a relative question — which of these is best — and it always answers, because there is always a best one. A gate answers an absolute one: is this good enough to send a stranger who might pay us. Different jobs, and I learned it the hard way.

I wrote about the Prototype Engine yesterday (coltsneckinteractive.com/blog/2026/08/03/the-prototype-engine). The quality gate got three paragraphs there. It deserves more, and a correction.

A grader that grades on a curve is not a gate

The fix was structural, not a better prompt. Image selection is now pick-then-validate: a multi-image call proposes a candidate, then a separate single-image call judges that photo, full size, against the page topic. A failed pick is dropped and the next-best tried, up to two rounds; if nothing passes, the caller tries a different query.

That generalization matters more than the fix. Use the model for ranking, where relative judgment is the point. Do not use it for pass/fail, where you need a standard that does not move with the batch. Draw that line and "can AI check AI" becomes an ordinary engineering question: which of these decisions can I write in code.

Drawing that line put 27 checks into code and left ranking to the model. The gate is a pure function over the render snapshot — no network calls, no model calls — so it runs on every build, on every repair, and again at the moment of sending. Same snapshot in, same verdict out.

Every check is a defect that already shipped

The rule sits in a comment at the top of the gate file: each check is the codification of a defect class we shipped once and fixed, and new defect classes found in review get a check there first.

That rule does more work than it looks like. The list is a record of real incidents, not of things I imagined going wrong. One gallery shipped a Google logo, a Facebook logo, and a pricing flyer classified as photographs, and the flyer became the hero image. That is why there is a check called gallery-junk. Nobody writes that one from first principles.

It also means the number moves. Yesterday's post said fourteen checks. When I started this one the gate defined 26; a commit this morning made it 27. Any single build runs 20 to 25 of them, because some checks apply only when the prospect's site has the thing being checked. Every one of those numbers was true when written. A fixed count of checks is a snapshot of an incident log.

Fail-open and fail-closed are per-check decisions

Not every check deserves the power to stop a build. Gallery curation fails open, deliberately and in writing — the comment reads "fail-open — a hiccup must not empty their gallery." An image the fetcher cannot reach is kept: unreachable is not the same as junk. The vision call retries once, because an early batch came back empty and ten junk images sailed through.

The logo check fails closed, and so does the gallery-junk check that catches what curation lets through. Across 31 completed builds, three carry a failing verdict: two logo-real, one a .gif that reached a gallery. A slightly worse gallery is a worse prototype. A prototype with somebody else's logo on it is not their business.

The rule I would give someone else: fail open where the degraded state is still recognizably their site, fail closed where it is embarrassing. Decide per check, and put the reason in the code.

The gate has to sit where something is at stake

A gate that only advises is a linter people learn to mute. Ours is enforced at the send boundary, where the pipeline does something it cannot take back: a real person gets a real email.

The send endpoint re-runs the gate against the exact snapshot about to be published. Any failing check returns an HTTP 409 naming the check IDs, and the send proceeds only with an explicit override.

Its own comment says why: builds carrying hard failures used to sail through, because nothing on the send path consulted the gate. The gate file landed July 18. Blocking at the send boundary landed July 31. For thirteen days the gate recorded every send and stopped none. Reporting is not enforcing.

Twenty-eight prototypes carry a send timestamp, the earliest July 18, and every one has a stored gate report. Verdicts run 11 pass, 15 warn, 2 fail. Warn does not block, and none of the 15 warn runs holds a failing check; they carry advisory flags, an image used three times, a tool-parity note. Of 622 checks across those sends, two came back fail.

Both of them went out. Of the three failing builds, one never left the console; the other two are live. Somerset Family Dentistry went August 3 — "placeholder/non-http: /brand/logo-light.svg". West Washington went this afternoon, same check, same placeholder, while I was drafting this post. Both headers show a brand-colored dot beside the practice name in type: clean, professional, and not their logo. I could not tell from the database how either got past the gate, so I read the send path line by line.

The enforcement is one if-statement in one route

The gate is enforced in exactly one place: a single if-block in the send-to-lead route. That block is skipped whole when the request carries override: true — and the console supplies that flag itself. The button POSTs override: false. On a 409 it raises one window.confirm with the failing check IDs in it, and one OK click re-POSTs with the flag flipped.

Nothing records that. The route writes no column, no log line and no note when the flag is set, and there is no audit table to check instead. Because the skip wraps the whole block, an overridden send never computes the list it overrode. So I cannot tell you whether a human read that dialog and accepted the risk on either dentist. Neither can the system. A control you can bypass has to write a row when it is bypassed, or an override and a silent bug become the same event in your data.

The block sits in a try whose catch is empty. If the snapshot will not read, or the gate throws on a field it expects, the exception is swallowed and the send continues. I argued fail-open versus fail-closed per check, wrote the reasons into the code, then put the whole enforcement inside a catch that fails open for all 27 at once. The default that governs a system is the one in the error path.

The gate covers one of three ways a prototype reaches a prospect. The "copy link" button marks a lead sent through a sibling route that stamps the timestamp and arms the follow-up sequence, ungated — so a run reading "sent" is not evidence the gate ran on it.

Follow-ups have no gate at all: at three, seven and fourteen days the system re-emails the same URL. Twenty-eight have gone out across 23 runs, and both failing prototypes were armed for one. Stopping the first took a note hand-typed into a sales field, because there is nowhere else to put it. I gated the route I was thinking about. The boundary is every path that ends at a stranger.

One more, because it changes what the numbers above are worth. Somerset's snapshot and stored report were both written 2 hours and 51 minutes after its send. "Every send carries a stored report" is true. It is not the same sentence as "every send was checked."

What the gate does not catch

The nav route check used to skip child menu items, justified in the code: no theme renders dropdowns, verified against live deploys. True on July 18. On August 3, 123 commits later, a mobile nav shipped that renders every child, and links that had never been tappable became 404s on phones. The bug was not in the content but in an assumption inside the check, where running it could never surface it. A justification of the form "nothing renders this" expires the moment something does.

There is more I cannot claim. Seventeen of the 26 check IDs in the stored reports have never returned anything but pass, and I cannot tell whether they guard something or are vacuous: the repair loop runs before the report is saved, so a check that caught a real problem reads as a pass afterward. The unit tests are the only evidence those seventeen do anything — 51 cases, 82 assertions.

The gate is offline by design, which is its ceiling: it validates the snapshot, not the deployed page.

This is half-automated on purpose. The gate decides what is broken; a human decides what to do about it. I still think that is right — a block you cannot get past is one someone deletes. But the human's half has to leave a record, and it leaves none. That is the next commit.

Every prototype we have sent carries a stored report from this gate, and two of those reports carry a failing check the prospect got anyway. If you want to see what comes out the other end for your business, ask us for a preview.