Who watches the builders? What implementation oversight actually looks like

The AI implementations that fail rarely fail on bad code. I’ve been on the delivery side of this for years — at Last Rev, the platform engineering firm I co-founded, and before that inside WebMD, Clickability, and Optimizely — and the builds that die usually die of something quieter: unsupervised scope.

Here’s the mechanism. The builder is optimizing for shipping, because that’s what they’re paid for and how they’re judged. The buyer can’t tell a shortcut from a feature, because both look like a working demo. And nobody in the room is paid to say no. Every individual decision is small — reuse this account, skip that edge case, add this nice-to-have while we’re in there — and each one is invisible to the client until the sum of them shows up as a system that technically works and can’t be trusted.

That’s the gap oversight fills. Not code review. Judgment review.

Why the strategist can’t be the builder

The first rule I hold myself to is structural: the person setting AI strategy cannot be the person selling the implementation. Not because implementers are dishonest — the good ones aren’t — but because an implementer’s recommendation carries a sales quota whether they want it to or not. When the person who decides what to build also bills for building it, “should we do this?” and “can we sell this?” collapse into one question. The client hears advice; the advisor’s firm hears pipeline.

The industry numbers show what happens when nobody independent owns outcomes across a build. S&P Global found the share of companies abandoning most of their AI initiatives jumped from 17% to 42% in a single year, with an average of 46% of proofs of concept scrapped before production. Gartner had already predicted 30% of generative AI projects would be abandoned after proof of concept by the end of 2025. And MIT’s NANDA research put the bluntest number on it: 95% of generative AI pilots deliver no measurable P&L impact.

Those aren’t stories of incompetent engineers. They’re stories of builds where scope, acceptance, and go-live were decided by whoever had the strongest incentive to keep building. The abandonment happens at the end because nobody was empowered to say no in the middle.

Three no’s in one month

So what does oversight concretely look like? Not a weekly status call where the builder presents green dashboards. It looks like documented pushback. Here are three examples from a single month of an engagement I’ve written up as a sample brief — the client in it is a fictional composite accounting firm, but all three moves are ones I’ve made for real.

No to the service-account shortcut. Mid-build on a document workflow, the builder proposed connecting to the document store through a single service account with broad read access. It’s the standard shortcut: one credential, no permission plumbing, a week saved. I rejected it and required per-user permission passthrough, which cost four extra days. Non-negotiable, and here’s why: with a service account, the system can retrieve any document for any user, and the access controls the firm spent years maintaining stop existing the moment a query goes through the tool. That’s not a technical nuance. That’s the difference between governed retrieval and a data leak with a user interface. No demo distinguishes the two — both return the right document. Only someone reading the architecture on the client’s behalf catches it.

No to the scope add, even a good one. The builder proposed folding e-signature chasing into the current phase: +$9K, +3 weeks, genuinely useful. I recommended deferring it to the next version. Not because the feature was wrong, but because go-live timing beat feature completeness — the workflow’s value was in shipping before the fall deadline crunch, and every week of delay burned real hours the automation existed to save. Builders propose scope adds in good faith; adds are interesting to build and easy to justify one at a time. Someone has to own the ship date the way the builder owns the feature list.

No to paying for the builder’s miss. A piece of delivered work didn’t match the statement of work and needed rework. The builder positioned it as evolving requirements. I held them to the fixed SOW: their miss, their cost, $0 change order. And here’s the part I want to be straight about — the builder in this example is Last Rev, the firm I co-founded. Which is exactly the point. If the oversight only worked when the advisor had no relationship with the builder, it wouldn’t be oversight, it’d be luck. The reason I can hold my own firm to a $0 change order without a fight is that the SOW, the acceptance criteria, and the decision log are written down where the client can read them. Documentation is what makes the arrangement honest instead of merely convenient.

None of these three required writing code. All three required knowing what shortcut looks like from the inside, and being paid by the client — only the client — to say so.

Gates a managing partner can verify

The other half of oversight is structural: go/no-go gates with acceptance criteria, agreed before the build starts.

The trap with technical acceptance criteria is that they’re technical. “Retrieval respects source ACLs” is true or false, but the client can’t check it, so it operates on trust — which recreates the original problem. So I write every gate as a sentence a managing partner can verify personally. Not “permissions are enforced,” but: log in as a staff accountant, ask for a partner-compensation document, confirm the system returns nothing. Not “output quality is acceptable,” but: two partners review ten drafts against the checklist and sign the sheet. Not “the system is performant,” but: a draft comes back in under two minutes during a normal Tuesday.

Gates like that change the power dynamic of the whole build. The builder knows from day one what “done” means and that it can’t be softened in a status meeting. The client knows they can walk the acceptance test themselves without hiring a second engineer to check the first one. And when a gate fails, the conversation is about a sentence everyone agreed to, not about whose interpretation of “working” wins.

A go/no-go gate is the client’s protection, but it protects the builder too. The $0 change order I described above was quick and non-toxic precisely because the SOW’s acceptance criteria were unambiguous. Nobody had to litigate intent. The sentence either passed or it didn’t.

Direction you can read

My operating model is “I direct, builders build” — I don’t take implementation revenue, and the firms that build are accountable to a plan I own on the client’s behalf. But I’ve come to believe that sentence only means something if the direction is documented and the client can read it. Undocumented oversight is just another person in the meeting.

So the artifact matters as much as the activity: a monthly brief that records what was reviewed, what was rejected and why, what it cost, and what gate comes next. The sample Implementation Oversight Brief shows one month of this for a composite client — the service account, the scope deferral, the change order, all of it — and there’s more at /examples.

If you’re paying for an AI build right now and can’t point to a document like that, it doesn’t mean your builder is bad. It means every scope decision in your project is currently being made by the party who profits from it. Given where the abandonment numbers sit, I wouldn’t run that experiment on my own money.