Faros AI looked at telemetry from about 10,000 developers and found that teams with high AI adoption merged 98% more pull requests than before. Their delivery metrics stayed where they were. Separately, METR ran a controlled trial with experienced open-source developers. With AI tools, they took 19% longer to finish their tasks. Asked afterwards, they estimated AI had made them 20% faster.

None of this means AI is useless. Plenty of teams get real value from it. What it does mean is that the usual dashboard can make AI look like a success well before anyone knows whether it is one. A jump in PR count is a reason to start asking questions.

This guide is written to be printed and kept on the table during metric reviews. Go through the checklist before the meeting. When a number moves on screen, find it in the cheat sheet, see which diagnosis it fits, and use the questions listed there. There's a tracker at the end for follow-ups.

If you only remember one thing: PR counts and other activity numbers react to AI within weeks and are easy to inflate. Delivery and quality numbers take longer to move, and they're the ones that tell you whether anything improved. Read them together, every time.

Before the meeting

Five checks. If the answer to any of them is no, ask for the missing piece before anyone starts drawing conclusions.

• [ ] Is each team being compared with its own numbers from before AI, rather than with other teams?

• [ ] Does the data cover at least eight weeks? A single sprint is too noisy to read.

• [ ] Are reorgs, migrations, hiring waves, big incidents and holidays marked on the timeline?

• [ ] Are new features, maintenance and legacy work shown separately?

• [ ] Does every activity metric, like PR count, have a quality or delivery metric next to it?

Cheat sheet

One row per metric. The last column is the question to ask when that metric comes up.

Terms

It saves time if everyone in the room uses these the same way.

• Rework: code that gets rewritten or reverted within roughly three weeks of being merged.

• Review participation: the share of PRs reviewed by at least one person other than the author.

• Escaped defects: bugs found after release, by users or in production, rather than in review or testing.

• Ease of delivery: how easy developers say it is to get a change into production. This is different from how easy it is to write the code.

• AI-assisted PR: a PR where a meaningful part of the code came from an AI tool, either tagged by your tooling or declared by the author.

• Guardrail metric: a quality or delivery metric that shows whether the extra activity is costing you somewhere else.

What to expect

A few things are worth saying out loud before the numbers come up, because they set the tone for the whole review.

Most teams don't start out in good shape with AI. The common early picture is more activity and flat delivery, or more activity with quality slipping a bit. That's a normal stage and shouldn't be treated as failure.

There's usually a dip before any gain. Teams need time to figure out where AI is useful and to adjust how they review and release. If delivery metrics improve at all, it often takes two quarters or more.

Expect mixed signals. One metric getting better while another gets worse is the usual state of things. Your job is to work out which one matters more for that team.

The same tool can help one team and hurt another. AI tends to do well on new, self-contained work and badly on large, old, tightly coupled codebases.

If a team's AI report is all good news in the first quarter, that's a reason to look harder.

Which diagnosis fits?

1. AI is helping

What you see. PRs go up and PR size stays about the same. Review time and review participation hold. Builds stay healthy and rework stays flat. A quarter or two later, lead time drops and deployment frequency goes up, while failure rate and time to restore stay steady. In surveys, developers say getting changes out got easier, and they mean shipping, not just typing.

What's usually going on. The team already had the basics in place: small changes, fast CI, decent tests and enough people reviewing. AI added speed, and the team's process could take it.

Ask before you believe it.

• Did anything else change in the same period? A strong new hire, a migration that finally finished, a quiet quarter?

• Is the gain all in one kind of work, say new services, while maintenance work stayed the same?

• Is review time steady because reviewers are keeping up, or because they've started skimming?

• Does it hold for a second quarter?

What to follow up on. Find out what this team does differently and write it down. Try it with one neighbouring team and see if the same pattern shows up. Keep an eye on rework and escaped defects, because a team can look healthy now and drift later.

2. Busy but not faster

What you see. PRs go up. Lead time and deployment frequency don't move. Failure rate, rework and build health all look fine. Developers say writing code feels quicker but getting a release out feels the same as before.

What's usually going on. Writing the code was never what slowed this team down. Changes now sit waiting somewhere else, usually in review, in test environments, in release approvals or on another team. AI made a step faster that wasn't the bottleneck.

Ask.

• Where does a typical change spend most of its time between commit and production?

• How long does a PR wait before anyone starts reviewing it?

• How many approvals and manual steps does a release need?

• Where did the time AI saved go?

What to follow up on. Take three to five recent changes and trace each one from first commit to production, noting every place it waited. Fix the longest wait before spending more on AI tools. Decide on purpose what the saved time should be used for. If nobody decides, it turns into more work in progress, and the waits get longer.

You'll know it's working when lead time starts to fall and PR size and failure rate stay where they were.

3. Faster but breaking

What you see. PRs go up, often by a lot. PRs also get bigger and harder to review. Review time climbs, or review participation drops and PRs merge with barely a look. Build success drops, or builds get slower. Rework rises. Change failure rate goes up, and time to restore may go up with it. Reviewers say they're swamped, and burnout starts showing up in surveys.

What's usually going on. Producing code got cheap. Reviewing, testing and releasing it didn't. The team is putting out more than its process can safely handle.

Ask.

• Which PRs this month were too big to review properly?

• What got merged without a real review by a person?

• Which recent incidents or rollbacks involved AI-assisted code?

• Who is doing most of the reviewing, and what have they stopped doing to keep up?

• Does anyone feel pressure to show more output now that AI is available?

What to follow up on. Put a size limit on PRs and split the big ones. Protect reviewers' time and spread reviewing across more of the team. Tighten tests and CI checks, and roll out risky changes gradually or behind feature flags. If any target rewards PR count or AI usage, drop it.

You'll know it's working when PR size and review time come down first. Failure rate and rework usually follow within a quarter.

4. Looks good but isn't

This one is hard to spot, because the headline numbers all look fine.

What you see. Review time goes down, test coverage goes up, and people are happier. Underneath, rework is creeping up, a few more bugs are reaching users each month, duplicated code is growing, and security findings or leaked secrets are starting to turn up in AI-assisted code.

What's usually going on. The quality checks are passing without really checking. Reviews have become lighter, or a review bot has taken over and people accept its approval. AI is writing tests that run the code without testing much. Scanners catch the simple problems and miss design and access-control issues. Delivery hasn't suffered yet, but the cost is building.

Ask.

• Why did review time drop? Are people reading the code or just approving it?

• Do our tests catch real bugs, or do they mainly push coverage up?

• How much of our review is done by a bot now, and do people approve after it without looking themselves?

• Is anyone still refactoring?

• Who reviews AI-assisted code for security problems that scanners won't find?

What to follow up on. Pair each quality metric with an outcome: review time with review participation, and coverage with escaped defects. Pick a few recently approved PRs and read them to see how careful the reviews were. Plan refactoring as real work. Add secret scanning, and have a person review sensitive AI-assisted changes for security.

You'll know it's working when escaped defects, rework and security findings stop rising while review participation stays high.

5. Slower, and nobody notices

What you see. Lead time goes up, or tasks take longer to finish. PR counts are flat or a little higher. Developers still say AI makes them faster.

What's usually going on. On complex, mature code, checking and fixing AI suggestions can take longer than writing the code by hand. Typing feels faster, so people don't notice the time going into reviewing, correcting and explaining context to the tool. This is the pattern METR found.

Ask.

• Which parts of our codebase does AI handle well, and where does it struggle?

• How much time goes into correcting AI output or re-prompting?

• Is the slowdown mostly in one codebase or one kind of work?

• Could someone here say AI slowed them down without it being taken as resistance?

What to follow up on. Compare lead time by type of work. Agree as a team where AI is useful and where it's optional. On the hard codebases, improve the context AI has to work with, such as documentation and code structure, before expecting gains.

You'll know it's working when lead time on that work gets back to where it was, or better.

Things you'll hear, and what to ask back

When to look closer

These numbers are rough starting points, not findings from a study. Adjust them to each team's normal ups and downs.

• PR size up by more than 25% for four weeks or longer

• Median review time double what it was before AI

• Fewer than 90% of PRs getting a human review

• Rework rising two months in a row

• Change failure rate up by a few points over a quarter

• Lead time rising for two months while PR counts go up

• Any rise in leaked secrets or security findings in AI-assisted code

For teams that don't deploy often, failure rate jumps around because the numbers are small. Give it a full quarter before reacting.

What not to do

• Don't use these metrics to rank individuals. They describe teams and systems, and once people know they're being scored personally, the data stops being honest.

• Don't set targets for AI usage. People will hit them, and you'll learn nothing about value.

• Don't compare teams with each other. Their codebases and work are too different. Compare each team with its own history.

• Don't judge on one sprint. Wait for a pattern over eight to thirteen weeks.

• Don't average DORA metrics across the whole organisation. They're meant to be read per service or per team.

• Don't treat bad news about AI as resistance. If it isn't helping a team, you want to know.

If you can, separate AI-assisted work

Some tools tag AI-assisted PRs, or you can ask authors to declare them. If you have that information, compare AI-assisted and other PRs within the same team on:

• PR size

• review time and review participation

• rework

• failures and escaped defects traced back to each group

If the AI-assisted PRs turn out to be much bigger, get rewritten more often or show up more in failures, you know where to focus. The tags won't be perfect, since people don't label consistently, so read the comparison as a rough guide.

Worked example

Here's one team's first quarter after rolling out an AI coding assistant.

• PRs merged: up 60%

• Median PR size: up 80%

• Median review time: from one day to three

• Deployment frequency: no change

• Change failure rate: from 8% to 13%

• Survey: developers say coding is faster, reviewers say they're overloaded

The team's report leads with "PRs up 60%." Looking it up, it's diagnosis 3, faster but breaking, and both PR size and review time are past the "look closer" thresholds. The team is writing more code in bigger chunks, reviews take three times as long, and more changes fail in production. Users aren't getting anything faster.

In the review, the Director asks which PRs were too big to review, who is carrying the review work, and which of the failures involved AI-assisted code. The fix doesn't involve cutting back on AI. The team adds a PR size limit, sets aside reviewer time and tightens its CI checks. Next quarter, the first thing to look for is PR size and review time coming down. After that, failure rate should settle and lead time should start to fall.

Who watches what

Questions to ask yourself

1. If AI disappeared tomorrow, which of my metrics would actually get worse? If I can't answer, I haven't measured what AI is doing yet.

2. Am I reporting activity or delivery? If PR count is the first number on my slide, I'm reporting activity.

3. What would convince me the rollout isn't working, and is that on my dashboard? If failure can't show up, I can't prove success either.

4. Can my teams tell me AI isn't helping without worrying about how it looks? If they can't, the surveys and status updates are already slanted.

5. Where did the saved time go? If it all turned into more scope, the team isn't faster. It just has more work.

For Directors: what your VP will ask

Have an answer ready for each of these.

• Is AI helping us deliver to users faster, or only helping us write code faster?

• Which teams are getting value from it, and what are they doing differently?

• What has it cost us in quality, review load or incidents?

• What are we changing next quarter, and how will we know whether it worked?

• Can we justify what we spend on AI tools with delivery results rather than usage numbers?

Follow-up tracker

Closing

AI speeds up whatever delivery process a team already has, good or bad. Teams with small PRs, fast feedback and careful review can turn that into faster delivery. Teams without them tend to end up with longer review queues, more rework, and sometimes a slowdown that nobody notices.

So when a team brings good news, ask the same questions you would for bad news. Keep the quality and delivery numbers next to every activity number, work out which diagnosis fits, and keep asking until you get a specific answer.

Which of these five do you run into most often? Tell us in the comments.

References

• DORA, State of AI-assisted Software Development 2025. Google's long-running research program on software delivery.

• METR, Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity. Independent randomized controlled trial; the source of the 19% slower and 20% faster-perceived figures.

• Faros AI, The AI Productivity Paradox (2025). Vendor telemetry across about 10,000 developers; the source of the 98% figure. Treat the exact numbers as directional.

• GitClear, AI Copilot Code Quality (2025). Vendor analysis showing more duplicated code and less refactoring. Also directional.

The worked example and the thresholds in "When to look closer" are illustrations, not data from a particular company or study.

"What changed for the business?"

It usually arrives in budget season. The team has been working hard. The slides show tickets closed, services migrated, pipelines onboarded.

And the room goes quiet, because activity is not evidence.

If you can't explain the value in budget season, the organization will eventually treat the work as optional, no matter how hard the team worked.

If you're a director, or about to become one, this is for you. At that level the job quietly changes. You're no longer judged only on delivering the work, but on making the case for it.

The Market Data Is Blunt

Gartner's 2026 CIO and Technology Executive Survey found that only 48% of digital initiatives meet or exceed their business outcome targets. The same research shows 57% of CIOs under pressure to improve productivity and 52% under pressure to reduce costs. Yet only 33% consistently pursue financial outcomes from their technology initiatives, even though those who do are 25% more likely to excel.

AI has made the gap more visible, not less.

PwC's 29th Global CEO Survey (4,454 CEOs, January 2026) found that 56% of CEOs have seen no significant financial benefit from AI to date. Only 12% report both lower costs and higher revenue.

BCG's Where's the Value in AI? study of 1,000 executives found that 74% of companies had yet to show tangible value from AI. The reason matters more than the headline. Roughly 70% of the challenges were people and process, 20% technology and data, and only 10% the algorithms.

The bottleneck is rarely the technology. It's whether anyone can show, in terms other leaders trust, that the change mattered.

Value Is Lost Before the Work Starts

In McKinsey's research on organizational transformations, respondents said nearly a quarter of a transformation's value loss happens during target-setting. That's before anything ships. Another 55% is lost during and after implementation.

Many programs don't lose credibility in the budget review. They lose it months earlier, when work starts without a baseline. By the time someone asks what changed, the only answer available is memory, and memory doesn't survive a finance review.

Tip: Before any transformation work begins, write down today's number for the thing you claim you'll improve. A rough baseline beats a perfect recollection every time.

A note on a statistic you've probably seen: "70% of transformations fail." McKinsey's own write-up attributes that figure to John Kotter's estimates rather than a single clean measurement. It's worth retiring. If you're asking finance to trust your numbers, hold your own sources to the same standard.

ROI Is a Test of Executive Legibility

ROI isn't mainly a finance exercise. It's a test of whether your work makes sense to people outside your team.

Transformation creates value through reliability, speed, lower cost-to-serve, and lower risk. None of it becomes visible on its own. When a team reports activity, everyone else has to infer the business meaning. In budget season, nobody infers generously.

Strong ROI communication isn't about protecting funding. It's about proving the transformation is real enough to steer.

A Practical Structure That Holds Up

Programs that keep their funding tend to share a few unglamorous habits.

Use Five Value Categories, Not Fifty Metrics

A compact value model is easier to defend than a crowded dashboard. One workable set:

1. Time-to-market — how long an idea takes to reach a customer.

2. Reliability — incidents, recovery time, customer-facing disruption.

3. Cost-to-serve — what it costs to run a service per unit of usage.

4. Risk reduction — exposure removed, defined concretely.

5. Adoption — whether people actually use what was built.

Every metric should roll up into one category. If it doesn't, it probably belongs off the executive view.

Cost-to-serve is becoming non-negotiable. Flexera's 2026 State of the Cloud report found that use of unit economics rose from 40% to 49% of organizations. Over the same period, estimated wasted cloud spend rose to 29%, the first increase in five years, which Flexera links to growing AI workloads. Finance is learning to ask "cost per what?" Have the answer ready.

Capture Baselines Before the Work

This is the cheapest high-value habit available.

A baseline changes the shape of the funding conversation. Without one, the pitch is "trust us, it's better." With one, it becomes "here's where we started, here's where we are, here's what we still don't know."

That shifts the question from "should we keep funding this?" to "what should we do next?"

Give Every Metric a Named Owner

A metric without an owner is decoration.

At director level, that owner is usually one of your managers, not you. When one person is accountable for each number, the data tends to get cleaner, the story sharper, and stale metrics get noticed faster.

Pair Every Number With a Sentence

Numbers rarely explain system change on their own.

A short narrative next to each metric closes the gap: What changed. Why it matters. What's still uncertain.

The uncertainty line feels risky. It's often the part that earns the most trust, because it signals the rest isn't spin.

Report it as a monthly trend, not a dramatic quarterly snapshot. And skip composite "transformation scores": if nobody outside the team can explain the formula, nobody outside the team will believe it.

Watch for the Comfortable Vanity Metric

The most dangerous metrics are the ones that always go up.

A classic example is counting things onboarded to a platform. The number climbs every month. Executives don't care, and they're right not to.

Onboarded is not adopted. A service can be "on" a platform and still route around every capability it offers. Unused transformation isn't transformation.

Goodhart's law applies here: when a measure becomes a target, it stops being a good measure. Swapping a count for a usage-based signal often makes the chart look worse and the conversation better.

Price the Whole Lifecycle, Not Just the Build

Most business cases are written around build cost. Finance and your CTO care about total cost of ownership (TCO): what it costs to build, run, support, secure, migrate, and eventually retire.

AI has widened that gap. Gartner found that more than 90% of CIOs say managing cost limits their ability to get value from AI. It estimates that CIOs who don't understand how GenAI costs scale could make a 500%–1,000% error in their cost calculations. Gartner also predicts AI inference costs per agentic workflow will rise more than fivefold through 2028, even as model prices fall, because agents consume far more tokens than simple chatbots. Its advice: build proofs of concept that test how costs scale, not just how the technology works.

The status quo has a TCO too, and it's often the strongest part of the case. In a McKinsey survey of 50 CIOs, respondents said 10–20% of the technology budget meant for new products is diverted to resolving tech debt. They estimated tech debt at 20–40% of the value of their entire technology estate. That's the cost of standing still, and it rarely appears on the slide.

A defensible TCO view usually covers:

• Build and migration cost, including the parallel-run period.

• Run cost per unit of usage, projected as adoption grows.

• Support, on-call, security, and compliance effort.

• The cost of the old system, until it's actually switched off.

Tip: Count decommissioning savings only once the old system is off. A planned retirement isn't a saving yet.

Honest Tradeoffs Build More Trust Than Optimism

Transformation often costs more before it costs less. Running old and new systems in parallel is expensive. Some benefits lag by several quarters.

Smoothing that over rarely works. Finance teams review spending for a living and are good at spotting a polished story.

Plain framing works better: "Cost rose this quarter because we're running both systems in parallel. Here's when that ends. Here's what we'll stop if it doesn't."

That framing builds credibility for the next request, and turns finance from an auditor into a partner who helps shape the case.

Think One Level Up: What Your CTO Gets Asked

Directors who keep their programs funded tend to answer the question their CTO will face next, not just the one in front of them.

At the top, the question isn't "did this program work?" It's "is this the best use of the money compared to everything else?"

That changes what a strong value case includes:

• Comparable categories. If every team reports against the same five categories, your CTO can compare programs side by side. If each team invents its own metrics, nobody can.

• Multi-year TCO for both options. Compare the cost of the change with the cost of standing still, over the same horizon.

• Explicit kill criteria. BCG found AI leaders pursue about half as many opportunities as other companies, and expect more than twice the ROI. Saying what you'd stop makes what you keep more credible.

• Evidence early enough to change a decision. Gartner found only 18% of CIOs embrace dynamic, off-cycle reprioritization, yet those who do are 24% more likely to be top performers. Bring data while it can still redirect money, not only defend it.

• A board-ready sentence. Your CTO will compress your program into one line for the CEO or board. Write that line yourself, so the nuance survives.

The Limits of All This

Risk reduction is still the hardest category. You can define exposure concretely, such as how many critical services lack a tested recovery path. Converting that into dollars finance fully trusts is still more judgment than math.

Attribution is messy. When several teams and a market shift all touch the same outcome, claiming full credit is overreach. A credible contribution beats an inflated total.

Some value is slow. Platform work often pays off quarters later, in decisions other teams make faster. Be skeptical of any framework claiming to capture that cleanly.

The goal isn't false precision. It's making value legible enough that the organization chooses to keep investing.

The Takeaway

The work and the translation are two different jobs. At director level, the second one is yours.

Transformation doesn't become real when the work is hard. It becomes real when the value is visible enough that the organization chooses to keep funding it.

So, a question for anyone who has sat through a budget review recently:

What's the one transformation metric executives in your org actually believe, and which vanity metric did you have to let go of?

Share yours in the comments. The answers will likely be more useful than this article.

Sources

• Gartner, The CIO Agenda 2026: Master Agility, Risk and Tenacity (Nov 2025) — 48% of digital initiatives meet or exceed business targets; 57% / 52% productivity and cost pressure; 33% consistently pursue financial outcomes; 18% embrace off-cycle reprioritization (24% more likely to be top performers).

• PwC, 2026 Global CEO Survey press release (Jan 2026) — 4,454 CEOs; 56% no significant financial benefit from AI; 12% both cost and revenue benefits.

• BCG, AI Adoption in 2024: 74% of Companies Struggle to Achieve and Scale Value (Oct 2024) — 1,000 executives; 74% yet to show tangible value; 70/20/10 split of challenges; leaders pursue about half as many opportunities and expect more than twice the ROI.

• McKinsey, The science behind successful organizational transformations — nearly one-quarter of value loss at target-setting; 55% during and after implementation; 70% failure figure attributed to Kotter.

• Gartner, Gartner Says CIOs Need to Overcome Four Emerging Challenges to Deliver Value With AI (Nov 2024) — 90%+ of CIOs say cost limits AI value; 500%–1,000% GenAI cost-estimate error; test how costs scale in PoCs.

• Gartner, AI Inference Costs Per Agentic Workflow Will Increase More Than Fivefold Through 2028 (Aug 2026).

• McKinsey, Tech debt: Reclaiming tech equity (Oct 2020) — survey of 50 CIOs; 10–20% of new-product budget diverted to tech debt; tech debt = 20–40% of tech-estate value.

• Flexera, 2026 State of the Cloud Report summary (Mar 2026) — 753 respondents; unit economics use 40% → 49%; wasted cloud spend up to 29%.

I'm going to start with the strongest argument against this post, because I'd rather you hear it from me. This month one of my open-source tools shipped with 1,558 tests passing and an authentication guard that never actually ran. An import failed silently, the suite stayed green, and an unauthenticated read path went out the door. A Docker test that ran the real thing in a real container caught it. Not my unit tests, and not my coverage number.

So when I say test evidence has become the most important thing I can show about my work, I'm not saying it as someone who has it figured out. I'm saying it as someone who got burned by his own green checkmarks and came out more convinced, and more curious, than before. That surprised me, and I've spent a good part of this year exploring why. What follows is where my thinking is right now, not a conclusion. I'm writing it partly to find out where I'm wrong.

I Used to Treat Tests as the Chore

For most of my career, tests were the thing you did so the real work could ship. I wrote them, I asked my teams to write them, and I enforced coverage gates, but if I'm honest, I never thought of tests as something that said anything about me. Code was the craft. Architecture was the craft. Tests were hygiene, like flossing. You're supposed to do it, and nobody compliments you for it.

That quietly flipped this year, and it took me a while to notice. I build a lot now with AI agents doing a large share of the typing. The code shows up fast, often faster than I can read it carefully. And I noticed that when I look at my own repos, the part I get excited about has changed. The code looks fine. It almost always looks fine, and looking fine has become cheap. What feels like mine now, the part an agent can't produce for me, is the evidence that the thing does what I claim when it meets reality. That shift has made building more fun, not less.

What I'm Learning to Point To

When I think about which of my projects I'd put in front of someone I respect, I don't reach for the cleverest code. I reach for three things, and all of them are about proof.

The first is the test count and where it runs. My agent control plane shipped v0.1.0 with 925 tests, and I've learned to report both coverage numbers, 93% on my laptop and 95.45% in CI, because the gap between them is information too. The second is the field test report: what I ran the tool against, how many cases, what broke. For one planning tool, that meant 170 real goals for $0.49, which is cheap to run and was anything but cheap to design. The third, and the one I've come to value most, is the failure list: the bugs, the misses, and the things the tool still can't do.

That last one took some getting used to. There's an old instinct, probably from years in leadership rooms, that says you lead with strengths and handle weaknesses if someone asks. Publishing a failure list goes against that. A few releases in, I realized the failure list was the part people read most closely, because it proves the rest isn't marketing. It's also where the best conversations start.

What Worked

Making my evidence checkable by strangers

I started committing field test reports, test reports, and security audits directly into the repos: public, linked, and dated. The result I didn't plan for turned into one of the best things that happened to me this year. A reader audited one of my releases in public. He wrote down what he expected to find before looking, checked it against the published artifacts, and recorded every place where my release story didn't match my own evidence.

He found real contradictions. The engine was mostly right, but the story I'd told about it was partly wrong. My first reaction was a flinch, which I think is normal. My second reaction, a day later, was gratitude, because it was the most rigorous review I've ever received, and it only happened because the evidence was sitting there for him to check. You can't audit a vibe. You can audit a report, and being auditable turns out to be a very different feeling from being right.

Treating "it passed" as a claim

AI work taught me that a confident answer and a correct answer look identical until someone checks. Somewhere along the way that lesson moved from how I treat model output to how I treat my own. A green run is now a claim I have to back up. What ran? Against what data? Where's the report? When I built the review engine where two models debate a pull request, I fixed 13 bugs before I trusted a single number it produced, including a join that silently collapsed my dataset from 2,333 rows to 359. None of those were clever architecture bugs. All of them were "you haven't looked closely enough" bugs, and they were only visible because I'd forced myself to ask for evidence instead of accepting green.

What Didn't Work

A big number still fooled me

Back to the 1,558 tests. What I found most interesting about that bug wasn't the bug. Imports fail. It's that I relaxed. I saw a large green number and felt safe, and that feeling is exactly what I've spent months telling other people not to trust. I wrote the whole story up here, but the lesson I took personally was simpler. A big test count can be theater just as easily as a big README. Now I treat a big green number as the start of a conversation, not the end of one.

Any metric I show off, I'll eventually inflate

The other thing I learned is that the moment coverage becomes something to be proud of, it becomes something that's tempting to grow for its own sake. In one of my repos, a test whose entire body was pass made it into the suite. It could never fail and would never catch anything. In the same project, a harness found zero assertions to run in 57 of 65 files and reported 0 / 0 as success. Code review caught both. My metrics didn't, which I found oddly reassuring: people reading carefully still matter here.

With AI in the loop, inflating a metric like this costs almost nothing. You can generate a thousand tests in an afternoon. If coverage alone becomes the thing we judge engineers by, we'll get exactly what we measured. The good news is that the thing underneath coverage is much harder to fake, and that's where I've landed.

Where My Thinking Is Right Now

After a few months of exploring this, my current guess is that coverage isn't the career metric. The evidence trail is. The test count is the headline. What makes it believable is everything around it: the field test that ran the real thing, the report with the bad numbers left in, the list of what still breaks, and the fact that a stranger can check all of it without asking me.

This matters to me more than a methodology debate, because it's changed what I want to be known for. Early in my career I wanted to be the engineer with the clever design. Later I wanted to be the leader with the right strategy. Now, with agents writing more of the code every month, I find myself wanting something humbler and, I think, more durable: to be someone whose claims hold up when you check them.

What I'm Still Trying to Learn

Evidence is expensive. My field tests often take longer than the features they test, and I'm not convinced that scales beyond open-source work where I control the schedule. Hiring hasn't caught up either. Most interview loops still care more about solving a puzzle live than about a public failure report, and I understand why, even though I think it's measuring the past. And I've only done this as one person. On a real team, who owns the evidence trail? Is it the author, the reviewer, or some role we haven't named yet? I don't know yet, and these feel like some of the most interesting open questions in our field right now.

Your Turn

I'd love to learn how other people are thinking about this, especially anyone whose job has shifted toward reviewing more code than they write. If you think I've got this backwards, I want to hear that too.

When you look at someone's repo, what actually makes you trust it? Stars, tests, commit history, the README, or something else entirely?

Have you ever published a bad number on purpose? How did it feel, and how did people react?

If AI writes most of your code next year, what will you point to as proof that you're good at your job?

That last one is the question I'm most excited to hear answers to. I have a feeling the comments will teach me more than this post did.

For 25 years I've been the person in the room asking to see the data. In incident reviews, in planning meetings, in one-on-ones with engineers who were sure their service was fine because nobody had complained yet, I've said some version of the same sentence: if you can't see it, you can't run it. I believed it. I still do. But this summer I noticed something that made me more curious than I expected: I had spent a whole career measuring other people's work and had never once measured my own.

My own career ran on good intentions: learn more about agents, write more, finally ship something. All true, all vague, and all impossible to learn from, because you can't miss a target you never set. And if you can't miss, you can't find out anything new.

So in late June I tried to do to myself what I'd done to every system I've ever owned. Metrics, a schedule, a few hard rules, and a promise to look at the numbers even when they were bad. It turned into the most interesting experiment I've run in years. I'm not writing this as advice. These are notes from someone still in the middle of figuring it out, and I'm hoping people who've tried something similar will tell me what I'm missing.

What I Built, and Why It Was So Small

I kept it deliberately boring: three markdown files in an Obsidian vault. A metrics page tracking where my time goes, what's covered, and how fast things ship. A weekly schedule with slots for building, writing, open source, and talking to people, planned out to October 2027, which is absurd and which I rephase constantly. And one rule: a project isn't real until it has code, docs, and a published article. All three, or it doesn't count.

I made it small on purpose, because I know my pattern. Designing systems is my comfort zone. Using them, and letting them tell me things, is where the learning actually happens. Keeping the setup plain meant I spent my energy on the second part.

The Numbers, and What They Hid

Twelve weeks produced forty-plus articles on dev.to and Hashnode, more than a dozen open-source tools, flagship test suites between 800 and 1,500 tests each, and a metrics file now on version 93. When I first wrote that list down I felt proud. Then I got curious, because these are exactly the numbers I'd ask questions about if someone on my team presented them. The interesting stuff lives in the gaps between them.

What Didn't Work, and What It Taught Me

I broke my own rule on day one, and I know why

My publishing rule says one article a week, so each piece can stay the best thing on its topic for months. On July 10, the very first day, I published three. In September I published eight in four days from a single series.

I had reasons both times. The drafts were done, and waiting felt wasteful. The more honest reason is that I wanted to see the numbers move. The cadence rule was there to protect the reader, and I'd forgotten that. If someone on my team had proposed eight posts in four days, I'd have asked who it was for. Now I ask myself that before I hit publish, and that one question has made my writing noticeably better.

I built a dashboard that told me what I wanted to hear

My first metrics page counted output: articles written, notes created, projects started. The lines went up every week, and I felt productive looking at them. It took me weeks to see I'd built a vanity dashboard. It couldn't tell a throwaway note from a shipped tool.

I've caught that mistake in other people's quarterly reviews in five seconds. Missing it in my own taught me something useful: blind spots are called that for a reason, and even a file you wrote yourself can give you the outside view you need. The page now opens with a note to myself: "This page no longer primarily measures content volume." It tracks systems that work, writing people actually read, and evidence I could hand a stranger. It's a much more useful page now, and honestly a more interesting one to open every week.

The dashboard turned into a diary

By version 93, the header of that file had grown so long my editor truncates it at 2,000 characters. Every release and decision got appended, because deleting anything felt like erasing my own history. It stopped being a tool for seeing clearly and became a record that I'd been busy. I suspect many engineers do the same thing. We keep old alerts, dashboards, and branches around as receipts for effort. Personal observability drifts the way production observability does, and it turns out the fix is the same too: prune, archive anything past 400 lines, and keep only what helps you decide. It's a small habit, and I'm glad I picked it up.

What Worked, and Why It Surprised Me

Finishing mattered more than starting

The "code, docs, and article" rule broke my oldest habit: the project that's 80% done. I've carried dozens of those over the years. They feel like progress without ever reaching the point where you find out if the idea works. Under this rule they earn nothing, so I started finishing smaller things instead of starting bigger ones, and finishing turned out to be where all the surprises were.

Writing the article also turned out to be my most honest test. When I wrote up an AI review engine where two models are supposed to argue about a pull request, I went back to the raw logs for a quote and found the second model wasn't arguing at all. It was replaying pre-generated text. 89% of what I'd built was theater. The tests were green and the demo looked great. Only explaining it in plain words made me read closely enough to see it. I've come to think writing isn't how I share what I've learned. It's how I find out whether I've learned anything.

Letting go of 22 ideas felt lighter than I expected

In one week this quarter I retired 22 project notes. They weren't shower thoughts. They were designed, scoped, and some of them had been with me for months. A bigger effort covered what they were meant to do and did it better, so each got a banner saying what absorbed it, a link to its new home, and I moved on.

I expected that to hurt. For most of my career I've quietly tied my identity to being the person with ideas. Retiring them felt like it should be a loss. Instead it felt like deleting dead code, one of the most satisfying things an engineer gets to do. That told me something. Maybe what I'd been attached to wasn't the ideas. It was the feeling of having options. Committing to fewer things has been freeing in a way I didn't expect, and the ideas I kept are getting far more of my attention.

A wrong plan was still an honest one

My schedule is wrong constantly. When I made agent security my main focus from October through February, that single decision pushed about 18 weeks of plans back, and I could see exactly which ones moved. Without it, I'd have said yes to everything and spent months wondering why nothing finished. The same happened with writing: in September I studied the top 30 recent posts on dev.to and rewrote my publishing rules around them. Several of my strong opinions turned out to be wrong. That was genuinely exciting, because it was the first time my writing got an opinion from something other than my own taste.

What I'm Still Learning

I went into this expecting better numbers. What I got was an argument. The dashboard kept disagreeing with my gut, and about half the time the dashboard was wrong and half the time I was. When everything lived in my head, I never lost an argument with myself, so I never learned anything from one. Now there's something on the other side of the table, and I'm learning faster than I have in years.

The question I ask most weeks has changed too. It used to be "what should I learn next?" Now it's "what's starved?", which usually points somewhere I hadn't thought to look.

What I'm Still Trying to Figure Out

I can game my own metrics, and on tired weeks I have. I don't know how people who've done this longer keep themselves honest. Nothing on my dashboard goes red when I'm running on four hours of sleep, and I haven't found a good way to measure rest. And this is one person over twelve weeks. It's an experiment I'm going to keep running, and I'd much rather learn from people further along than reinvent everything myself.

Your Turn

I'd really like to learn how other people handle this, especially people who spend their working lives measuring systems.

Do you measure your own career, or only your team's work? If you do, what's worked for you that I should steal?

How do you keep a personal metric from being gamed by the person who wrote it?

What's the metric you'd be most curious (or nervous) to put on your own dashboard? For me it was "articles people actually read" sitting next to "articles I wrote."

Have you ever let go of an idea you were attached to, and felt lighter instead of worse?

And the one I keep coming back to: if an honest dashboard showed you your career tomorrow, what do you hope it would say, and what do you think it actually would?

You didn’t lack raw intelligence—you lacked context. You didn’t know where the internal credentials lived, how the microservices communicated, or why the CI/CD pipeline had that bizarre manual step on deployment days.

When developers treat AI agents like disposable code-generators, they constantly get frustrated by generic, half-baked answers. But if you shift your mental model and start treating your AI agent like an ambitious, hyper-eager intern, everything changes.

The Onboarding Blueprint

You wouldn’t drop a junior developer into a massive, undocumented repository and say, "Build feature X, good luck." You’d give them an onboarding guide. AI agents need the exact same courtesy.

To turn an agent into a true teammate, invest in three core practices:

1. Provide Institutional Context: Give your agent clear blueprints—style guides, architectural decision records, and repository norms (using files like ⁠.cursorrules⁠ or context registries). The clearer the codebase rules, the better the output.

2. Persist Its Knowledge: Don't start from zero in every session. Save successful workflows, prompt templates, and edge-case guardrails into version control. When an agent misinterprets a pattern, update the documentation so it never makes that mistake again.

3. Review, Don't Just Reject: Great managers don't just overwrite an intern's bad pull request without explanation. Treat errors as learning moments. Provide direct feedback on why a solution fell short, and ask the agent to revise its approach.

Why the Investment Pays Off When you spend ten minutes teaching your agent your domain’s specific nuances, you aren't losing velocity—you're compounding future output. Over time, an onboarded agent stops generating raw boilerplate and starts anticipating team-specific edge cases, drafting accurate PR summaries, and keeping unit tests aligned with your architecture. Watching your agent transition from a noisy assistant into a dependable force multiplier is one of the most rewarding upgrades to your daily workflow.

Questions for Team Dialogue

To start shifting your team’s mindset, bring these questions to your next engineering sync or retrospective:

• Where is our team losing time by re-explaining project architecture to our AI tools?

• What team-specific conventions or documentation could we check into Git today to immediately make our agents context-aware?

• When an AI output fails, do we view it as a model limitation or as a missing piece of documentation in our system?

• How can we structure persistent feedback loops so our agents grow more capable as our codebase evolves?

Engineering leadership isn't just about managing people—it's about building an environment where every resource, human or synthetic, has the context required to succeed. Give your agent the tools to learn your system, guide its growth with clear feedback, and watch it become an indispensable part of your engineering stack.

We kept delivering projects successfully and still living with the same problems.

Every initiative had a launch date, a steering deck, and a handoff that nobody loved. Teams moved on. The systems they changed kept aging, drifting, creating new friction. At first I thought we had a delivery problem. It took a painful handoff to see we had an ownership problem. That shift changed more than many reorganizations — and in the AI age, it matters more.

It is not just my experience. Marty Cagan has made this distinction for years — product teams are given problems to solve and held accountable for outcomes, not handed projects to finish. Harvard Business Review put it bluntly on the cover this March: temporary teams can build a system, but you need permanent teams to live with it.

The Handoff That Would Not End

The project closed on time. On budget. Green dashboard. Two weeks later the same friction was back — same integration debt, same support questions, same adoption gap, just under a new title. The problem stayed alive while the team was gone.

I have used project completion as a false proxy for impact myself. It feels good to ship. It is worse to watch the same problem return because no one was left to carry it.

I told that team: we did not have a delivery problem. We had an ownership problem. We rewarded completion while the business needed continuity. When we started asking "who improves this next quarter?" instead of "when does it ship?", the tension changed.

Today that question is louder. AI has collapsed the cost of the first version — prototype to plausible MVP in days. What has not gotten cheaper is everything after. If no team owns what happens post-demo, the prototype just becomes a faster way to create unowned debt.

Why Projects Crack — Especially Now

Projects create motion, not continuity. Most transformation problems are not solved at launch. They are solved when a team keeps improving the capability after launch.

In project mode, continuity breaks at the handoff. Ownership blurs. You see it in repeated incidents and the quiet integration tax no project owns.

We mapped a few value streams and named what needed to stay owned. Immediate pattern: internal platforms, core systems, customer capabilities suffered most as temporary projects. They needed roadmaps and health metrics, not milestones.

That matches Team Topologies' default for value flow — a stream-aligned team that is long-lived and owns the outcome end to end. Stable teams build shared context you cannot recreate by reshuffling. One 2023 review found becoming a high-performing team takes longer than picking up new skills — why stable teams beat dynamic task forces in stable domains (Wiley, 2023).

AI makes that brittleness more expensive. Team Topologies puts it this way for the AI age (Aug 2026): every extra AI-generated line is an extra line to test, maintain and secure. Flooding production with unverified AI output grows the attack surface that a temporary team has already left behind.

What We Changed Instead of Reorganizing

We started with ownership, not a reorg.

Named owners with real authority. Not a title on a slide. Clear decision rights and a sustained business partnership. Without that, you have re-labeled a project team. Cagan is blunt: empowered teams must be trusted to figure out how once leadership is clear on which problem (SVPG, July 2024).

Kept teams stable longer. We stopped reshuffling every quarter. Context stuck. Morale improved when teams owned something coherent. Team Topologies says it plainly: "I'll take a team that's worked together for years over rock stars any day" (Key Concepts). DORA backs it: high performers stay loosely coupled to test and deploy independently without hand-offs (Loosely Coupled Teams).

Planned around outcomes. Quarterly planning anchored on health, adoption, reliability — not just on-time, on-budget. Tech debt became part of product health, not a side battle. McKinsey frames it the same: tie funding to measurable goals and treat tech debt as a managed tax (McKinsey Dec 2023 via Planview Sep 2024).

Made health visible, simply. Adoption, reliability, cost-to-serve, backlog balance. Fewer dashboards. When health is visible, funding gets honest.

Let teams improve what they operate. No hard wall between "run" and "change." HBR found the winners did exactly that — permanent cross-functional teams that keep investing based on user feedback long after launch (HBR Mar–Apr 2026).

Made AI part of stewardship. Prototypes and agent workflows now pass the same product checks as any increment — evals, cost bounds, rollback — defined by the owning team, not the demo author. That is the stewardship model Team Topologies calls essential for AI: every workflow governed by a long-lived team (Stewardship for AI Age, Aug 2026).

What Compounded

Projects end. Products accumulate — outcomes, debt, risk, learning. Stable teams compound because context persists.

A team that shipped less in a quarter but cut repeat incidents and lifted adoption created more durable value than one that hit a launch date and handed off fragility. Launch is not the finish line. It is the start of stewardship.

Projects are good at finishing work. Products are better at carrying responsibility.

McKinsey's 2023–24 study of 400+ companies makes the business case: top-quartile product model maturity correlated with 60% higher returns to shareholders and 16% higher operating margins, plus 38% higher customer engagement (McKinsey).

In the AI age the gap widens. When a POC takes a week, the value is not the demo. It is who carries the model, prompt, evals, cost and incident after the demo. Projects get more demos. Products get more compounding.

How AI Tilts It Both Ways

Positive — AI makes the product model affordable. A small stable team can generate a prototype in days and test earlier. That is exactly what product thinking wants — outcomes over outputs, continuous investment — and why HBR's permanent teams pulled ahead in 2026 (HBR). AI shortens the discovery loop; it does not replace it.

Negative — AI makes the project trap cheaper to fall into. When the first version costs near zero, it is tempting to fund another pilot as a project and never fund the product work that makes it trustworthy — evals, data boundaries, cost controls. Unowned AI prototypes become shelfware or, worse, production code that expands the attack surface (Team Topologies AI Age). DORA's hand-off data predicts what breaks next: waiting on shared environments.

Same power, opposite compounding. Ownership matters more now, not less.

What Still Trips Us

Funding. Planning models still reset quarterly around scope. McKinsey flags funding and tech-debt as the top maturity gaps and suggests releasing funding on performance objectives instead of annual pots. I wish we had involved Finance earlier.

Incentives and AI. Saying "products" while rewarding milestone theater ignores health metrics. Without a product home, AI pilots become demos without owners. I would have made tech-debt and AI eval budgets part of the first planning cycle, not an afterthought.

Boundaries and operational work. Without clear boundaries, product ownership is just more meetings. Platform health rarely looks like transformation, so it gets devalued. The Wiley review is useful here — stable teams win in stable, predictable domains small enough to own, which is exactly internal platforms (Wiley 2023). I would have revisited org design sooner and published fewer, clearer health metrics from the start.

Our clearest before/after: stable ownership cut repeat incidents within two quarters; reshuffled teams kept re-paying the same integration cost. Seeing that delta once ended the vocabulary debate.

If You Are Considering the Shift

You do not need a new reorg. Pick one value stream where pain — and AI prototypes — already pile up. Give it a home for two quarters. Name an owner with real authority, keep the team stable, plan around one outcome users feel, require every AI POC to pass the same product checks: evals pass, cost bounded, rollback tested.

If it compounds, you will feel it in fewer surprises, not more slides. Start with one pilot team, as Cagan recommends, while the rest stays on project governance (SVPG pilot teams).

What About You? Let's Compare Notes

Are we wielding AI like a sword — to ship more, faster — or like stewardship — to own more, longer? The same power that lets a stable product team compound value lets a temporary project team compound debt. The tool did not change. The ownership did.

I am genuinely curious how this lands where you are — I read every comment and I learn more from these threads than any framework deck:

1. What should never be run as a temporary project in your org? For us it was the internal platform — every project handoff just re-created the support queue. What is yours? Integration layer? Data platform? Customer workflow? Drop the one you would protect first.

2. Where do handoffs already create repeat pain — and where have AI prototypes quietly added to it? If you have a shelf of AI demos that never got an owner, what did that cost you in rework or trust?

3. What blocks persistent ownership most right now — funding, org design, or mindset? If you could convert one value stream tomorrow — especially one with an AI POC already floating around — where would you start and what outcome metric would you hold it to?

Share the real friction, not the theory. If you have a handoff story that still stings, or a stable-team win that compounded, I want to hear it. The comments on pieces like this end up better than the article itself — let's keep that tradition going. What do you think?

If you found this useful, tell me which section you would challenge — I will reply to the first 20 comments with what we tried and where we were wrong.

Further Reading — All 2023–2026, AI-Relevant

• Product Model in Traditional IT — SVPG, July 2024

• Why Digital Product Model Beats Project — HBR Mar–Apr 2026

• Team Topologies — Organizing for Fast Flow + Fowler overview, July 2023

• Stewardship of Value Flow — Team Topologies for AI Age, Aug 2026

• Dynamic vs Stable Team — Wiley 2023

• Loosely Coupled Teams — DORA 2024

• Bottom-Line Benefit of Product Operating Model — McKinsey Dec 2023 via Planview Sep 2024

Part of AI Leadership in the Real World — how leaders turn scattered pilots into governed, adopted, measurable capability.

TLDR: A support agent doing 50,000 chats a month needs ~3.5 FTE and $500k+/year just to stay accurate — while a typical 100-seat Copilot rollout sees only 20-30 seats used weekly. For SMBs, build-vs-buy isn't about features. It's about what you can afford to own for 24 months.

We thought we were choosing a tool. We were really choosing a future dependency, a support queue, a governance burden, and a second bill that arrives a year later.

Every vendor demo promised acceleration, control, and simplicity at once. Every internal proposal promised flexibility, ownership, and leverage. Nobody said both bills arrive late — one in engineering on-call, the other in consumption meters.

Good platform decisions feel a little boring at first and very smart a year later.

Why AI is special (and why old build-vs-buy math breaks)

Traditional software mostly stays still when you leave it alone. AI doesn't:

1. It drifts. Knowledge changes, customer language shifts, users ask harder questions once they trust it. Accuracy quietly drops from 90% to 70% with no error log.

2. It speaks for you — legally. A wrong Confluence page is embarrassing. A wrong chatbot answer is a commitment a tribunal can enforce.

3. It lives on someone else's deprecation clock. OpenAI gives at least 6 months before retiring a GA model. That's a hard deadline, not a backlog item. Prompts, evals, and output parsers all need rework.

4. It multiplies cost per request. One human click = one action. One agent resolution = 6 lookups, drafts, updates, and logs — each potentially metered.

5. It turns connectors into permanent work. Salesforce, SharePoint, Jira, Zendesk all change auth, rate limits, and APIs. Your agent keeps running while its knowledge goes stale.

Gartner predicts 40%+ of agentic AI projects will be canceled by end of 2027 on cost, unclear value, and weak risk controls. McKinsey's State of AI 2025 (Nov 5, 2025) found 88% report regular AI use in at least one function, yet only 39% report any enterprise EBIT impact (most under 5%; ~6% high performers). BCG's Widening AI Value Gap (2025, n=1,250) found only 5% generating value at scale while 60% see minimal gains. The gap isn't building. It's owning.

Here are 6 public cases every SMB CTO should know — scenario, what happened, what didn't work, and why.

1. The chatbot that created legal liability: Air Canada (2024)

Scenario: Jake Moffatt, booking emergency travel after his grandmother died, asks Air Canada's website chatbot about bereavement fares. The bot says: book full fare now, claim the discount within 90 days.

What happened: That was wrong. The real policy barred retroactive claims. Moffatt flew, applied with screenshots and a death certificate, was denied. He took it to the British Columbia Civil Resolution Tribunal — Moffatt v. Air Canada, 2024 BCCRT 149 (Feb 14, 2024).

The tribunal ordered Air Canada to pay C$812.02 (C$650.88 fare difference + interest + fees) for negligent misrepresentation. It explicitly rejected Air Canada's "remarkable" defense that the chatbot was a separate entity responsible for its own actions: "it is still just a part of Air Canada's website."

What didn't work and why: No grounding to the actual policy page, no policy-layer guardrail, no correction path. The bot hallucinated a generous policy linked to the real bereavement page that contradicted it.

SMB lesson: If you ship a customer-facing agent — bought or built — you own what it says. For a 20-person company, one such incident is a support crisis and a trust crisis. Budget the eval + human escalation before launch, not after.

2. The "AI platform" that was mostly services: Builder.ai (2025)

Scenario: Builder.ai, founded 2016 as Engineer.ai, marketed "Natasha" as AI that builds software "as easy as ordering pizza." Raised \(450M+, valued at ~\)1.5B in 2023 with Microsoft and Qatar Investment Authority backing.

What happened: On May 20, 2025 it entered insolvency. Audits revised claimed 2024 revenue of $220M down to ~$55M. Creditor Viola Credit seized ~\(37-40M after a \)50M debt facility. The US Attorney's Office for SDNY subpoenaed records. UK entity Engineer.ai Global Ltd went into compulsory liquidation.

Viral coverage said "700 engineers faked the AI." That shorthand is contested — Gergely Orosz (Pragmatic Engineer), after talking to ex-employees, corrected it: there was a real AI team doing spec, estimation, and code workflows, alongside hundreds of outsourced developers doing delivery. WSJ had flagged heavy human reliance as early as 2019.

What didn't work and why: The hybrid model can be legitimate. The failure was marketing opacity + financial misrepresentation: buyers couldn't tell which part was automated, which was human, and what would happen if the vendor collapsed. Customers who outsourced their roadmap to it lost the platform overnight.

SMB lesson: When buying an "AI platform," diligence the labor model: what % is model vs. reusable components vs. humans? What happens to your code, data, and uptime if they go under? Get escrow and export in writing.

3. The seat license nobody sat in: Microsoft 365 Copilot

Scenario: A 100-person SMB buys 100 Copilot for Microsoft 365 seats at \(30/user/month (\)36,000/year) — plus the required underlying M365 license upgrade many teams forget to price.

List price: Microsoft lists Microsoft 365 Copilot at \(30/user/month (annual commitment) as an add-on requiring a qualifying M365 base plan — true all-in \)42-\(90/user/month depending on base tier. Copilot Studio is separate consumption: ~\)200/mo per 25,000-credit pack.

What breaks: Independent utilization surveys consistently report only 20-30% of purchased seats see weekly active use at scale (directional, not audited). Change management (training, comms, support) is repeatedly estimated at 30-50% of license cost and rarely in the proposal.

Separate cautionary tale (consumer, not enterprise): Australia's ACCC filed Federal Court proceedings Oct 27, 2025 alleging Microsoft misled ~2.7M Personal/Family subscribers by bundling Copilot rises ($109→$159 Personal, $139→$179 Family) while hiding the cheaper Classic no-AI option in the cancellation flow. The widely cited 100%+ ROI figures trace to a Microsoft-commissioned Forrester TEI study — real data, not independent data.

SMB scale-down: At 25 seats instead of 100, the waste is smaller in dollars but identical in ratio — 5-8 active seats carry the other 17. That's why seats must be earned, not allocated.

What didn't work and why: Per-seat pricing assumes uniform adoption. In SMBs adoption is spiky: 15 power users love it, 60 try twice, 25 never enable it. You pay for 100, get value from 25. The prerequisite upgrade + idle seats kills ROI by month four.

SMB lesson that worked: Pilot with a 25-seat power cohort, instrument weekly active use from day one, and only expand when active-use >60% for 4 weeks. Disciplined teams treat seats as earned, not allocated.

4. The second meter: Salesforce Agentforce ($2/conversation)

Scenario: A mid-size team handling 50,000 service interactions a month turns on Agentforce. List price: $2 per conversation — on top of existing Service Cloud seats.

List price: $2 per conversation at list, before discounts — and before the Service Cloud seats underneath. Salesforce's current official model is Flex Credits ($500/100k; 1 standard action = 20 credits = $0.10) as the flexible alternative, with unused credits not rolling over and Flex vs. Conversations not mixable in one org.

What breaks: Conversation definitions (failed/escalated/abandoned handling) and per-workflow action counts are contract-specific — which is exactly why pre-sign modeling matters. Internal IT/HR agents with lots of short queries get the worst unit economics on per-conversation pricing — same $2 whether the answer saved $200 or $2.

SMB scale-down: At 5,000 interactions/mo instead of 50,000, the meter drops 10x (~$10k/mo at list) — but the modeling work doesn't. You still must define the billable unit in writing before signing.

What didn't work and why — usage amplification: A human resolving a ticket = 1 action. An agent resolving it = query record + search KB + draft response + update case + send email + log interaction = 6 metered steps, depending on config. Teams estimated on user requests but were billed on agent actions.

Copilot Studio has the same pattern: $200/mo for 25,000 credits, but generative actions burn credits faster than classic flows, plus separate Azure token + compute charges. Three bills, three consoles.

SMB lesson that worked: Demand a cost-modeling pilot in writing: define "conversation/credit" for your workflow, exclude tests, cap overages, negotiate rollover + 24-month price lock. Design agents to cache lookups and confirm intent before chaining actions — teams report 20-40% consumption cuts from flow design alone.

5. The build that became a hidden product: the 3.5-FTE support agent

Scenario (composite SMB, modeled on SearchUnify's 50k/mo reference — not a single named company): A 30-person SaaS company builds a RAG support agent on LangGraph + Confluence + Zendesk. Pilot hits 88% answer accuracy. CEO calls it "done."

What happened (per SearchUnify's 2026 vendor field analysis — directional, not audited): To keep a 50k-interaction/mo agent production-grade you need roughly: 1.0 AI/ML engineer + 1.0 data engineer + 0.5 platform + 0.5 security/governance + 0.5 product analyst = 3.5 FTE, ~$500k-700k/year before infra, tokens, observability, and audits. Major model migrations take 6-10 weeks each (re-benchmark, re-prompt, re-test, re-secure). Connectors drift: auth changes, endpoints retire, docs grow by thousands of pages.

Three drifts compound: knowledge drift (docs change), data drift (tickets change), behavioral drift (users ask harder questions). Without evals, you find out from angry customers.

SMB scale-down: At 5k interactions/mo the token/infra meter drops ~10x — but the governance floor doesn't. You still need someone to own drift review, connector fixes, and the model-deprecation deadline. That's the firewall that pages you at 2 AM when the model retires or the connector breaks.

What didn't work and why: The team budgeted 8 weeks to build, zero heads to operate. Two backend devs now spend Fridays fixing retrieval, tuning prompts, and patching connectors — the "temporary" bot became an unstaffed platform product. Software maintenance is 80-90% of lifecycle cost historically; AI maintenance runs 2-3x the initial build in the first few years because drift + vendor deprecation never pause.

SMB lesson: Don't ask "can we build it?" Ask "can we staff 0.5-1.0 FTE for 24 months to own it — including who gets paged?" If not, buy the boring platform and own only prompts + evals.

6. What did work: narrow scope + outcome pricing (Intercom Fin + BCG 5%)

Scenario: Same SMB instead picks one high-volume, well-logged workflow — e.g., "refund-status + plan-change" tickets — and buys Intercom Fin at ~$0.99 per resolved conversation.

What happened: Intercom's official Fin meter is $0.99 per resolution (standalone available with a 50-resolution minimum, no seats required) — you pay only when Fin resolves, not per query or token. Why this pattern works for SMBs: price maps to value (resolution, not login), the team can calculate cost-per-resolution vs. human cost, and scope stays narrow enough to govern. Industry studies widely report the same shape even where exact numbers vary by survey: BCG reported only ~5% generate substantial AI value at scale with "deep and narrow" pilots outperforming sprawl, McKinsey reported high AI usage but far lower EBIT impact, and mature portfolios skew heavily buy (commodity) with selective build (differentiating). Treat those survey percentages as directional, not audited — the mechanics (narrow scope + outcome meter) are the reliable part.

Why it worked: Narrow blast radius, verifiable outcome, and a vendor whose meter matches your P&L. The team kept ownership of KB quality + weekly refusal/deflection review — 2 hours/week, not 2 FTEs.

What TCO actually means for AI (the 4 lines everyone underestimates)

The "second bill that arrives a year later" — that's TCO. Most teams price the build and forget the other three lines:

1. Build / purchase — the demo number everyone quotes (Copilot \(30/seat, Agentforce \)2/conv, or your 8-week sprint).

2. Operate & maintain — 2-3x the build in the first few years (drift detection, connector fixes, model migrations at 6-10 weeks each, prompt rework).

3. Governance & change management — 30-50% of license cost (training, comms, support) or 0.25-0.5 FTE for evals, security review, incident response.

4. Meter / exit reserve — the second meter (overages, usage amplification, repricing on renewal) + the lock-in reserve (what it costs to leave: data export, re-integration, M&A repricing).

Score the rubric's "24-mo TCO" row on the sum of these four lines, not the first one. Here's a worked example for a 25-seat SMB handling 5,000 tickets/month:

Read honestly: in this scenario the build runs ~26% more over 24 months and hands you the on-call burden. Swap your seat counts, ticket volumes, and rates — the shape stays. Build wins only when the workflow differentiates you and you can staff the operate/govern lines.

An SMB CTO rubric (steal this for your next review)

Score each use case 1 (buy) to 5 (build). Default to buy unless 2+ factors score 4-5.

Non-negotiables before any demo: SSO, audit logs, data residency/boundaries, admin kill-switch + human escalation, observability (resolution, hallucination, cost/interaction). If a vendor can't meet these, they're off the shortlist — no matter how good the demo.

Run pilots in real workflows, not sandboxes. Document re-evaluation triggers (e.g., "if cost/resolution >$X or active-use <Y%, we revisit in 90 days"). Revisit on schedule, not when pain forces it.

Key takeaway

Build-vs-buy for AI is really own-vs-coordinate. Own only the layer that differentiates you and you can operate well for 12-24 months. Coordinate (rent) everything else — but model the second meter, staff the 2-hour weekly governance loop, and keep an exit path.

Honest limitations

This piece has no access to your contracts or usage logs — substitute your seat counts, volumes, and rates into the math above. It doesn't cover fine-tuning economics or on-prem GPU breakeven (competitive only at sustained 5-6+ hrs/day utilization over 3-5 years). Regulated workloads (health, finance, public sector) shift strongly toward buy-the-certified-platform or build-in-VPC — get counsel early. And vendor pricing from 2025-2026 will move; re-benchmark every 6-12 months.

Open questions for your team

• What AI layer do you actually want to own at 2 AM, and why?

• What are your non-negotiables before you compare vendors?

• How do you stop tool sprawl before 15 pilots become permanent?

• What TCO line (support, governance, change management, overages) are you most likely underestimating?

• When does a tactical tool become a strategic dependency — and who decides?

• If you had to standardize on one AI lane tomorrow, what would drive the choice: cost, control, or speed?

The smartest build-versus-buy decision is rarely about the best feature set. It's about choosing what your organization is prepared to own well — and refusing to rent what you can't afford to lose.

Sources

Primary (official / court / regulator / analyst press release):

• Salesforce, Agentforce Pricing (official) — Conversations \(2/conversation; Flex Credits \)500/100k; 1 action = 20 credits ($0.10); unused Flex Credits do not roll over; Flex + Conversations cannot mix in one org

• Salesforce Help, Agentforce Pricing (May 19, 2025) + Flexible pricing press release (May 15, 2025)

• Microsoft, Microsoft 365 Copilot enterprise pricing (official) — $30/user/mo add-on, qualifying M365 plan required

• Microsoft, Copilot Studio pricing (official) — 25,000-credit packs $200/mo; credits billed per agent action/response

• Microsoft Learn, License options for Microsoft 365 Copilot

• ACCC, Microsoft in court for allegedly misleading millions (Oct 27, 2025) + Concise Statement PDF — consumer Personal/Family bundling case (~2.7M subscribers; $109→$159 Personal, $139→$179 Family; Classic option hidden in cancellation flow; alleged, not yet decided)

• Gartner, Predicts Over 40% of Agentic AI Projects Will Be Canceled by End of 2027 (June 25, 2025)

• Moffatt v. Air Canada, 2024 BCCRT 149 — BCCRT decision / CanLII; coverage: BBC, ABA Business Law Today — C$812.02 award, "separate entity" defense rejected

• Builder.ai insolvency — TechCrunch (May 20, 2025) (\(1B+ valuation, \)450M+ raised, insolvency proceedings); nuance correction: Pragmatic Engineer — Builder.ai did not "fake AI with 700 engineers" (hybrid AI + human delivery; WSJ flagged human reliance as early as 2019)

• Intercom, Fin — $0.99 per resolution (official) + Pricing + AI agent pricing comparison (Fin vs Zendesk vs Agentforce) — outcome meter (pay only on resolution), standalone Fin with 50-resolution minimum; resolution-rate and adoption-lift figures are vendor-reported

Secondary (vendor field analysis — directional, not audited benchmarks):

• SearchUnify, The Real Cost of AI Agents: Maintenance, Model Drift, and Hidden Ownership Costs — 3.5 FTE / $500-700k/yr reference model at 50k interactions/mo, 6-10 week model migrations

• MindStudio, Agentic SaaS Pricing: AgentForce and Copilot Credits — usage-amplification analysis, $100k/mo illustration at 50k conversations, triple-billing (seat + messages + tokens) explainer

TLDR: 60 ideas on a board. 4 product teams to build them. $1.2M in annualized run cost for pilots that produced $340K in measurable value. The week we made tradeoffs visible was the week AI stopped being a budget line item and started being a product strategy.

My AI backlog looked like a menu with 60 "top priorities" and no kitchen to cook them.

Every function had a smart idea. Every idea came with urgency and a sponsor. Customer support wanted a deflection chatbot. Engineering wanted a code review assistant. Sales wanted lead scoring. Operations wanted anomaly detection. HR wanted resume screening. Finance wanted invoice reconciliation.

Each one was a good idea. That was the problem.

When every idea is good, prioritization becomes political. The loudest pitch keeps winning. People start optimizing for being seen, not for being useful. And the organization quietly trains everyone to be louder.

I run a product P&L. I do not have the luxury of treating every good idea as a funded initiative. My job is not to maximize the number of AI pilots. My job is to maximize the return on the engineering capacity, infrastructure budget, and organizational trust I have been entrusted with. Those are finite. Every pilot I approve is a pilot I cannot fund somewhere else. Every dollar of inference cost is a dollar that did not go to a product feature, a reliability improvement, or a person.

We were treating AI like a lottery ticket instead of a managed investment. The P&L was telling us that before I was willing to listen.

What Changed: The Cost of the First Prototype Collapsed

Two years ago, prototyping an AI use case took weeks — data pipeline, model, inference endpoint, UI, deployment path. The cost itself was a prioritization mechanism. Only ideas that survived a viability check got built.

That barrier is gone. Today, a prompt, an API key, and an afternoon get you a working demo. A LangGraph agent can be built in an evening. A RAG pipeline can be running by lunch.

This is great for product. I can test a hypothesis before writing a quarterly business case. A product manager can answer "would this help our users?" in days, not months.

But it also means the filter is gone. Now every idea can get a prototype, every prototype a demo, every demo enough excitement to justify keeping it alive. And every live pilot has a run cost: API calls, cloud infrastructure, developer attention, security review cycles, roadmap slots.

The prototype is cheap. The pilot is not. The production system is expensive. The distance between those three stages is where most AI budgets quietly bleed out.

HBR Noticed the Same Pattern

I started seeing this pattern in my own portfolio before I saw it in print. Then HBR published three pieces in six months that described exactly what I was living through.

In November 2025, Goutam Challagalla, Mahwesh Khan, and Fabrice Beaulieu (IMD/BCG) published "Stop Running So Many AI Pilots". Subtitle: "Instead of testing lots of use cases across the company, pick one area and go deep." They used Reckitt as their case study. Reckitt found use cases spanning the business — presentations, customer support, procurement. Each guaranteed time savings. But the executives realized "the effort wouldn't transform the company's strategy or create a meaningful advantage. They were hoping for something more dramatic, not just marginal efficiency improvements."

That sentence hit me. We had pilots that worked, saved time, produced decent demos. But they were not changing the product. They were making the same product slightly faster.

The same month, Ania Masinter published "Prioritizing AI Investments That Create Real Value" as an HBR Executive Playbook. Her argument: "It's time for companies to move from experimentation to disciplined, focused investment in AI." She referenced the 2025 Wharton-GBK AI Adoption Report on executive pressure for AI ROI. That pressure is not theoretical — I feel it every quarter.

Then in January 2026, Faisal Hoque, Erik Nelson, Tom Davenport, and Paul Scade published "Manage Your AI Investments Like a Portfolio". Their framing: "Business leaders now face intense pressure to transform their organizations with AI, even though the technology, public attitudes, and the competitive landscape are all still in flux. The result is often too many pilots with too little coordinated oversight." They cited an IBM study finding "isolated, piecemeal deployments," a Deloitte report on "limited buy-in by senior executives," and McKinsey's State of AI 2025 finding "weak linkage to strategic goals."

Three articles. Three author groups. One diagnosis: too many pilots, too little oversight, too little connection to strategy.

The Question That Changed How I Think

Larry Page, co-founder of Google, famously said: "Put more wood behind fewer arrows." The phrase is sometimes attributed even earlier to Scott McNealy, co-founder of Sun Microsystems. The idea is old and simple: when you have limited strike capability, concentrate resources on fewer bets so each one hits harder.

The opposite is the scattergun: build lots of small arrows, fire them all, hope something sticks. In AI, that means launching pilots across every department, demoing them all, declaring whatever survives as "the strategy." It feels productive — your board sees activity, your teams feel empowered. But every live pilot consumes maintenance, infrastructure, security review, and trust. A pilot that does not get retired becomes a permanent tax. And it fragments your roadmap — you cannot ship a coherent product when engineering is maintaining six pilots that were never supposed to become products.

The question is not whether you should experiment. You should. The question is whether you experiment to explore — cheap and healthy — or to avoid committing, which is expensive and corrosive. I do not think there is a universal right answer. There is the answer that fits your capacity, your P&L, and your market's window.

The Framework That Helped: GIST

I did not invent the portfolio model from scratch. I borrowed from GIST — Goals, Ideas, Step-Projects, Tasks — a framework created by Itamar Gilad during his time at Google. The piece that matters for this conversation is the transition between Ideas and Step-Projects.

Ideas are a hypothesis bank. Everything goes in. Nothing is committed. The low cost of AI prototyping makes this stage richer than ever — you can test more hypotheses in a week than you used to in a quarter. That is the advantage.

Step-Projects are the gate. Small, timeboxed experiments with a measurable question and a kill criterion. An idea only becomes a step-project when it has a hypothesis, an owner, and a metric. The prototype proves technical feasibility. The step-project proves it is worth scaling.

That transition — from idea to step-project — is where most AI portfolios fail. The prototype is cheap, the demo is impressive, and the idea skips the gate and goes straight to production. GIST forces a different discipline: scaling is earned at the step-project gate, not granted at the demo.

I am not claiming we adopted GIST wholesale. We borrowed the principles and adapted them to an organization with monthly planning, security review cycles, and a finance team that wanted ROI narratives. But the core insight — ideas are hypotheses, experiments are gates, scaling is earned — is what made the portfolio work.

Two things nobody tells you about the gate

There are two problems I see teams hit at the step-project gate that GIST does not fully solve on its own. Both are culture problems dressed up as process problems.

How much do you build to test the hypothesis? This is the question I get asked most. Some teams build too little — a thin wrapper around an API call that cannot answer the actual question. Some teams build too much — a full production system with auth, monitoring, and a deployment pipeline, as if the step-project is already the product. And some teams build the whole thing, skip the gate entirely, and ship it. The right amount is the smallest thing that answers the hypothesis with enough confidence to make a scaling decision. Not more, not less. That judgment comes from practice, not from a framework. I have seen teams spend three months building what should have been a two-week step-project, and I have seen teams ship a one-evening prototype to production because it looked good in a demo. Both are failures of the same instinct: the inability to distinguish "enough to learn" from "enough to ship."

Killing a bad idea is a success. This is the culture shift. GIST treats a killed step-project as a healthy outcome — you learned the idea was not worth scaling, and you saved the organization months of wasted effort. Most organizations do not work that way. In most cultures, a killed project feels like a failure. The person who proposed it feels embarrassed. The team that built it feels like they wasted their time. The sponsor who championed it feels political loss. So instead of killing, the project gets extended. It gets reframed. It gets "one more quarter." And it quietly becomes a permanent tax on the organization.

This is not a process problem. You can have the best kill criteria in the world, and it will not matter if the culture treats a kill as a career risk. The shift has to come from leadership. When a step-project gets killed, the response should be: "Good. We learned something. What is the next hypothesis?" Not: "What went wrong?" The first response builds a culture of evidence. The second builds a culture of self-protection.

I made this change deliberately. The first three times we killed a step-project, I said the same thing in the review: "This is the system working. We saved the organization from scaling something that would not have earned its budget." By the sixth time, teams stopped defending dying projects. They started bringing their own kill recommendations. That is when I knew the culture had shifted.

What We Tried

We shifted from an ideas list to a portfolio model. The goal was to make experimentation deliberate and scaling earned. Every stage had a gate, and every gate had a P&L implication.

Four stages, not a backlog

I created a simple portfolio board: explore, validate, scale, retire.

• Explore — sandbox. Small budget, two weeks, no production commitments. You get a slot if you can state the question and the metric that answers it. The prototype is free. The slot is not.
• Validate — pilot stage. Real users, real data, real success metrics. Required: risk tier, baseline measurement, rollout plan, named product owner. No owner, no pilot.
• Scale — production. Not everything that passed validation scaled. We scaled what fit the month's strategy and had operational backing. Scaling meant a roadmap commitment, a budget line, an adoption plan.
• Retire — pilots that stalled, demos without owners, use cases too small to justify maintenance. We made retirement a normal part of the lifecycle, not a failure. A retired pilot freed capacity. That capacity was a product asset.

Kill criteria upfront

Every use case entered the portfolio with kill criteria defined before it started:

• No data after 30 days? Killed. No baseline, no decision to scale.
• No product owner after pilot? Killed. An orphan pilot is a future maintenance burden.
• No adoption path? Killed. A tool with no users is a demo, not a product.
• No measurable impact against baseline? Killed. If you cannot prove it worked, it did not work.

The criteria were kindness, not punishment. They prevented zombie work and the slow accumulation of run costs nobody audits.

Monthly portfolio review as a decision forum

Not a status meeting. A decision forum. Finance, Security, and product owners in the room. Moves between buckets required a decision. Kills required a decision. Scale commitments required a decision. By the third review, people came prepared with data instead of defending work that had not produced evidence.

The Uncomfortable Moments

The first exec-sponsored kill. A VP pushed for a customer-facing support chatbot. Six-week pilot, 12% deflection — decent but not transformative. The kill criteria said no adoption path: the contact center team had not committed to workflow changes. I had to say no to a senior leader in a room full of peers. The P&L reality: extending meant another quarter of API costs, another developer's partial allocation, another security review — all for a use case with no path to production. The cost of saying yes was not zero. It was the cost of every other bet that slot could have funded.

Two teams building the same thing. Month two: the portfolio review surfaced that engineering and operations were both building anomaly detection for the same data pipeline. Neither knew. We merged the efforts. The transparency helped — it was not about who was right, it was about paying for the same outcome twice.

The quick pilot that consumed three months. A "two-week" invoice reconciliation pilot stretched to three months. The tool worked. The demo was impressive. But finance had not been consulted on workflow integration, the data pipeline needed unscoped changes, and the success metric was never baselined. Three months of developer time and API costs. Zero documented impact. That one was on me — I used to reward activity because it was easier to see than outcomes. The P&L does not reward activity. It rewards outcomes.

What I Think the Evidence Shows

The surprising result: fewer bets created more momentum, not less. With 15 pilots simultaneously, each got a fraction of the attention it needed — the platform team context-switched across five integrations, security review was a bottleneck, adoption planning was an afterthought. None had enough depth to become a real feature. They were all shallow.

With four bets, each got real ownership. The platform team went deep on one integration. Security review happened early. Adoption planning started before launch. The four survivors had enough depth to become products — not demos, but features with users, metrics, and a roadmap.

The Reckitt case followed the same pattern. They picked one area and went deep. Their executives said they "were hoping for something more dramatic, not just marginal efficiency improvements." That instinct — to look for the bet that changes the product, not just shaves hours off a process — is the real work of a product leader. AI did not create that instinct. It made the cost of ignoring it visible faster.

What I Would Do Differently

• Start the rubric simpler. Our first scoring had twelve dimensions. Teams spent more time gaming the rubric than building. We cut it to four: value, feasibility, risk, adoption readiness. That was enough.
• Invest earlier in baselines. "Impact" is a debate without a baseline. Without one, every product decision is a story. With one, it is a number. Finance trusts numbers.
• Invite Finance and Security in from day one. We brought them in at month four. Every late-stage challenge could have been an early-stage conversation. Finance in the room makes the cost of yes visible.
• Track fully loaded cost from day one. API costs, cloud, developer time, security review, PM attention. The prototype is cheap. The pilot is not. Know the difference before you scale.
• Create a small fast-experiments fund outside the portfolio. Not every idea needs the formal process. The rule: fast experiments cannot enter validate without the portfolio review. Exploration should be cheap. Scaling should not be accidental.

What I Am Still Uncertain About

I do not have a clean ROI number for the portfolio approach. I can tell you we retired 14 pilots in six months, merged 6 duplicates, and avoided scaling two use cases that would have failed in production (one with a projected $180K annual run cost). But I cannot give you a single number that proves the portfolio was worth it.

The strongest evidence is cultural and financial. Teams stopped bringing half-formed ideas. They started bringing ideas with baselines, owners, and cost estimates. The run cost of our AI portfolio stopped growing without corresponding growth in measurable impact.

This approach can feel overly bureaucratic to small teams. If you have three engineers and one AI use case, you do not need a portfolio board. You need to ship. The portfolio earns its keep when demand exceeds capacity and politics distorts prioritization. I am not sure where the exact threshold is.

I am also uncertain about the balance between exploration and focus. Too much focus and you miss emergent opportunities. Too much exploration and you never build depth. The Reckitt case suggests going deep wins. But I have seen the scattergun surface an unexpected winner nobody would have prioritized. I do not think there is a formula. There is a discipline — the discipline of making the tradeoff visible and revisiting it regularly.

Closing

If everything is transformative, nothing is.

AI lowered the cost of the first prototype to near zero. That is a gift. But it also removed the natural filter that forced teams to think before they built. The result, as HBR documented across three pieces in six months, is too many pilots with too little oversight and too little ROI.

The fix is not to stop prototyping. The fix is to make scaling a deliberate decision — a product decision, a financial decision, a strategic decision.

Larry Page was right. Put more wood behind fewer arrows. Not because small experiments do not matter — they do, and the low cost of running them is an advantage you should use aggressively. But because the organization that can choose which bets to concentrate on is the one that turns AI from a backlog of demos into a product strategy that ships, earns its budget, and shows up on the P&L as something other than a cost center.

I do not think this is the only answer. I think it is the answer that fit my organization, my P&L, and my moment. Yours may be different. But the question is the same.

Your AI backlog is not a strategy. It is a P&L liability until you prove otherwise.

Three Questions for You

• Are you building many small arrows or putting more wood behind fewer? What would change if your CFO asked you to justify the run cost of every live pilot — could you?
• What kill criteria do you use, and do you actually enforce it — or does it bend for the loudest sponsor? Does it bend for your VP?
• Where has the low cost of AI prototyping helped you, and where has it created pilots that are quietly costing you engineering capacity you cannot afford? I genuinely want to hear both sides, because I am still calibrating the balance myself.

References

1. Challagalla, G., Khan, M., & Beaulieu, F. (2025). "Stop Running So Many AI Pilots." Harvard Business Review, November–December 2025. hbr.org/2025/11/stop-running-so-many-ai-pilots

2. Masinter, A. W. (2025). "Prioritizing AI Investments That Create Real Value." HBR Executive Playbook, November 2025. hbr.org/2025/11/prioritizing-ai-investments-that-create-real-value

3. Hoque, F., Nelson, E., Davenport, T., & Scade, P. (2026). "Manage Your AI Investments Like a Portfolio." Harvard Business Review, January 2026. hbr.org/2026/01/manage-your-ai-investments-like-a-portfolio

4. Gilad, I. GIST Planning: Goals, Ideas, Step-Projects, Tasks. Framework developed at Google. See: ProductPlan Glossary — GIST Planning.

5. Page, L. (attributed). "Put more wood behind fewer arrows." Co-founder, Google. The phrase is also attributed to Scott McNealy, co-founder of Sun Microsystems. See: English Stack Exchange discussion.

6. IBM Institute for Business Value. "CEOs Double Down on AI While Navigating Enterprise Hurdles" (May 2025). newsroom.ibm.com. Key finding: 50% of surveyed CEOs report that rapid investment has resulted in disconnected, piecemeal technology within their organization.

7. Wharton-GBK AI Adoption Report (2025). Referenced in Masinter (2025) regarding executive pressure for AI ROI. ai.wharton.upenn.edu.

Part of the "AI Leadership in the Real World" series: how leaders turn AI from scattered pilots and executive excitement into governed, adopted, measurable business capability.

TL;DR: In METR's 2025 randomized study, experienced open-source developers took 19% longer when using AI tools, even though they believed they were faster. I do not read that as a case against AI. I read it as a reminder that the win is not outsourced thinking. The win is knowing where AI gives you leverage, where curiosity matters, where caution matters, and where a human needs to pull the work back into their own hands.

I was not skeptical of AI coding tools when they started showing up in my teams.

If anything, I was predisposed to like them.

The early results were easy to love. Pull requests appeared faster. Scaffolding was faster. Boilerplate that used to take half an afternoon now showed up in minutes, formatted cleanly, with tests that looked respectable at a glance. It felt like one of those rare moments in engineering where a tool genuinely changes the slope of the work.

And to be clear, some of that story was true. We were getting faster in real ways.

For a while, I let myself believe the obvious conclusion: faster generation would naturally translate into cheaper software.

What broke that story for me was not a dramatic outage. It was a small bug that made me uneasy for a different reason.

A junior engineer shipped a change that looked clean. The code read well. The tests passed. The comments were better than most human comments. Nothing about it looked reckless. But the logic was subtly wrong. It handled the happy path, and it quietly misread an API response shape when an optional field was absent. Nobody caught it in review. I did not catch it in review. We only found it after the behavior downstream stopped lining up with what users expected.

The bug itself was fixable. That was not the part that stayed with me.

What stayed with me was the feeling that we had approved code none of us had really interrogated.

That was the moment the question changed for me. The question stopped being, "Can AI help us write code faster?" The answer to that was already yes. The better question was, "How do we use that speed well once code becomes cheap to produce?"

The Painful Truth

The real cost of software was never typing.

It is understanding the code later. Reviewing it well. Changing it safely. Operating it under pressure. Trusting it when the signal is noisy and the stakes are high.

That is why "faster to build isn't cheaper to own" has become the phrase I keep coming back to.

AI has changed the economics of generation. It has not removed the economics of ownership. If anything, it has made them more visible.

I think that is good news, if we are honest enough to respond to it.

When code gets easier to produce, review quality matters more, not less. Testing matters more. Good templates matter more. Clear architecture matters more. Observability matters more. The teams that ignore that shift can still look productive for a while. But they are doing it by borrowing against future understanding.

That is the trap. But it is also the opportunity. Once you see where the real constraint moved, you can redesign around it.

The Counter-Intuitive Part

The hardest part to admit is that more output can make a team feel healthier even while the system is getting worse.

That is one reason the METR result landed so hard for me. In their 2025 randomized controlled trial, experienced open-source developers working on their own repositories took 19% longer with AI tools, despite expecting to be faster. The researchers were careful not to overgeneralize the finding, and they later published updated results as tools improved. But the 2025 result captured something important: human perception of speed can be badly miscalibrated when AI is in the loop.

That matched what I was seeing in a less rigorous, more lived way.

People felt productive because visible output rose immediately. But the bottleneck moved. We were spending less time producing first drafts and more time validating whether those drafts deserved trust. Review started to feel heavier. Subtle errors became easier to hide inside plausible-looking code. The output looked finished long before the thinking behind it was finished.

That does not mean AI is disappointing. It means the leverage is real, and leverage changes where discipline has to live.

I keep thinking about a line from GitHub's 2025 post on spec-driven development: AI-generated code often "looks right, but doesn't quite work." That is exactly the kind of failure mode that wears teams down. Not obvious garbage. Plausible output. Code that passes the glance test and fails the understanding test.

Thoughtworks writers have been circling the same truth from a different angle. Birgitta Boeckeler's work on coding assistants keeps returning to the fact that these tools are useful and unreliable at the same time. Helpful enough to change the workflow. Unreliable enough that your judgment still does the real safety work.

That combination is what makes leadership more deliberate. You cannot say yes to everything the tool can do. But you also should not cling to work the tool can handle perfectly well.

That is the new management muscle: know where to let go, and know where to pull back.

Where The Debt Hides

In my experience, the first cracks do not show up in model demos or hackathon prototypes. They show up in ordinary team mechanics.

Reviews get shallower because the volume goes up before the review habit changes.

Tests look more complete than they really are because AI is good at producing the shape of rigor.

Patterns start to drift because every engineer can now generate a slightly different local optimum.

Security and dependency hygiene become easier to neglect because more code enters the system faster than humans can reason through it.

And junior engineers can suddenly ship polished-looking work long before they can explain the tradeoffs behind it.

That last one worries me most, because it is also where the upside is largest if we handle it well.

I have sat with junior engineers whose output improved dramatically once they started using AI tools. Faster features. Cleaner structure. More complete tests. And then I would ask them to walk me through why this pattern, why this library, why this failure mode, why this boundary. Sometimes they could. Sometimes they could not. Not because they were weak engineers. Because some of the reasoning had been outsourced before it had been learned.

That is not a tooling complaint. It is a coaching problem, a review problem, and eventually a leadership pipeline problem.

Used well, AI should accelerate learning, not replace it. It should get people to better questions faster. But that only happens if we keep insisting on thought.

What I Changed

I did not respond to this by banning AI tools. That would have been lazy, and it would have missed the point.

The problem was not that the tools existed. The problem was that the rest of our SDLC had not adapted to their existence.

So I changed the system in a few specific ways.

The point was not to slow teams down. The point was to make the speed usable.

First, I made standards explicit. We wrote down what kinds of changes needed more scrutiny, what kinds of code required stronger tests, and where a second reviewer was mandatory. The value was not in the policy document itself. The value was in removing ambiguity. Teams should not be improvising their AI norms PR by PR.

Second, I pushed harder on automated checks. SAST, secrets scanning, dependency scanning, linting, type checking, coverage expectations. None of this is new. But when code volume rises, basic engineering hygiene stops being optional. Humans do not scale attention linearly. The checks have to absorb more of the routine skepticism.

Third, I changed what I asked reviewers to optimize for. I wanted fewer style passes and more reasoning passes. Less "this naming feels off" and more "do we understand why this works, what assumptions it makes, and how it fails?" That sounds obvious. It is not how many reviews actually happen under delivery pressure.

Fourth, I changed the conversation with junior engineers. If AI can generate a decent first draft, then learning has to move up the stack. I started asking for explanation, tradeoff thinking, design notes, and failure-mode reasoning much earlier. If the code arrives faster, judgment has to be trained more deliberately, not less.

None of these changes are glamorous. None of them look exciting in a leadership update. But they are the difference between using AI inside a disciplined system and letting it quietly deform the system around it.

They also made me more optimistic, not less. Once the standards were clearer, the benefits of AI became easier to trust. People moved faster without pretending speed was the same thing as progress.

What This Changed For Me As a Leader

The biggest shift was not technical. It was managerial.

Before AI-assisted coding, I could tolerate a surprising amount of local inconsistency because code arrived at a human pace. Reviewers had time to absorb context. Juniors had time to learn while writing. Teams could afford some sloppiness because the throughput of the system put a natural brake on how much ambiguity entered at once.

That brake is weaker now.

When generation gets cheap, ambiguity compounds faster. Weak review habits get exposed faster. Poor architectural boundaries get punished faster. Gaps in coaching show up faster.

So leadership has to get more opinionated about quality, not less. More explicit about standards, not less. More serious about the economics of maintenance, not less.

And also more curious. Which tasks should humans stop doing manually? Which first drafts are no longer worth precious engineering time? Where does AI genuinely free people up for better work? Those are optimistic questions. They are worth asking.

That is why I do not think the right leadership question is, "How do we get everyone using AI?"

The better question is, "What has to change in our review model, our test discipline, our templates, and our coaching so that AI speed becomes sustainable leverage instead of downstream chaos?"

That is a much less exciting question. It is also the one that matters.

Honest Limitations

• The strongest quantitative evidence in this article comes from public studies, not from a controlled internal experiment I can publish in detail.

• METR's 2025 result was explicitly time-bound, and even METR notes that later tools produced different outcomes. The exact number is less important than the underlying lesson about trust, review, and perceived speed.

• Some teams are absolutely getting meaningful gains from AI-assisted engineering, especially where the underlying SDLC was already strong. I do not think the lesson is "AI slows everyone down." I think the lesson is that local speedups do not automatically become system speedups, unless the rest of the engineering system evolves too.

• This piece focuses on engineering ownership cost, not on every possible benefit. AI is clearly useful for scaffolding, exploration, boilerplate, learning unfamiliar areas, and helping people get unstuck. I use it for those things myself, often.

• I still do not think we have a clean answer for how junior engineers should learn judgment in a world where the first draft is often written by a model.

Closing

AI made code cheaper to produce. It did nothing to make unclear, untested, poorly owned software cheaper to live with.

The teams that navigate this well will not be the ones with the most demos or the highest PR counts. They will be the ones that stay optimistic enough to use the tools aggressively, curious enough to keep learning where the leverage really is, and cautious enough not to outsource their thinking.

Because the real risk is not that AI writes bad code.

The real risk is that teams mistake fluent output for reduced responsibility.

Use the leverage. Keep the judgment.

References

• METR (July 2025) — Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity. The core result: experienced open-source developers took 19% longer with AI tools in that study, despite believing they were faster.

• GitHub Blog (September 2025) — Spec-driven development with AI: Get started with a new open source toolkit. Strong framing for why AI code can "look right" while still missing intent.

• Martin Fowler / Thoughtworks collection — Exploring Generative AI. A broad set of practitioner essays on coding assistants, developer judgment, unreliability, and workflow change.

• Thoughtworks / Birgitta Boeckeler — I still care about the code. Good articulation of why fluency from LLMs does not remove the need for engineering judgment.

• Thoughtworks / Birgitta Boeckeler — The role of developer skills in agentic coding. Useful support for the point that developer skill still determines whether AI assistance helps or harms.

TL;DR: Across separate 90-day cycles (2018, 2023, 2025), we shipped platform bets that hit real adoption in 90 days (telemetry: ~20 DAU / ~150 MAU; Backstage IDP: ~30 DAU / ~500 MAU) and a CI bet that saved $120K/month while running ~30% faster on AWS self-hosted runners (EC2 for Linux, Dedicated Hosts for macOS) than on GitHub-hosted runners. In 2025, we also had 10 AI ideas and 0 production outcomes, and we killed one popular bet after ~1,500 PRs and 65%+ unhelpful feedback. The marginal cost per commit-run (CI + tests + LLM review steps) was about $6.

I do platform work, sometimes with AI. I have also been the person writing the slide deck and cheering the demo.

The prototype is usually the easy part. The hard part is owning the consequences.

The Painful Truth

If you are an IT leader or an engineering leader in SaaS, you do not get paid for "starting" AI work. You get paid for shipping something people use.

The fastest way to lose credibility is to build a portfolio of pilots that are all "almost ready" and still cannot answer one basic question:

What changed for customers or engineers this month?

The Counter-Intuitive Data: Engagement Is Not Value

In 2025, one of our most popular ideas was an AI code review bot.

It sounded like a shortcut to faster PRs. It integrated directly into the PR workflow.

Then we shipped the beta and got a very clear answer.

Time window: 2025 (a 90-day cycle)

This was the success bar we set going in:

• Engagement: > 80% meaningful interaction (author opened the review and left a signal: thumbs up/down, comment, or follow-up action)

• Competitive: > 25% more helpful than Copilot-style review (Copilot stayed on, our tool was added in the same path; compared thumbs up/down + which suggestions were accepted)

• Satisfaction: > 70% satisfied across our language mix (Swift, Kotlin, BrightScript, TypeScript, C++, Java, Go, Python)

• Outcomes: iteration ~4-5 commits/PR to ~2, PR cycle time ~2.5 days to ~1 day, and human reviewers ~2-3 to 1 (AI-first pass, then a single human pass)

The spreadsheet logic that made this feel like a no-brainer was simple:

• Volume: ~8,000 PRs/month

• If AI review saves ~3 iteration commits per PR, that is ~24,000 fewer commit-runs/month

• At ~$6 per commit-run, that is roughly $144K/month in marginal pipeline cost

That is what made it tempting.

(Caveat: commits do not always map 1:1 to CI runs. Some pushes batch commits. Some pipeline cost is fixed. Treat this as an order of magnitude model.)

• ~1,500 PRs processed across a fraction of the traffic over a 90-day period

• developers interacted with it inline (reactions, replies)

• and then, when asked if it helped: 65%+ said no

The most humbling part was that it did not just fail to help. Early on, it made things worse.

Because the suggestions were noisy, authors started reading them line by line just to discard most of them. In the first stretch of the beta, PR cycle time went up by ~0.5 days.

And the social signal shifted fast. By around day 60, engagement in the targeted demographic had fallen into the single digits. Not because we turned it off. Because people stopped believing the suggestions were worth reading.

The quote that ended the debate was not from a dashboard. It was from a developer:

"Maybe 1 useful comment out of 10, and some are wildly wrong."

All-in, the marginal cost per commit-run (CI + tests + LLM review steps across multiple pipeline stages) was about $6.

That combination (low trust + real cost) forced us to do the most strategic thing we did that quarter:

We killed it.

The detailed diagnosis was not "LLMs are bad". It was that quality was uneven across languages.

The most cost-effective models produced too many false positives. We saw the strongest performance on Java and Python, and the weakest on Swift and BrightScript. In 2025, that gap was large enough that it broke the overall experience. That is also why I am careful about universal claims here: the model landscape changes fast.

What I got wrong was basic. I assumed that wiring it into the PR workflow would make it "obviously valuable". It made it visible, not trusted.

If you want a public data point that rhymes with this: a 2026 empirical study of 1.02M PRs found that AI- or agent-involved review can correlate with faster decisions, but those efficiency gains do not necessarily translate into better review quality. That matched our experience uncomfortably well.

• Zhong et al., 2026: "From Human-Centric to Agentic Code Review" (arXiv:2607.13196) https://arxiv.org/abs/2607.13196

Table: The Shift That Actually Matters

This is the smallest table that helped me stop lying to myself:

The Only AI That Mattered (Faster Feedback)

The only AI that mattered was not a model choice.

It was the AI that shortened the distance between an idea and undeniable feedback.

Sometimes that meant we got proof of value quickly. Sometimes it meant we got proof of non-value quickly.

The failure mode is the middle: months of activity, no adoption, and no courage to call it.

If you have ever watched a team quietly keep a pilot alive because killing it is socially hard, you know what I mean.

Concretely, this is what "faster feedback" looked like for us in 2025:

1. We used AI to shape the experiment. We pulled CI and PR telemetry to choose a realistic slice: languages, repos with complex PRs, and a mix of teams.

2. We compared three models with one rubric. What developers accepted, thumbs-up/thumbs-down, and LLM cost.

3. We forced a decision. We wrote a closure report and made the call: ship, kill, pivot, pause, or continue.

If you take nothing else from this post, take that: AI can accelerate your learning loop. It cannot substitute for your willingness to accept the answer.

Sometimes the answer is "ship". The discipline is treating that answer the same way you treat "kill": with thresholds, owners, and a plan.

The 90-Day Bet System

Here is what we ran. It is boring on purpose.

1. Pick a small number of bets for 90 days. If you cannot say what is out of scope, you do not have a strategy.

2. Name owners. One business owner, one technical owner, both accountable.

3. Define graduation criteria before build. Adoption target, workflow change, operational readiness.

4. Define kill criteria before build. What would make you stop, even if it is politically awkward?

5. Keep a decision log. If you do not write decisions down, you will relitigate them.

Three Platform Bets (Across Three Different Years)

These were separate 90-day cycles. Not one giant program.

Bet #1 (2018): Make telemetry end in action

In 2018, we did not call this "AI strategy." We called it "stop paging the wrong person" and "stop guessing."

The bet was straightforward:

If you can detect and correlate problems early, and route them to a clear action, you waste less human time.

Adoption: started at 0. By day 90: ~20 daily active users (platform and on-call engineers) and ~150 monthly active users.

Impact (anonymized internal estimate): ~$5M avoided across churn risk, support ops, and vendor/software spend.

That number is directional. The point is that a small set of people on the critical path can drive outsized outcomes when time-to-detection and time-to-action drops.

The best public piece I have found on why correlation matters in practice is Amanda Dias' writeup on Bing dialtone failure correlation. The connection to this bet is direct: it is not about "more telemetry". It is about taking multiple weak signals and turning them into one decisive operational story.

• https://www.linkedin.com/pulse/bing-dialtone-service-failure-correlation-amanda-dias/

Bet #2 (2023): Fix the CI layer before you optimize everything else

This one is not AI. That is the point.

We moved from purely GitHub-hosted runners to self-hosted runners on AWS.

• For Linux workloads: self-hosted runners on EC2.

• For macOS workloads: EC2 Dedicated Hosts (mac runners).

The public docs describe the tradeoffs plainly: you get more control, but you also own patching and maintenance. That mirrors the real decision: speed and control are not free.

• GitHub Actions: "Self-hosted runners" docs https://docs.github.com/en/actions/hosting-your-own-runners/about-self-hosted-runners

Our internal outcome was not subtle.

• It cost ~$120K/month less on AWS than it did on GitHub-hosted runners (direct runner infrastructure spend; does not include the engineering time to maintain images/runners).

• For the workflows that mattered, it ran ~30% faster (dedicated hosts for macOS, tuned EC2 runners + our own images for Linux).

The reasons were boring and decisive:

• We could pick cheaper SKUs with the right mix of CPU and memory.

• We had a wider set of regions to choose from.

• We controlled the machine images (which eliminated a lot of "it works on runner A but not runner B" drift).

If you want a public reference point for Dedicated Hosts (used for mac runners), start here:

• AWS docs: "Amazon EC2 Dedicated Hosts" https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/dedicated-hosts-overview.html

(The GitHub self-hosted runner docs are already linked above.)

The human reason we made the bet was simpler:

When CI is slow or flaky, engineers adapt in bad ways. They rerun without reading. They merge past red builds. They stop trusting the signal.

If you are trying to prove value quickly, you cannot build on a signal people have learned to ignore.

Bet #3 (2025): Backstage as the front door for developer experience

Backstage is not an AI tool. But it is one of the highest leverage "AI strategy" moves I know.

If your dev experience is fragmented, AI just becomes another tab.

Backstage gives you an opinionated place to hang:

• service catalog

• ownership

• docs entry points

• templates and golden paths

Then AI becomes an accelerator inside a system that already makes sense.

For us, this was not abstract. The portal became the place where humans actually did work, like:

• onboarding a new engineer (GitHub org/repo/team access)

• requesting AWS access (account/role)

• requesting access to approved AI models (and requesting new models)

• raising support requests that file into Jira automatically

• provisioning / modifying / decommissioning services and infrastructure

Backstage is an open-source framework (CNCF project) for building developer portals. The reason I cite it is that it names the real platform problem most orgs are living: "restores order" to the sprawl by anchoring on a centralized catalog.

• https://backstage.io/

• https://github.com/backstage/backstage

Adoption: started at 0. By day 90: ~30 daily active users (platform + application developers) and ~500 monthly active users.

The difference in daily actives between these two bets is not a mystery.

• The telemetry bet worked even when people did not visit a dashboard much. Alerts and anomalies showed up in email and chat (Teams), and the dashboard was mostly read-only.

• The IDP bet required humans to click buttons, fill forms, and complete workflows. If the portal is the entry point, the UI becomes the work.

The humbling part here is that the win was not a clever AI feature. It was giving developers a place to go that made sense.

What I Would Tell My Past Self

You do not need a bigger AI roadmap.

You need the ability to run a few bets, measure adoption, and kill the ones developers do not trust.

And if you are wrong (you will be), be wrong quickly.

If that sounds harsh, it is. But the kinder version is worse: keep a tool alive that developers do not trust because admitting the miss is uncomfortable.

Related Reading (If This Resonated)

These are other pieces where I went deeper on parts of this story:

• Roadmaps and sequencing: https://pragmatic-engineer.hashnode.dev/the-roadmap-didnt-die-its-origin-story-did

• SDLC as the execution system: https://pragmatic-engineer.hashnode.dev/ive-been-doing-sdlc-for-25-years-ai-didnt-kill-it-it-made-it-more-useful

• CI trust and signal quality: https://pragmatic-engineer.hashnode.dev/stop-rerunning-an-ai-agent-that-tells-you-which-ci-failures-actually-matter

• Adoption and the human layer: https://pragmatic-engineer.hashnode.dev/what-if-middle-managers-are-the-real-ai-adoption-accelerators

Honest Limitations

• I am publishing a blended "cost avoided" estimate without the full spreadsheet. Treat it as directional.

• DAU/MAU are adoption proxies, not value in themselves.

• Some details are anonymized for obvious reasons. That can make the story feel cleaner than reality.

• The code review pilot is one example; do not generalize from it to "AI is useless". The lesson is about thresholds and kill criteria.

Closing

Platform strategy that ships is not a slide deck. It is a habit: pick a few bets, name owners, define kill criteria, and let real users be the judge.

References