C:\CHANGELOG> type v1-13-0-the-code-is-confident-the-tests-are-not.md
v1.13.0 · released · 4 min read · by

The code is confident. The tests are not.

CloudBees released its State of Code Abundance 2026 report in May, surveying over 200 enterprise technology leaders and finding that 81% reported production failures tied to AI-generated code. Despite 92% of leaders expressing confidence in the production readiness of AI-generated code, those same failures kept arriving. The report has a name for the financial piece: token anxiety, as finance teams struggle to forecast AI spend quarter to quarter. I have a name for the operational piece: verification lag.

The code ships faster than anyone can check it. The commits look clean. The test suite falls further behind every sprint. Then production fails and nobody can trace the cost back to a decision, a model, or a team.

I run ten production platforms. Three of them generate code with AI assistance now. The construction ERP at coenconstruction.com uses Claude to draft database migration scripts. The estimating SaaS at estimate.pro uses GPT to generate invoice validation logic. The review pipeline at Valhalla K9 uses a small model to write SMS template variations. Every one of those outputs hits a review gate before it touches production. Not because I distrust the models. Because I can't afford to debug generated code I didn't write and can't attribute to a prompt I didn't log.

Confidence is not the same as verification

61% of organizations' code is now generated by AI or written with AI assistance, and 64% of engineering teams say AI is widely or fully integrated into their workflows. The survey also found 70% reporting that test-suite maintenance is now a larger burden than writing code. That ratio is backwards. If verification costs more than generation, the economics only work if you skip verification. That's what's happening.

I added AI code generation to the ERP's reporting module in March. It cut feature delivery time from two weeks to four days. It also tripled the test maintenance burden, because every generated query needed five new test cases to cover edge conditions the model didn't consider. By May I was spending more time writing tests than I saved on the original feature. The confidence score on each commit was above ninety. The rollback rate went up forty percent.

I pulled the generation hook and went back to writing SQL by hand. Not because the AI was wrong. Because I couldn't verify it was right faster than it could write new code, and production doesn't care about confidence intervals.

Attribution is worse than the failures

Organizations score highest on ROI measurement confidence at 51% very confident, yet only 31% of AI spend can be attributed to specific business outcomes. That gap is the whole problem. You're confident you're getting value, but you can't say which dollar bought which outcome. The CloudBees data includes 46% saying the CTO or VP of Engineering is ultimately accountable for AI-related failures, while only 12% report having a dedicated governance function in place.

I track every API call by route, model, and outcome in a separate D1 table that lives outside the application database. When the invoice validation logic fails, I can see which model generated it, what the prompt was, how much the call cost, and whether the same model produced working code on the previous invoice. That audit trail is the only reason I can justify keeping AI generation turned on for any module. Without it, a failure is just noise. With it, I know whether to retune the prompt, swap the model, or write the logic myself.

The estimating SaaS had three production failures last month. Two came from AI-generated validation logic that passed unit tests but failed on real contractor data. One came from a caching bug I wrote by hand. I can attribute $47 in API costs to the two AI failures. I cannot attribute any revenue gain to the speed improvement, because the feature shipped the same week a competitor raised prices and we picked up twelve new customers. Did the AI help or did the market move? The log says I spent the money. It doesn't say I earned it back.

You can generate code faster than you can verify it, but you can't ship faster than you can trust it.

The test suite is the bottleneck now

I added a test coverage gate to the ERP two weeks ago. No commit merges unless test coverage stays above eighty-five percent and the new tests run in under twelve seconds. The AI can write a database migration in ninety seconds. Writing the tests for that migration takes eleven minutes. The generation is instant. The verification is not.

This is the same problem I wrote about in classifier design: the thing that scores the output has to be faster and cheaper than the thing that produces it, or you create a queue that never clears. The AI writes code. The human writes tests. The human is slower. The queue grows. Eventually you ship untested code or you stop shipping.

The construction ERP generates about two hundred lines of code per week with AI assistance now, down from eight hundred in April. I didn't slow the AI down. I just stopped merging commits that didn't have tests I could read and verify in the same session. The model is still confident. The test suite is still behind. The difference is I know which commits are verified and which ones are guesses, and only the verified ones touch production.

Confidence shipped the code. Verification caught the bug. The agents are in production, but the review gates are not optional.


— Cole Ciprari · Business Systems Architect · Worcester, MA
my résumé is an operating system → ciprari.ai · linkedin.com/in/coleos · cole@ciprari.ai
WAS THIS ANY GOOD?
Anonymous, one tap, no account. Tap again to undo.
▚▞ GET THE NEXT RELEASE
New releases Monday, Wednesday and Friday, plus the Sunday Rollout Report — the week's AI and tech news, summarized by a human with production access. No spam. Unsubscribe by emailing a mildly disappointed cole@ciprari.ai.
PHOSPHOR