The Test Passed. The Feature Was Wrong
Part 3 of 6 · Why AI Code Fails (And How to Fix It)
The Test Passed. The Feature Was Wrong.
The ops incident was contained. The migration ran. Two days later, the guest_orders collection was gone and the unified collection had everything. The tests passed. CI was green. The developer marked the task done.
One week after that, a product manager doing a walkthrough on a staging device noticed that the "Continue as guest" button was rendered smaller than the "Sign in" button. Not dramatically — maybe 20% smaller, positioned below it, slightly lighter font weight. Easy to miss if you weren't looking for it. But the product requirement had been explicit: the two options must be at equal visual weight. The business reason was specific too — the team had studied a competitor's checkout where guest was a small secondary link below the primary sign-in button, and conversion data showed it dramatically underperformed. Making the two options equal weight wasn't a design preference. It was the revenue rationale for the feature.
Nobody had tested this. The test suite confirmed that the "Continue as guest" button existed. It did not confirm that it was the right size, in the right position, with the right visual weight. There was no test for that — not because anyone decided not to write one, but because the AI that generated the implementation also generated the tests, and the tests confirmed the implementation's behaviour, not the requirement's intent.
The button was there. The test said so. The feature was wrong.
This is the AI self-testing trap, and it's the most reliable way to ship a feature that passes CI and fails the business.
When the same generation session produces both the code and the tests, the tests inherit the assumptions of the implementation. If the AI decides that "Continue as guest" should be a TextButton positioned below the ElevatedButton for "Sign in" — a perfectly reasonable UI pattern — and then generates tests for that implementation, those tests will pass for exactly the implementation that was generated. They won't ask whether that's what the requirement said, because the AI never read the requirement the way a human reads it: as a standard to be verified against the output, not a description to be translated into code.
The question an AI-generated test asks is: "Does this code behave consistently with itself?"
The question a Step 6 human reviewer asks is: "Does this code do what we agreed it would do?"
These feel similar. They are not. The first can be answered by running tests. The second requires a person who has the requirements open in one window and the implementation running in another, going through each acceptance criterion one at a time.
84% coverage is a metric that CI can measure. It tells you that 84% of the code paths are exercised by the test suite. What it doesn't tell you is which acceptance criteria those tests cover — or whether the 16% of uncovered paths includes the one that would catch the GDPR guest email opt-in that nobody tested, the edge case where a guest tries to return an order with a mismatched email, or the scenario where the post-purchase account creation offer fires at the wrong step in the checkout sequence.
The coverage gate in a sound AI-assisted process is set at 80% minimum, but it's treated as a signal rather than a goal. When coverage comes in at 65%, the question isn't "how do we write tests to get to 80%?" It's "which acceptance criteria did we miss?" Coverage below 80% on a new feature almost always means the ADR's edge cases weren't tested. It's a diagnostic, not a target.
But even at 84%, the tests can be green while the feature is wrong. That's what happened here. The coverage number didn't fail because the button existed and was testable. What wasn't tested was the specific property the requirement described — relative visual weight between two interactive elements. No automated test framework considers that question without being told to. It wasn't told to.
Here's what Step 6 actually looks like for the guest checkout feature.
The developer, before opening the PR, has the requirements file on screen. Not the code. The requirements. They go through the acceptance criteria one by one:
"Continue as guest" option is displayed at the same visual weight as "Sign in" — they open the app on the simulator. They look at both buttons. They compare. They do not run a test to answer this question. They look.
A confirmation email is dispatched within 60 seconds of order placement — they open the Cloud Function trigger configuration. They confirm it fires on the Firestore write event, not on a scheduled job. They don't infer this from the code structure. They check the actual configuration.
Guest order appears in admin order panel with identical fields to a registered-user order — they open the admin panel in staging. They complete a guest purchase. They find the order. They look at every field.
No account is created during the transaction — they check the auth users list in staging after a guest purchase. No new user. They don't trust the implementation. They verify the output.
This is human verification. It is not glamorous. It is not automatable in any useful sense. It takes twenty minutes per feature. And it is the only mechanism that can catch the kind of failure that CI is structurally unable to catch — the gap between a passing test and a satisfied requirement.
When the developer finds that the "Continue as guest" button is the wrong widget type, the PR doesn't open. The implementation is fixed. The test that should have caught it is written — checking that both buttons share the same widget class — and that test now lives in the CI pipeline permanently. The regression can never silently return, because fixing the root cause also closes the gap in test coverage.
The bug costs thirty minutes to find and fix at Step 6. It would have cost a sprint cycle to find in production: a bug report, an investigation, a fix PR, a new review, a new deployment, a regression check.
The test passing and the feature being correct are related but not the same thing. This distinction matters enormously in AI-assisted development because of how tests are generated. In a human-written codebase, tests are often written against the requirements — a developer reads the acceptance criteria and writes code that checks them. In an AI-assisted codebase where the same session generates both implementation and tests, the tests document what the code does. They're accurate. They're just answering the wrong question.
The requirement says: equal visual weight. The test says: button exists. Both are true. Only one of them tells you whether the feature works.
The fix isn't to distrust AI-generated tests. They're useful — they catch regressions, verify edge cases, confirm that the userId == null check behaves correctly, ensure the guest email index is queried in the right direction. The fix is to separate the question of "do the tests pass?" from the question of "does the feature meet the requirements?" — and to answer the second question with a human, using the requirements as the checklist, before the PR opens.
CI answers the first question automatically and reliably. Nobody has to ask it twice.
The second question requires a human, a running implementation, and a requirements file. It takes twenty minutes. It cannot be delegated to the test suite. And it catches the class of failures — visual weight, timing constraints, user experience specifics, operational workflow gaps — that no automated test will find unless someone explicitly tells it what to look for.
CI can tell you the tests passed. Only a human can tell you the feature is done.