The AI Made a Decision Nobody Asked It To
Part 2 of 5 · Why AI Code Fails (And How to Fix It)
The AI Made a Decision Nobody Asked It To
The ticket said: "Add guest checkout so users can purchase without an account."
That was it. No constraints. No context. No reference to the admin panel, the returns flow, the GDPR requirements, or the operations team's single-workflow policy. Just a task description in a sprint board and a developer who opened a chat window and got to work.
The AI did exactly what it was asked to do. By the end of the day, there was a working guest checkout flow. It passed tests. The developer reviewed it, saw nothing obviously wrong, and pushed it. Code review was a naming-conventions conversation. The PR merged on Friday afternoon.
On Monday, the operations manager noticed something odd in the admin panel. Guest orders weren't showing up.
The AI had created a guest_orders collection.
Not because it was a bad model. Not because there was a bug in the generation. Because when it hit the decision point — where should guest orders be stored? — it made a reasonable call with the information it had. A separate collection is a clean architectural choice. Keeps things tidy. Easy to query independently. Easy to explain to a new developer. The AI had no way to know that "tidy" and "separate" were exactly the wrong values to optimise for here, because nobody had written down the constraint that mattered: the admin panel was built around a single orders collection, the operations team had one workflow, and a second collection broke everything downstream.
The AI didn't know about any of that. It wasn't in the ticket. It wasn't in the context window. So the AI made a decision — a good-sounding architectural decision — and nobody noticed until the operations manager opened the admin panel on Monday.
This is what makes AI-assisted architectural decisions so much more dangerous than AI-assisted logic errors.
A logic error shows up in tests. A wrong loop condition, a null pointer, an off-by-one — these are detectable. You write a test, it fails, you fix it. The feedback loop is tight. The cost of discovery is low because the signal is immediate.
An architectural decision doesn't show up in tests. The guest_orders collection passed every test. The data was written correctly. The read queries worked. The confirmation email sent. If you wrote a test that said "does guest checkout work?" — yes, it did. The test would pass. The ops team's Monday morning discovery wasn't in any test file. It was in the gap between what the requirements said and what an operations workflow assumes.
Architectural mistakes live in the space between things. Between the code that works and the system that the code has to fit into. Between the feature that shipped and the feature the business actually needed. And because AI generation is fast and confident, these mistakes accumulate quietly — each one a reasonable-sounding call made from incomplete context, each one invisible until a human in a different part of the organisation bumps into the consequence.
There's a subtlety here that matters: the guest_orders decision wasn't random. The AI made it for reasons that are perfectly legible once you understand how generation works.
When a model generates code for a new entity — in this case, guest orders — it looks for patterns in the existing codebase to anchor its decisions. The existing orders collection has a userId field that's assumed to be non-null throughout the codebase. Queries filter by it. The order service passes it in constructor calls. The access control rules require it. Making userId nullable would require touching all of those places — a change with a wide blast radius. Creating a separate guest_orders collection, by contrast, leaves the existing orders model untouched. It's the minimal-change interpretation of the task.
This is the AI optimising for something real: backward compatibility. Don't break what exists. Make the smallest change that achieves the stated goal. That's a sound instinct in most contexts. It's the wrong instinct here, because the stated goal was underspecified — "add guest checkout" didn't communicate that the backward-compatibility constraint had a competitor: the operations team's single-workflow requirement. From the model's perspective, there was only one constraint. It satisfied it.
Understanding this mechanism changes how you think about the fix. The problem isn't that the AI made a wrong decision. The problem is that the AI was given a decision to make that a human should have already made. The ticket was a question disguised as a task. "Add guest checkout" is the answer to "how do we store guest orders?" before anyone has decided the question.
The fix, in this case, was to migrate the guest orders data into the main orders collection and update the schema. It took two days. But the more interesting question is: what would have prevented it?
Not a better AI. The model that made the guest_orders decision is perfectly capable of making the correct decision — it just needs the right information. Specifically, it needs one committed document that exists before generation starts and tells it what was decided before the AI was ever involved.
An Architectural Decision Record — an ADR — is exactly this. For the guest checkout feature, the ADR for order storage would look like this:
Decision: Store guest orders in the unified
orderscollection withuserId: nullandguestEmail: string.Rejected alternative: Separate
guest_orderscollection — this was evaluated and explicitly rejected because it duplicates all order management logic, creates a second surface for bugs, and violates the operations team's single-workflow requirement.
That's two sentences in the rejected alternatives section. Two sentences that, if they'd existed in the AI's context window before generation started, would have produced a completely different output. The AI reads the rejection. The rejected option is off the table. Generation continues with the unified collection as the only viable path.
The AI doesn't make the same mistake twice when the first mistake is documented. That's the point of an ADR — not to record what was built, but to record what was decided, including what was considered and ruled out. When that ADR is part of the generation context, the AI knows that guest_orders is off the table. Not because it's prohibited by some rule, but because the decision was already made by a human, for documented reasons, and the AI is executing within that decision.
There's a specific moment in this workflow that people consistently underestimate: the conversation that produces the ADR.
Someone has to ask: "How do we store guest orders?" And then: "What are the alternatives?" And then: "Why would we choose one over the other?" And then: "What are the constraints we're operating under?" Those four questions, asked before any code runs, are the conversation that produces the two sentences in the rejected alternatives section.
That conversation is engineering work. It's not overhead. It's not bureaucracy. It's the work that prevents a Monday morning ops incident. It's the work that means the next developer who touches the orders collection can read the ADR and know — immediately, without asking anyone — why userId is nullable, why there's no guest_orders collection, and why they shouldn't add one even if it seems like a cleaner approach for the feature they're building now.
The conversation takes about thirty minutes. The ADR takes about twenty minutes to write. The Monday incident it prevents would have taken two days to fix and two weeks to verify was safe.
The returns flow is worth dwelling on as a second example, because it illustrates a different class of AI-assisted gap.
When the developer asked the AI to implement guest order lookups for the returns flow, the AI saw a pattern in the existing codebase: the returns handler used userId to fetch the relevant order. For guest orders, userId didn't exist. The AI generated a new endpoint — getGuestOrderByEmail() — that queried guest_orders by guestEmail. Clean. Self-contained. Working.
What it didn't do was check whether the access control rules permitted unauthenticated reads of the guest_orders collection. Unauthenticated reads weren't in the existing rules. The new endpoint worked in testing because the developer ran it with admin credentials. It failed in production because guest users, by definition, are not authenticated.
This bug was subtler than the Monday ops incident. It was only discovered when the first guest customer tried to initiate a return and received a permissions error. The fix — adding an access control rule that permitted reads by orderId plus guestEmail pair — took thirty minutes. But the scenario it created — a live feature that silently failed for a specific user action — is the scenario that costs the most to recover from. Not because the code is hard to fix, but because the trust damage is real.
An ADR for order storage that included the returns flow as a design consideration would have prevented this. Not by anticipating the exact access control rule, but by naming the returns flow as a downstream dependency in the consequences section — which would have prompted the developer to check the access control rules before shipping, rather than discovering the gap when a customer couldn't initiate a return.
The AI is good at executing within decisions. It is not good at making decisions it doesn't know need to be made. That's not a limitation of the technology — it's a structural property of how generation works. The model sees what's in its context window. If the architectural constraint isn't there, the constraint doesn't exist as far as the model is concerned. It will make the most reasonable-sounding call with the information it has, and "reasonable" is not the same as "correct for this specific system with these specific constraints."
This is why the architectural decision has to come first. In writing. From a human who understands the system, the business constraints, and the implications of choosing one approach over another. The AI's job is to execute within the decision. The human's job is to make the decision.
When those roles are reversed — when the AI makes the architectural decision and the human reviews the code afterward — the review can't catch it. You're reviewing an implementation of a decision that was never visible. The diff looks fine. The tests pass. The ops team finds the gap on Monday.
There's a version of this story that engineering teams find uncomfortable, because it implicates the developer as much as the process.
The developer who pushed the Friday PR wasn't careless. They checked the code. They ran the tests. They opened the app and completed a guest checkout end to end. It worked. What they didn't check was whether the admin panel showed the order — because that wasn't in their mental model of "does guest checkout work?" It was in the operations manager's mental model. It was never in the ticket.
This is the gap that a structured AI-assisted process closes: not the gap between a careful developer and a careless one, but the gap between what a developer thinks "done" means and what every stakeholder in the organisation needs it to mean. A requirements file with explicit acceptance criteria — including "guest order appears in admin panel identically to a registered-user order" — makes that criterion visible before the PR opens. Not because the developer forgot it; because nobody had written it down as a criterion in the first place.
The Monday incident wasn't a review failure. It was a requirements failure that looked like a review failure. The code did exactly what the requirements said. The requirements didn't say enough.
The full picture of what the AI-assisted process needs to produce reliable output is not complicated: a decision record that existed before generation, a requirements file that described every acceptance criterion including the operational ones, and a developer who verified both before calling it done. Three documents. Two conversations. One of which could have happened in ten minutes on the Monday before the feature was built.
Instead, two days of migration. Two weeks of investigation. One operations manager who found the gap, because nobody else's workflow ever put them in the position to see it.
The cascading cost is worth calculating explicitly, because teams tend to feel the migration and forget the investigation.
The two-day migration is the visible cost: write the migration script, test it, run it in staging, run it in production, update the schema, redeploy the Cloud Functions, update the access control rules, verify the admin panel. Two days, one developer.
The two-week investigation is less visible but more expensive. A senior engineer spent roughly four hours a day over ten working days answering one question: is the fix isolated, or does it have implications we haven't found yet? That meant reading every query that touched the orders collection, every Cloud Function that wrote to or read from either collection, every access control rule that referenced order documents, every admin panel component that displayed order data, and every scheduled job that aggregated or reported on order data. Not to fix anything. Just to establish confidence that the migration hadn't introduced a new gap.
That's forty hours of senior engineering time. For a two-day fix. The fix itself was straightforward. The confidence-building that justified shipping the fix was the expensive part — and it was expensive because there was no document that said "here is everything that depends on where orders are stored." No ADR consequences section. No impact analysis from a planning step. Nothing that would let the engineer say "I've checked everything on this list" rather than "I've read everything I can think of and I think I got it all."
The ADR that should have existed for this feature takes forty-five minutes to write. It would have saved forty hours of investigation. The ROI is not subtle.
The AI made a decision nobody asked it to make, because the process never required anyone to make it first. That's the correctable part. That's what every article in this series is about.