It worked, so I shut it down
I spent about a week building an agentic coding orchestrator: a system that drives language models through a gated, test-first loop and lets them write code on their own. It worked. That is why I shut it down.
The orchestrator automated the implementation step. The bet was that if writing code becomes reliable and cheap, the hard work moves to either side of it: choosing what to build, and confirming it did what you meant. The only way to test that was to build the thing and run it on real work, so I did. The work was the project's own codebase: the LangGraph orchestration, the nodes, the tooling. State, control flow, gates, boilerplate. About the easiest case there is for code generation, since it was greenfield and self-authored with no legacy. The bet only holds where the contract can be made stable. Exploratory and performance-critical work is different.
Even in the easy case, deciding what to build was harder than building it. The spec was always the executable contract. But implementation still took most of the time and attention, and AI changed that.
Here is what one task went through.
First, a RED phase. A model wrote failing tests, which had to clear deterministic gates before any of it counted. Did they fail for the right reason, or just fail to compile? Did they pass static analysis and type checking? Did every test the spec named exist? Only then did a reasoning model review them against a behavioral checklist. Were they testing real behavior, or asserting that the code returns the type it returns? If the reviewer found them weak, its objections became the next prompt and the loop went again.
Then a GREEN phase. The model wrote code until the tests passed and static analysis came back clean. If a task stalled, the orchestrator escalated to a stronger, costlier model with a fresh budget.
Each step got a context built from scratch, holding only what it needed: the rules for the files it was about to touch, the files it had to read, the tools it could call. A running record tracked every failure as open, resolved, or regressed, and shrank as things got fixed, so the model never re-read noise. A plan took about a minute to read before the loop ran, a result about a minute to check after.
When a run failed I did not patch the run. I found the root cause, wrote it down as a general rule, and re-ran the same task on a fresh branch to prove the rule closed the gap.
The first substantial task took ten reruns to pass, each failing a different way. One was a loop that kept retrying a failing call until it ran out of budget. Another was a pattern where each new fix quietly dropped an earlier one, which I closed by feeding every rerun the previous review's findings as a do-not-regress list. Every failure became a rule and a commit, and run ten passed clean.
The implementation, every task and every rerun, cost just over two dollars on cheap models. I am not claiming that as a saving. The real cost was the week of building and the rule-by-rule debugging. What two dollars bought was the ability to re-run an entire task to prove a single rule, which is the only reason that discipline was affordable. I used the frontier model where it counted: writing the spec at the start, reviewing the result at the end.
I expected writing the code to be the hard part, and the first task looked that way. Every rule I added was about making the generated code compile, type-check, and not break what already worked. Then the next two tasks passed on the first run, and so did nearly every task after. The failures that remained were about the spec. A test no implementation could satisfy. Output that changed from one run to the next. A spec loose enough that a useless implementation passed it.
Thirteen tasks ran through the loop in all. Once it was reliably producing passing pull requests, I had a frontier model review them. It kept finding the same thing: holes in the spec. The loop worked, and cheaply. But it was doing the wrong job perfectly. The problem was upstream of it, and no amount of polishing the loop would move it there. So I stopped building the orchestrator. A finished one would have handled a narrow class of tasks cheaply for a release or two, until the mainstream tools caught up.
The loop builds exactly to the contract it is handed. An under-specified contract gets built faithfully, passes every gate, and ships wrong. One task asked for a classifier that sorts a test run into one of five verdicts:
OutcomeKind = Literal[
"RED_VALID_FAIL", "RED_COMPILE_FAIL", "GREEN_PASS", "GREEN_FAIL", "TEST_CRASH"
]The contract test checked that the result was one of them:
assert result.outcome in {
"RED_VALID_FAIL", "RED_COMPILE_FAIL",
"GREEN_PASS", "GREEN_FAIL", "TEST_CRASH",
}That assertion holds for any of the five, so it cannot notice when one is never produced. One never was. GREEN_FAIL, the verdict for tests failing in the GREEN phase, was unreachable. The classifier did not distinguish that case and the membership check stayed green regardless. The dead verdict shipped to main before a human caught it. The code did exactly what the spec said. The spec was imprecise and nothing in the loop could tell.
It is a trivial bug, and that is the point. Plenty of developers write assertions that weak. The difference is that when the same person writes the test and the code, the intent sits in their head, and the code usually ends up correct even when the assertion is loose. In the loop nobody holds the intent. A model writes the test to satisfy the spec, a model writes the code to satisfy the test, and the spec is the only place the intent ever lived. Miss it there and the weak spec goes through every gate to a review fast enough to miss it too.
The same gap kept appearing, so catching it became a standing rule: can the spec survive a literal-minded implementer? For the classifier, the fix is to pin each verdict by value instead of membership, and to require that every label the contract names can actually be produced:
assert outcome(failing, {"pytest": 1}, "GREEN") == "GREEN_FAIL"
assert outcome(failing, {"pytest": 1}, "RED") == "RED_VALID_FAIL"That works while the output is a fixed set of labels. Once it ranges over a whole input domain you cannot enumerate the cases, since there is no listing every shape of source a complexity rule might judge. There the check is a metamorphic property: an invariance the verdict has to hold across the whole family of inputs, such as a violation at any nesting depth still being caught, plus a check that the cases do not contradict each other. These techniques are decades old. Property-based and metamorphic testing predate the models. What is different is aiming them at the spec instead of the code.
No tool can make design validation correct. Whether a spec captures what you meant is a judgment that lives in your head, and sometimes it is not settled until you build. What a tool can do is make applying that judgment cheap. The orchestrator never made code correct, and the classifier that shipped wrong proves it. It made code cheap, and that was enough. Once writing code costs almost nothing, the spec is what is left to get wrong. Same bet, one level up.
Most of what feels like validation can be automated. A tool can find the gaps in a spec, generate the implementations that pass the tests while doing the wrong thing, and put the decision in front of a person in a form they can judge fast. The contract-adequacy check is the first piece already working. What I am testing now is how much of the rest is automatable, and how much is really judgment.
The orchestrator, chdr, is public with a fuller post-mortem: github.com/AlexTavor/chdr. Proof-Driven Development, the methodology that came out of it, has since been concluded; its post-mortem is here. Both are personal open-source research. There is nothing to buy.