How I let models make high-stakes decisions in production without ever trusting a model.
My work lives at the point where a machine's decision becomes a real one. Somewhere in a payment network, in real time, a system I built decides what a program will and will not cover, and that decision holds. The economics are unforgiving, and an error is never abstract: it lands on a family at a register, or on a program's books. Errors happen anyway. The work is making them rare, visible, and cheap to catch. Language models are extraordinary instruments for this kind of work and entirely unqualified to do it unsupervised. The architecture below is how both of those remain true at once.
Judgment runs in stages, ordered by cost and confidence. The cheapest, most certain logic goes first, and only the genuinely ambiguous cases, a few percent, ever reach a model at all. A classical screener sits between them, chosen from a field of seventeen candidates and validated so it performs on data it has never seen, not just the data it grew up near.
No model verdict is ever accepted on its own. A panel of three models judges each hard case, majority to pass, disagreement escalated to a person. Model errors correlate within a single model and far less across different ones, so a split is the most honest signal there is that a human should look. In the last run, more than ten thousand decisions, all but four were unanimous.
Nothing reaches production directly. Every change is rendered against the last trusted state, cleared through an evaluation suite that must stay green, and dry-run verified end to end before a human releases it, paced behind a safeguard I wrote after learning, once, exactly how a careless push behaves. Once was the tuition.
Audit the ground truth before the model. Ours plateaued no matter the architecture. The ceiling was not the model; a third of the training labels contradicted themselves. Repairing the data moved the same model from good to excellent. An evaluation is only ever as good as the truth it is measured against.
Fail closed, always. Output that cannot be read is not a retry in a decision that touches money. It is a no. A verifier you cannot understand must never be allowed to approve anything.
Suspect the instrument before the intelligence. One set of results looked so poor the natural conclusion was that the model could not do the work. A controlled comparison proved the model was fine and the harness around it was quietly breaking every measurement. The apparatus is part of the system, and it is often the part that is wrong.
I am rebuilding this discipline in a different field entirely, in a project called begriff: formal specifications in, machine-checkable proofs out, held to the same standard of evidence, juries, and human-held release. Different stakes, the same refusal to trust without proof.