The CrowdStrike crash came down to a field count: 21 where the kernel expected 20. No independent layer was checking.

On July 19, 2024, a single vendor update crashed millions of Windows machines in under 90 minutes, and the cause was a number. A CrowdStrike "Rapid Response Content" channel file declared 21 fields where the deployed kernel interpreter expected 20. The extra field produced an out-of-bounds read, an instant blue screen, and because the crash landed so early in boot, the crashing agent could never come back up to receive a rollback command. Recovery meant walking to each machine and fixing it by hand in Safe Mode.

I read CrowdStrike's own root-cause analysis, published that August, more than once before the thing that bothered me surfaced. This was not a hack. It was not a bad model. It was a decidable arithmetic fact, 21 against 20, sitting in a payload that no independent layer ever checked before it reached production. The vendor's validator approved it. The enterprises that went dark did not own that validator. They owned the consequences.

I spent the last stretch building a demo around that gap, a console I named Kestrel that sits between a software vendor and a production fleet and decides, in code, what the vendor is allowed to ship. You can see how it works at veriprajna.com/demos/software-update-integrity. What surprised me while building it was where the fix turned out to live. I went in certain I would need a smarter model, and a few lines of plain Python caught the crash first.

I rebuilt the crash, then let code decide it

I rebuilt the July-19 failure signature as a fixture and pointed my own system at it, half expecting to be underwhelmed by my own replay. The package is C-00000291, a Rapid Response Content channel file from a fictional vendor I called SentinelEdge, pushed to a synthetic 8,500-endpoint fleet I named Acme Financial. None of those are real companies. The failure signature is the real one: schema declared 20 growing to 21, pushed to 100% of the fleet in a single wave, no canary plan.

The gate fires on four checks at once, and each is plain arithmetic or a plain lookup, never a judgment call. The schema diff sees 21 fields where the interpreter expects 20 and flags the out-of-bounds read. A simulated sandbox, which is a deterministic per-profile outcome model and not a farm of real Windows VMs, boot-loops 5 of the 6 fleet profiles across reboot cycles, and it derives that from a driver-compatibility signal independent of the schema check, so the two findings corroborate each other rather than echo. The dead-agent detector marks the rollback loop true, because the crashing agent is itself the thing that would receive the rollback, and it is dead before boot. The blast radius is 100% against a 5% canary policy. Verdict: BLOCK. On screen it reads, blocked before any production endpoint rebooted.

Estimated prevented downtime on that one update reads $5,000,000, and I want to be exact about what that figure is. It is the demo's own model, the affected share times a $5M-per-hour input times a one-hour recovery floor, with the formula printed on screen. It is not money a customer saved. The real July-19 recovery took days, not an hour, so the floor is deliberately conservative.

The green case scared me more than the red one

I was more nervous about the green case than the red one, because a governance layer that blocks the dangerous update and also strangles the safe one is just an outage you scheduled for yourself. The same fictional vendor pushes RRC-7741, a benign detection-signature update, declared schema 20 to 20, with a staged 1.2% canary plan. The crew runs, the schema matches, 5 of 6 profiles clear their reboot cycles, the dead-agent loop is false, the blast radius sits inside policy. Verdict: APPROVE ROLLOUT, released to a canary ring of 102 endpoints. Green, fast, boring.

Across the six benign updates in the set, the gate produced zero false blocks. I say that with the denominator attached, because six is six, and I will not let it round up into a promise about your fleet. The value of the ALLOW case is narrower and more important than a percentage. A gate is credible only if it is invisible on normal traffic and immovable on the one turn that could take your fleet down.

Why I pulled the verdict out of the model

I started this build assuming the hard part was the reasoning, and that a sharper model or a cleverer critic would be the thing that caught the bad update. I was wrong in a way it took me a while to admit. There is an LLM crew inside Kestrel: a normalizer, a sandbox-interpreter, and two opposed critics, one arguing the update is safe to ship and one arguing it will crash. The adversarial pair earns its place, because it red-teams the verdict from both directions before anything is decided. But not one of those agents sets the verdict.

The verdict is set by two plain Python files, verifier.py and gate.py, that live outside the agent framework entirely. The crew runs on Pydantic AI with a default model of claude-opus-4-8, and the whole thing also runs offline with no API key through a deterministic advisory fallback. In every one of those modes the gate is identical and returns the same decision, because the decision is arithmetic, not inference. Agents advise, code decides. An advisory agent leaning "allow" cannot clear a critical deterministic finding, and that is not a matter of taste.

A layer built to check the vendor cannot take the vendor's word for safety. It cannot take its own model's word either.

That sentence is the reason the architecture looks the way it does. Trust in a product whose only job is to govern what a vendor ships must never route back through a component that can be talked into a yes.

What I would hand an auditor

I kept the EU Cyber Resilience Act open on a second monitor while I built the evidence record, because that record is the artifact I would actually have to defend. Every decision exports an immutable HTML file and a signed JSON file carrying a SHA-256 content hash, the verdict, the deterministic proofs, the per-profile sandbox results, the advisory agents' verdicts with their model id, the policy rules that fired, and a per-step evaluation trace where each step carries its own latency.

The trace is the piece I underrated until I clicked into a single step. One event reads "Normalize signed vendor manifest, completed in 184 ms," and it is retained with the decision output for audit review. Every step is re-derivable. A regulator does not have to trust my dashboard. They can re-run the arithmetic and get the same answer.

I am careful about what the signing is and is not. It is a local SHA-256, not enterprise PKI. The vendor update feed and the ITSM tickets behind it are fixture stubs, not live connectors. The record is designed to align with filing needs, the CRA's short-window incident reporting, the SEC's four-business-day disclosure of a material cybersecurity incident, the vendor-liability questions raised by Delta v. CrowdStrike in Fulton County in 2025. Designed to align with. It certifies no one, it is not legal advice, and anyone selling you an audit log that makes you compliant is selling you something.

There is one more decision I am proud of, and it is a refusal. Fixture XX-0000 is an encrypted proprietary content blob the gate cannot parse, so it does not guess. It returns ABSTAIN and routes to a human, because a gate that green-lights what it cannot read is worse than no gate at all. Legacy hosts the sandbox cannot model get flagged and excluded, never assumed safe. The vocabulary is four words, ALLOW, HOLD, BLOCK, ABSTAIN, and the last one is the one I would defend hardest.

What 12 out of 12 is allowed to mean

I have to slow down here, because this is exactly where a founder starts rounding up, and I named the company Veriprajna, true wisdom, so the rounding up is off the table. Across a fixed, labelled set of twelve updates, the gate returns the correct decision on all twelve. Six of them are benign and it blocks none of them. One is the honest ABSTAIN. The scoreboard reads 12/12 verified, 0/6 false blocks, and $13.3M in estimated prevented downtime across the set, of which $5M is the single CrowdStrike-class block.

Now the part I refuse to shorten. Those are results on twelve labelled items, not a promise about the next update that lands in your fleet. Six benign items is six. This is not "blocks 100% of bad updates," it never will be, and if you catch me writing that sentence you should stop reading me. The number I do stand behind is a different kind. Same input, same decision, every run, because the verdict carries no model temperature. Run the fixture set again tomorrow and it returns byte-identical, which is what lets a deterministic layer be audited in a way a probabilistic one cannot.

The question I am left with

What stays with me from this build is how ordinary the failure was. Twenty-one fields where twenty were expected. A number any independent verifier could have caught by arithmetic before a single machine rebooted, if any independent verifier had been standing between the vendor and the fleet. There wasn't one. There still mostly isn't.

Every enterprise runs eight to twelve kernel-privileged agents from vendors it does not control, and each of them can push a file straight into ring 0. SBOM tools watch open-source dependencies. Identity watches access. Nobody reads the vendor's proprietary update on the way in and proves it safe. Kestrel is not an EDR and never touches the kernel. It sits above those agents and governs what they are allowed to ship. That is the layer I tried to build, and the full breakdown is at veriprajna.com/demos/software-update-integrity.

And if you would rather watch it than read me describe it, here is the whole thing running end to end.

So here is what I now ask about every fleet I see. When the next vendor update arrives, what stands between that file and production, and can it show its work? If the answer is a change-advisory board that trusts the vendor, then the arithmetic that took down millions of machines is still running unchecked. It will not announce itself. It will look exactly like every update that came before it, right up until the reboot.