The morning I rebuilt the CrowdStrike file
On July 19, 2024, a single CrowdStrike Rapid Response Content channel file crashed millions of Windows machines in under 90 minutes, and the first thing I did when I sat down to build this demo was rebuild that exact file. The published root cause (CrowdStrike's Root Cause Analysis, August 2024) was not a hack and not a bad model. It was a schema mismatch. The cloud validator approved a 21-field update while the kernel interpreter still expected 20, which produced an out-of-bounds read and an instant blue screen. Because the crash landed so early in boot, the failing agent could never re-initialize to receive a rollback command, so recovery meant hand-fixing machines one at a time in Safe Mode.
So I encoded that signature into the demo as a fixture and pointed the checker at it. The fleet is synthetic, an invented "Acme Financial: Global Endpoint Fleet" of 8,500 endpoints, and the vendor "SentinelEdge" and its "Falcon-class EDR" agent are fictional stand-ins, not CrowdStrike or its live systems. The update, C-00000291, arrives declared at schema 20 to 21 and pushed to 100 percent of the fleet in one wave. Four deterministic checks fire together: the field-count mismatch, a simulated sandbox that blue-screens on 5 of 6 profiles, a dead-agent rollback loop that comes back true, and a blast radius of 100 percent against a 5 percent canary policy. The disposition reads Block Rollout, blocked before any production endpoint rebooted. The estimated prevented downtime shows $5,000,000, which the demo computes on screen as affected share times a $5M-per-hour rate times a one-hour recovery floor, a conservative model on synthetic data rather than a number anyone banked.
The rebuilt CrowdStrike signature, C-00000291, blocked before any endpoint rebooted. All four deterministic proofs fire, and the $5,000,000 is the demo's estimated prevented-downtime model on a synthetic fleet, shown with its formula.
What I couldn't find when I went looking for the layer that should have caught it
When I went looking for the layer that should have caught July 19 before it reached production, I couldn't find one. The vendor's own validator is self-policing, which is exactly what failed. SBOM and SCA tools watch open-source dependencies, not a vendor's proprietary channel file. Change-advisory boards wave vendor updates through on the logic that we trust the vendor. Nobody reads the actual payload on its way in.
What made that land for me was counting the agents. The fleet I modeled runs 8 privileged agents, 5 of them at ring 0 in the kernel, from vendors it does not control, and any one of them can push a channel file straight into the most privileged code on the machine. That is the exposure the outage revealed, and it is structural. So I built Kestrel to sit in that gap: an independent control plane between a vendor's update pipeline and a production fleet, vendor-neutral, above every agent, and without kernel access of its own. It is not an EDR and it does not compete with the agents it governs. It reads the update the vendor is about to ship and decides what is allowed through. For a finance or healthcare fleet, where an hour of critical-system downtime runs into the millions (industry context, in the range of $2M to $5M an hour, not a Kestrel measurement), the layer that reads that payload first is the one nobody had.
The check that turned out to be a field count
I expected the hard part of this to be the AI, and the check that actually caught the CrowdStrike replay was arithmetic. A schema-compatibility diff compares the update's declared field count against what the deployed kernel interpreter expects, and 21 where the interpreter expects 20 is a decidable fact, catchable before any endpoint reboots, by anyone willing to count. The demo does run an advisory crew, a Normalizer, a Sandbox-Interpreter, and two opposed Critics (one arguing the update is safe to ship, one arguing it will crash), built on Pydantic AI, provider-neutral, defaulting to claude-opus-4-8 and falling back to a deterministic advisory path with no API key at all. But the verdict is set by a plain-Python verifier and policy gate that sit outside the agent framework, re-derivable by a regulator. Agents advise, and the code decides. An advisory agent leaning allow cannot clear a critical deterministic finding.
I made the other three checks corroborate rather than echo. The sandbox is a deterministic per-profile outcome model, not real Windows VMs, and it derives its blue-screen result from a driver-compatibility signal independent of the schema check, so the 5-of-6 result is a second witness rather than a copy of the first (the legacy Server 2012 profile is flagged and excluded, never assumed safe). The dead-agent check returned true because the failing Falcon-class agent is itself the rollback receiver, the same trap that forced Safe-Mode recovery in the real event. The blast-radius check caught a 100 percent first wave with no canary plan declared. None of the three leans on the model, so a critic in a good mood cannot talk any of them down.
The benign update I made sure to keep
I was more worried about the update I had to let through than the one I had to block. A gate that stops your good updates is an outage of its own, so I built the benign case on purpose. RRC-7741, from the same SentinelEdge vendor, is a Rapid Response Content update declared at schema 20 to 20 with a staged 1.2 percent canary plan. The schema matches, 5 of 6 profiles pass 5 reboot cycles, the dead-agent loop is false, and the 1.2 percent first wave is within policy, so it is approved and released to a canary ring of 102 endpoints in seconds. Green, fast, and boring, which is why I keep it in the demo.
The same vendor's benign update, RRC-7741, approved to a 1.2 percent canary in seconds. Schema matches, 5 of 6 profiles pass, blast radius is within policy, and the evidence record and 7/7 trace still export.
Across the demo's 12-item labelled fixture set, 6 of them benign, the gate returns the expected decision 12 out of 12 times with 0 false-blocks on the 6 benign updates, and it does so deterministically, the same input yielding the same decision every run. One of the 12 is an honest ABSTAIN: an encrypted payload the gate cannot parse gets routed to a human rather than waved through, counted as correct because it never green-lights what it can't prove. On screen the scoreboard reads 12/12, 0/6, and $13.3M of estimated prevented downtime across the whole set, the demo's own model on synthetic fixtures, not money a customer saved.
The value scoreboard on the demo's 12-item labelled fixture set (6 benign): 12/12 expected decisions, 0/6 false-blocks, and $13.3M of estimated prevented downtime, computed deterministically, not an open-world guarantee.
The receipt I wanted a regulator to be able to re-run
I wanted every decision to leave a record a board and a regulator could re-run without me in the room. One click exports an immutable HTML record and a Signed JSON file carrying a sha256 content hash (a local SHA-256, not enterprise PKI), the verdict, the deterministic proofs, the per-profile sandbox results, the advisory agents' verdicts with their model id, the policy rules that fired, and a per-step evaluation trace where each step keeps its own latency, down to "Normalize signed vendor manifest, completed in 184 ms," 7 of 7 events complete. It is framed against the EU Cyber Resilience Act's short-window reporting, the SEC's four-business-day disclosure rule, and Delta v. CrowdStrike (2025, Fulton County). It is designed to align with those filing needs, not certified against them, and it is not legal advice. In the demo the vendor feed and the ServiceNow and Jira hooks are simulated integrations, so the record stays honest about which steps were real.
The signed evidence record for the blocked update: an HTML record plus Signed JSON with a sha256 hash, the deterministic proofs, and a 7/7 evaluation trace. A local hash, framed to align with EU CRA, SEC, and Delta-precedent filing needs, not a certification.
If you want to see how it works, the full breakdown is at the software update integrity demo. The uncomfortable part was never that one vendor shipped a bad file in 2024. It is that the fleet I modeled trusts 8 privileged agents to count their own fields correctly, forever, and until I built this there was no arithmetic sitting in front of that pipeline to check them. I would rather keep the receipt.