When Mythos Owns The Loop: Self-Verifying Vulnerability Research

Summary

A couple of months ago I shared details of the Netskope security team’s efforts to teach a frontier AI model (OpenAI 5.5) to hunt memory corruption bugs inside a scaffold that we built and controlled. Encouraged by our results, we have been experimenting again, and this time we handed Anthropic’s Claude Mythos preview a much harder job. We asked it to own the loop. It had to decide what to look at, prove the bug against a real system, have a second copy of itself verify that proof on clean machines, and then explain how the pieces fit together.

We ran Mythos across three tracks of our own offensive security work, and every target was software that Netskope builds and ships. Between the three, the model produced 9 kernel memory corruption findings behind a strict reachability bar, 17 previously unreported bugs in the open source code our products are built from, and four confirmed n-day fixes reconstructed from a binary diff. It also wrote a full privilege escalation chain analysis that we never asked for. 

This post is about the harnesses and the mindset rather than the individual offsets. The most useful lesson is that the model no longer just runs the loop. In several places it ran ahead of the loop, and we rewrote our scaffolds so they would stop being the bottleneck.

What changed since we did this with 5.5?

When we last wrote about pointing a frontier model at memory corruption research, the finding was simple. The model becomes useful the moment it can test its own hypotheses against a real system. The scaffold’s job back then was to keep every claim honest by forcing it through a verifier, whether that verifier was a compiler, a live target, a debugger, or a crash dump.

This time we asked a different question. The question was not whether the model could find a crash if we built the loop around it. The question was whether the model could own the loop by itself. Could it choose the target, prove the bug, have an independent copy of itself, confirm the proof on freshly reverted machines, and then compose the individual findings into something a defender can act on? The answer was yes, and it is worth saying plainly. This is a real step up from the last generation. The model reasons across a problem the way a senior researcher does, and it closes the distance between “this looks wrong” and “here is the crash, three times, on a clean revert” without our help.

It did not just keep pace with the scaffolds we had built for earlier models. In several places it moved faster than we could feed it. The most productive change over the whole engagement was not us tuning the model. It was the model’s own lab notes tuning our scaffold.

Why we test our own products against frontier AI models

Every finding in this post came from testing software that we build and ship, and that is deliberate. Customers put Netskope in the most sensitive part of their network, so the security of our own endpoint is utmost important for us. We hold it to the same standard that we ask of everything else we inspect. The most useful adversary simulation is the one that starts where a real attacker starts, with an installer and no source code, and ends with a report our engineering teams can act upon before a release ever reaches a customer. When a model can run that simulation from end to end, the output turns into a release gate.

Pointing the newest offensive research at our own products, ahead of anyone else, is the whole idea. The same capability that helped us find these bugs will help an attacker find them too, so we would rather find them first, fix and hold anything that belongs to an upstream project under coordinated disclosure. That is what staying ahead actually looks like, and it is the bar we intend to keep raising as these models get stronger.

Three tracks, one rule

We pointed Mythos at three different shapes of the same problem, and the same rule held underneath all three. A claim only becomes a finding when something independent can reproduce it, working from nothing but the evidence a stranger would need to reproduce it too, such as a crashing input or a small proof of concept. In most runs, that independent check was a second copy of Mythos, running in its own sandbox with no knowledge of how the first copy got there.

On the first track we gave the model source code, a live kernel debugger, and two of our shipping Windows kernel drivers, and we set the bar at reachability rather than suspicion. Spotting an unsafe pattern in the source counted for nothing on its own. A finding was real only when the model had cleared every runtime precondition between the entry point and the unsafe sink, and had crashed the driver three times from a clean snapshot. Nine kernel memory corruption findings cleared that bar, including one that an ordinary user can reach to write attacker controlled bytes into the kernel pool.

1. On the second track we turned the model on the open source C and C++ that our product uses. Here a finding had to be an input that crashes under AddressSanitizer, confirmed by a second copy of the model in a clean sandbox against the current upstream code. Anything the model could only describe in prose did not count. 17 previously unreported bugs cleared that bar.
2. On the third track we handed the model two installed versions of our own Windows endpoint client, an older release and the latest one, with no source and no symbols. We asked it to work backwards from the binary diff, decide which changes were security fixes, crash the older release to prove each one, and confirm the latest release is safe against the same input. It found four confirmed n-days, but the result that mattered more was the pipeline that produced them.

Track 1: Reaching the unsafe sinks

Most kernel memory corruption bugs in a mature product are not hard to read, they are hard to reach. The unbounded copy sits right there in the source. What stands between an unprivileged process and that unbounded copy might be a watchdog that rewrites the configuration you just poisoned; a single connection port already held by the real service; a check on the caller’s process name; a feature flag that ships turned off; or an allocator rounding rule that quietly absorbs your first attempt.

So the harness on this first track was built around one rule. A hypothesis from the source is not a finding until the model has listed every precondition between the entry point and the sink, satisfied each one against the live product, and reproduced the crash three times from a clean snapshot. The model generated more than 40 distinct memory corruption hypotheses in a few hours of review, and almost none of them survived contact with the running system unchanged. The real work was everything that happened between “this loop has no boundary checking” and “here is the same kernel crash, three times, on fresh reverts.”

To manage the complete workflow, the model kept a durable ledger across sessions. It held a map of every entry point a user could reach, a log of confirmed bugs with their full precondition chains, and a cache of every hypothesis that died, tagged with the reason it died. The reasons mattered, because each one feeds back differently. A hypothesis parked because a precondition cannot be met at the current privilege level can come back to life later. That is exactly what happened when we widened the scope from a low privilege user to a local administrator, and three parked hypotheses returned as live leads.

One confirmed finding shows the shape of the work. The sink was a configuration parser that copies a registry value into an exactly sized pool allocation, then walks it with a string length loop that has no bound. A textbook bug, found on the first pass. Reaching it took five preconditions and two failed attempts. Several independent runtime defenses stood between an ordinary process and that code, and each one had to be satisfied, in the right order, before the parser would run at all. Even then, a correctly poisoned value did not crash, so the model went to work on why. It broke into the kernel, dumped the allocation, and read the page table past the end of it. The operating system had appended a terminator and the allocator had rounded the buffer up to a full page, so the first out of bounds read was landing in harmless slack. The model resized the payload to fill an exact page, and the next run faulted on the first read past the buffer. Three clean reverts, three identical crashes.

None of those five steps is hard on its own. What we had not seen a model do before was hold all five as open obligations across many context windows, discover the fourth by instrumenting its own failed attempt in the kernel debugger, and fix the fifth by reasoning about allocator geometry rather than by fuzzing sizes at random. Across the two drivers the harness produced nine findings that cleared the reachability bar, including that one low privilege pool overflow. It also left a cache that closed at 43 entries on the first driver alone. Each entry tells the next reviewer which plausible looking pattern is already known to be safe, and why.

Track 2: Hunting the code you did not write

Our endpoint client leverages open source C and C++ libraries such as TCP/IP stack, JSON parsers, JWT library, template engine and database engine, and none of whose maintainers have ever seen our build. Even though we have guardrails built, if any one of them mishandles a length field, our product might have the risk of mishandling it too. The bug is upstream, but we might fall in the blast radius as well.

Large models are fluent at writing vulnerability reports, and that fluency is exactly the problem. A model that has read ten thousand CVE reports can produce a flawless, well argued analysis of a bug that does not exist. And so we designed this harness so that it will not accept a written description. Instead it requires a small package that anyone can run, a build script, a C file, and a blog of input bytes file for the parsers. The script clones the upstream project at its current head, compiles the C file against the unmodified upstream source with AddressSanitizer, and runs it. The finding is real only if the sanitizer crashes with its faulting frame inside the upstream project’s own source. Whatever the model wrote about the bug counts for nothing next to that.

The verification boundary is the whole point. The find agent works in an isolated sandbox with our vendored source and a compiler. The grade agent is a fresh copy of Mythos in a second sandbox that has never seen the first one. It receives only the build script and the input bytes. It clones the current upstream code itself, rebuilds, runs, and verifies it.  If the find agent talked itself into something the input does not actually show, the grade agent has no way to inherit that mistake.

Before a bug is graded, it must survive three independent ‘reader’ checks, where each AI instance is tasked with disproving the finding from a unique perspective.

1. The first reader acts as a skeptic, assuming a security guard was missed and re-tracing the entire path from the public entry point to the sink. 
2. The second reader assumes the logic is flawed, independently recalculating the overflow from the source rather than relying on previous explanations. 
3. The third reader assumes the vulnerability is already public, cross-referencing it against CVE records, the latest upstream code, and open issue trackers. 

This final check was highly effective, flagging nine ‘new’ bugs as already known and patched. This allowed us to simply update our versioning which is a far more constructive outcome than unnecessarily reporting old bugs to maintainers.

The two sharpest results came from a single instruction, to read the last security fix and check whether it is actually complete. Following that idea, the model found a well known parser whose old recursion fix guarded one code path but not its twin, and a widely deployed network stack that enforces a length limit everywhere except the one place the specification demanded it. We are keeping the specifics out of this post while the fixes land. What matters here is the shape of the result. Of the seventeen findings, one is critical, three are high, nine are medium, and four are low. 14 are going to open source maintainers under a 90 day coordinated timeline, three go privately to a commercial vendor, and none of it publishes until the maintainers are ready.

Track 3: Reading the fix to find the bug

Every security fix leaves a trace in the next build. When a new release quietly adds a bounds check or a validator, it is often telling you (in effect) where the old release was weak. Patch diffing is the practice of comparing two builds and reading those traces to work out what was fixed and why. Here we started from nothing more than two installed versions of our own endpoint client (an older build and the latest build), with no source and no symbols. We asked the model to decide from the diff alone which changes were security fixes, to prove each one by crashing the older build, to prove the latest build is safe against the same input, and to tell us how the bugs fit together.

The scaffold enforced one rule with two halves. A patch diff finding is real only if the same input crashes the older build and runs cleanly on the latest one. If the reproducer crashes both, the model found a real bug, but not the one the diff pointed at. If it crashes neither, the model misread the diff. To hold that line, every candidate went through a differential grade. A fresh copy of the model, working on freshly reverted machines of both builds, was handed only the reproducer (the short program whose source triggers the bug), and nothing else. It rebuilt that program, ran it three times on the older build and twice on the latest one, and reported five pass or fail results. That is a stricter check than a researcher confirming their own work. It is closer to a second researcher who reproduces your result in a clean lab from your proof alone, then confirms the vendor’s fix closes it.

The first stage read the decompiled diff across multiple modules, ranked each changed function by how much it smelled like a security fix rather than a refactor, and checked that ranking against a live map of what an attacker could actually reach on the older build. Roughly 1,700 changed functions came down to 12 worth chasing. The ranking was specific enough to act on. Instead of noting that a function had changed, it explained why the change looked like a fix. In one case it explained why a new length check mattered. The old build worked out how much memory to reserve by adding a few bytes to a length the attacker controls. Make that length large enough and the addition overflows, wrapping around to a tiny number so that the old build reserves almost nothing, then copies the full oversized payload into it. That is a classic heap overflow, and the check the new build added is exactly what stops it. That kind of triage, holding multiple binaries and the reachable attack surface in mind at once, is exactly what a senior reviewer does by hand, and it is where most of the leverage sits.

The hardest confirmed finding took three attempts across three days and roughly 3,800 model turns in the final session, because the diff that pointed to it was easy to miss. In the latest build a kernel cache reader had quietly gained an allocator call and a debug string. In the older build, the same reader handed a cached pointer back to its caller and cleared the cache slot while holding only a shared read lock. A shared read lock is meant to let many readers run at once, precisely because reading is not supposed to change anything. Clearing a shared slot is a write, and doing it under a read lock means two readers can run at the same moment, walk away with the same pointer, and each free it later. That is a classic double free, and here the size and contents of the freed block are under the attacker’s control. 

Reaching that code meant clearing six independent gates, including driver state bits set by commands on a different port, and a reply protocol the model had to reverse from disassembly before its threads would stop blocking in the kernel. The first two attempts each solved most of the chain and stalled on one gate, and each left a precise note of what it had tried and what to try next. The third attempt read those notes, found the command that flipped the last bit, and landed the crash.

The double free hit five times out of five across two machines. On the latest build the same reproducer set the same flags and launched the same racing threads, and the counter that measured how many threads reached the vulnerable reader stayed at zero. The path was gone. That is what a fix looks like from the outside.

12 candidates became four confirmed n-days. Then the composition stage read all four graded reports together and produced something we had not asked for. It showed that three of the four combine into a feasible path from an ordinary user up to SYSTEM on a default secure Windows 11 host, and it named the one fix that can break the chain (and the one that does not). That is a backport priority answer, produced by the same system that found the bugs. 

None of this was designed in advance. 22 separate scaffold fixes came from simply running the pipeline, from decompiler memory limits to the order in which snapshots have to recover. The scaffold and the model debugged each other.

Building the harness yourself

The models get the headlines, but the reusable asset is the harness. If you want to try these AI-orchestrated self-verifying vulnerability research against your own software, here is the shape that worked for us. 

• Pick a target that you own and can crash freely, and isolate it before the model is anywhere near it. Stand up a target machine and a separate debugger machine, keep all of the orchestration outside the blast radius, and let a crash be just another thing the loop can observe rather than an event that tears the session down.
• Wire up at least one verifier before you write a single prompt. The verifier is the whole point. It can be a compiler, a sanitizer, a live target, a debugger, or a difference across two releases. Everything the model believes has to compress down to an artifact that the verifier can accept or reject.
• Make the model’s output something that can be proven false. A crashing input under AddressSanitizer is either real or it is not. A reproducer that crashes an old build and not a new one points at a specific fix or it does not. A paragraph does neither. Every later decision about grading, severity, and disclosure gets simpler once the unit of work is a file that a compiler accepts rather than a claim.
• Grade in a sandbox that never saw the hunt, and if you can, use a second copy of the model to do it. A find agent that grades its own work will talk itself into the answer it wants. Hand a fresh copy only to the reproducer, with none of the transcript and none of the reasoning, put it on clean machines, and let it reach its own verdict. That is the difference between the model checking its own work and the finding standing on its own.
• Treat preconditions and dead ends as first class results. The findings that landed were the ones where every runtime gate was confirmed open. The ones that did not land were retired with the first closed gate named and cached. That cache of dead ends, (whether stale commands, missing objects, known CVEs, or code that never ships) is what stops the next run from walking the same dead ends again. For open source work, a check for what is already public belongs right in the verifier.
• Let the pipeline improve the scaffold. Budget for the model’s lab notes flowing back into your harness, not just your prompts flowing into the model. Some of our best infrastructure fixes and reusable attack patterns came straight out of the model’s own progress logs.

What this means

For the Netskope team, the real value is not the individual bugs. Several were already fixed, and the rest are in remediation or coordinated disclosure. The value is a repeatable, adversary shaped check that runs the newest offensive research against our own releases before they ship, so the code underneath our customer’s security is held to a standard most attackers cannot match. The open source we depend upon is now audited with the same rigor as the code we write ourselves, and a single model can find a bug, prove it, verify the proof independently, and tell us which fix to ship first.

This is the kind of work we believe a security vendor should be doing with frontier models, and doing early. The capability is real, and it is available to both sides, so the teams that come out ahead will be the ones that put it to work first, point it at their own products, ground it in a verifier, and stay honest about what did not work. That is the standard we hold our own products to, and it is the work we intend to keep doing in the open.