A frontier model can write a plausible nixpkgs security patch today. That is not in question anymore. What is in question is what a maintainer does with a patch that merely looks right, since the model spends no more effort getting it right than getting it wrong.
trace has been in continuous development for just under six months: six months of trial and error, of failed builds and wasted compute, including real difficulty on Hugging Face finding a cost-effective GPU that would not run out of memory mid-training. It is an agent-owned nixpkgs fork, and the record below is what it found out, including the parts where it was wrong.
The pipeline
trace sets up a small team of agents, each with a name that also defines its job. Given a CVE, the Patcher drafts a fix. The Reviewer critiques that fix independently, with no view of the Patcher’s reasoning. The Reproducer writes a small test meant to trigger the vulnerability. Each agent is its own separate invocation, not one model switching hats: the Patcher ran on Opus, the Reviewer and Reproducer each ran their own separate Sonnet invocation. None of that is trusted on its own.
A Builder compiles the package twice in a hermetic Nix sandbox, once without the fix and once with it, and runs the reproducer against both. The unpatched build has to fail the way the advisory describes. The patched build has to survive. A Steward then decides whether the resulting evidence bundle gets signed, anchored to a Sigstore transparency log entry a maintainer can check without trusting trace either.
Eight roles sit in that pipeline. Three touch a model. The other five stay deterministic code that runs the model’s output rather than taking its word for what happened: eligibility, provenance, hashing, the build itself and the signing decision.
Why trace
trace did not start as trace. It started as Vulnpatch, a dashboard that pulled together CVE intelligence, triaged by what was actually being exploited, and gave a maintainer the context to act faster. The assumption underneath it was that the maintainer is the bottleneck: speed up the workflow and packages get patched sooner.
One piece of that workflow was cve.vulnpatch.dev, a CVE lookup built for an audience that is no longer only human. It serves a machine-readable dossier per CVE rather than a page meant to be read: canonical advisory data, source provenance and an availability state for an ID nobody has published yet, so an agent working a remediation job has a fact base to act on instead of a web page to scrape. That piece still runs. It is one of the sources trace’s own Scout role reads when it looks for new CVEs. Treating a CVE database as something an agent queries, not only something a human reads, held up. It was not the whole answer.
Vulnpatch was already built with frontier models in the loop, and the plan from there was to fine-tune a cheaper open model to do the remediation at scale once enough examples existed. A new frontier model’s release landed mid-build, and the plan no longer made sense. A dashboard to help a maintainer dispatch the work, or a cheaper model trained to attempt it, was solving the wrong problem next to a model that, set up correctly, could already attempt the patch itself.
What that release did not solve is whether a maintainer has any reason to
trust what the agent produced. Which package is actually affected. Which
upstream commit fixes it. What the new source hash is. Whether the package
still builds. Whether the tests still pass. Whether any of that is checkable
by someone who was not in the room when the model wrote it. The hard part was
never automating the patch. It is making an agent’s output something a
volunteer on a security team can verify without taking the agent’s word for
it. Nix is unusually suited to supplying that: reproducible builds and
passthru.tests give an agent a ground-truth verifier that most coding
agents, in most repositories, never get.
The measurement
The sharpest test trace has run starts with real nixpkgs security backports where the fix is already known. A frontier model sees only the vulnerability description and the pre-fix source, never the upstream patch itself, and its blind fix is graded against what the upstream project actually shipped.
| Backports screened | Matched or closed the fix | Blind accuracy |
|---|---|---|
| 41 | 16/41 | 39% |
| Outcome | Count |
|---|---|
| Exact match | 5 |
| Equivalent fix | 11 |
| Partially correct | 11 |
| Wrong | 14 |
41 real nixpkgs security backports, fix withheld from the model: 5 exact, 11 equivalent, 11 partially correct, 14 wrong. 2 of the 43 eligible backports could not be screened this way.
Two of every five is real signal. These fixes come from reading the vulnerability itself, not from having seen the answer. A model wrong more than a third of the time cannot run unsupervised, and that is the argument for the rest of the pipeline: a Reviewer pass it never shares context with, a build that actually runs and a signed gate before anything ships.
As of this week, 86 real bundles carry a build record showing the patched package compiled clean. 23 routed to auto-merge. 63 to human review. Most also carry a reproducer that discriminates, crashing unpatched and surviving patched. The rest are reviewer-attested: approved on review alone, with no fixture built against them. The site states which is which rather than letting one number stand in for both kinds of evidence.
What the evidence caught
The more useful finding is not a number. It is what happened when the pipeline’s own evidence turned out to be wrong, and what followed from it.
Twice, a Reproducer’s own log described a result the build never produced: a fabricated crash signal on a crafted font file and a fabricated parser failure that the real build showed never happened at all. A model’s account of its own success is a claim, not evidence. In both cases the bounded build probe trace runs before trusting any reproducer caught the gap before it reached a signed bundle.
A harder case: on a held-out canary, a fine-tuned Reviewer correctly rejected a flawed patch, but its written explanation for the reject was false on its face, citing a bounds check that was actually present in the code it was reviewing: a correct verdict with a fabricated reason behind it. This was only caught because the case had a known right answer. On an ordinary bundle the same failure mode ships a correct-looking rejection for a fabricated reason, and nothing downstream would know the difference. A citation a model makes has to be checked against the artifact it claims to quote, every time.
The signing chain had its own gap, found by auditing the part least suspected. Every signed bundle carries a cosign signature over a fixed manifest of its own file hashes, so a reviewer can check the bytes in front of them against what was actually signed. A sweep this month compared every signed manifest against the bytes currently on disk. Twenty-seven of sixty-three reconciled cleanly. The rest had drifted, mostly in bookkeeping files that never shipped in a pull request, but five carried a real post-sign change to the patch or the reproducer itself. The fix came in this order: write down the finding before any file moves, restore what git history could recover rather than silently re-sign it as if nothing happened and leave the rest flagged until a fresh build re-earns the signature.
None of this is the finding trace went looking for. It is the finding the verification layer exists to produce, and a project whose premise is that proof matters more than a plausible claim does not get to leave out the cases where its own proof briefly stopped being proof.
What this is not
trace does not claim frontier parity for cheaper open-weight models. A fine-tuned fourteen-billion-parameter Reviewer was trialed as a lower-cost substitute, came nowhere close to the frontier model it was meant to replace, and the result was published as it stood rather than measured against a smaller, friendlier benchmark.
trace also does not merge anything into real nixpkgs. The fork’s pull requests are accuracy-comparison artifacts against the upstream maintainer’s own fix, not merge candidates waiting on review.
Our part
Ad Astra Computing is publishing trace as a research artifact, not a product. Findings, cost breakdowns and the full bundle corpus are at trace.adastra.computer. The nixpkgs fork carrying the evidence bundles is public: github.com/Ad-Astra-Computing/nixpkgs. The talk behind this post, Nix Knows When the Agent Is Wrong, was presented at nix.vegas in August; the recording is on trace’s own site.
The posture holds the way it held through every finding above: publish what exists, name what is missing and let the record carry the result and the gap that produced it in the same breath.
We consider this the first iteration, intended to inform future work on agent remediation.