Where AI security tools fall short, and what deterministic build evidence covers

The argument, stated narrowly

This is not a case against AI in security. Models are good at reading code, explaining a finding and suggesting a fix, and teams that use them well move faster. The argument is narrower: for records that have to hold up later, a probabilistic model is the wrong primary source. That covers the SBOM you hand a regulator, the provenance of the artifact you shipped, and the answer to "were we affected?"

Three properties decide whether a record holds up. Each section below takes one, with examples and sources.

1. Reproducibility: ask twice, get the same answer

A security record is only useful if someone else can get the same answer from the same input. Language models do not promise that, and the vendors say so.

• Anthropic's documentation: "Even with temperature set to 0, the results will not be fully deterministic and identical inputs may produce different outputs across API calls."
• GitHub on Copilot Autofix: "Even with the same alert and code, it might fail to produce a viable suggestion, or the suggestion might vary across attempts."
• Researchers who built the SecLLMHolmes framework tested eight LLMs on 228 code scenarios for IEEE S&P 2024. They report that the models "provide non-deterministic responses," and that "by merely changing function or variable names, or by the addition of library functions in the source code," PaLM2 and GPT-4 gave incorrect answers in 26% and 17% of cases.

Example. Two engineers ask the same assistant whether a service is exposed to a new advisory. One gets "no," one gets "possibly, check the transitive tree." Neither answer is a record, and neither can be reproduced for an auditor six months later.

A recorded build does not have this problem, because nothing is being inferred. CRACI observes the packages a job pulls from package sources while it runs, and the SBOM is a record of that observation. Provenance is signed with Ed25519 over SHA-512, so the link between an artifact and its build can be checked later by anyone with the signature.

2. Auditability: show how you know

Audits turn on the question "how do you know?" A model can give a fluent answer, but its explanation is not evidence. The SecLLMHolmes evaluation found that LLMs give "incorrect and unfaithful reasoning" about the vulnerabilities they assess, so a correct verdict can come with an explanation that does not hold.

The standards are moving toward records that explain their own origin. The 2026 SBOM minimum elements, published by CISA with the NSA, the FBI and international partners, add an SBOM Tool Name, an SBOM Tool Version and an SBOM Generation Context that says whether the SBOM was produced "before build," during the "build," or "after build." The same document warns that "organizations may consider errors, whether stemming from SBOM author practices or selection of inadequate tools, in organizational risk management decisions."

Example. A customer questionnaire asks how your SBOM was produced. "An assistant read the repository and wrote it" gives a generation context, but not one a reviewer can check. "It was recorded inside the CI runner during the release build, and the job's completeness state is Complete" is something the reviewer can verify against the build.

3. Completeness: you cannot report what you never saw

A model reasons over what it is shown. If the files it reads do not describe the software, the answer will not either, and nothing in the output says so. CISA's 2026 guidance sets the bar at the recipient's decision: they "should be able to conclude that a newly reported vulnerability does not affect them if the SBOM does not list the component." That only works if absence from the list means absence from the build.

Example. The anthropics/claude-code repository at v2.1.259 has no dependency manifest or lockfile of any kind. The published npm package declares zero runtime dependencies and wraps a prebuilt native binary. Any tool that reads the repository, AI or otherwise, has almost nothing to list, while the program that runs is a compiled binary.

Example. AI coding tools can also introduce the gap themselves. GitHub warns that Copilot Autofix "may suggest fabricated dependencies published under statistically probable names." A USENIX Security 2025 study of 576,000 generated code samples from 16 models found an average hallucinated package rate of at least 5.2% for commercial models and 21.7% for open source models, with 205,474 unique hallucinated package names.

Deterministic is necessary, not sufficient

An honest version of this argument has to include the next point. SBOM generators are deterministic, and they still disagree with each other. Pointed at the same commit of expressjs/express, they can return different lists, because each decides for itself what counts as a component and which files to trust. The repository commits no lockfile, so a tool that resolves dependencies from one has none of the package.json dependencies to report at all.

A deterministic scanner reproduces the same incomplete answer every time. Repeatability is necessary, but what closes the gap is observation: recording what the build actually pulled, and saying when the record is incomplete. CRACI marks every job and cache as Complete, Complete with connections, Incomplete, Unavailable or Not recorded.

A worked example: scoping a new advisory

A hypothetical: an advisory is published for a version of a widely used npm package. Under the Cyber Resilience Act, manufacturers have 24 hours to report an actively exploited vulnerability to ENISA, so scoping has to be fast and right.

The AI first route

You ask an assistant to check your repositories. It reads manifests and lockfiles, and perhaps a Dockerfile. It does not see packages restored from a CI cache, tools downloaded by a build step, or versions resolved at install time in a repository with no lockfile. Asked twice, it may answer differently. Its summary is useful, but you cannot sign your report on it.

The record first route

CRACI aggregates vulnerabilities across builds and repositories, scoped to the default branch at the organization level, and re-evaluates monitored SBOMs as new vulnerabilities appear. Through the API you can trace an artifact to the build that produced it, to that build's CycloneDX SBOM, and to its network trace. If a job's record is incomplete, its completeness state tells you where to look by hand. You can still ask a model to summarize the result. It is summarizing a record, not generating one.

With the record, the question "which releases shipped this package?" is a lookup, not an investigation.

Side by side

Put each tool where it is strong

Use AI where judgment helps and a human checks the output: code review, explaining a finding, drafting a fix. Use deterministic build records where the output has to be reproduced, audited or relied on as complete: SBOMs, provenance and incident scoping. CRACI automates a significant part of that supply chain evidence for a wider CRA process. It does not make you compliant on its own.

Read how the record is made in build-time SBOM generation, or start with what the CRA asks for.