The Nightmare Repo Has Real Numbers. Now You Can Check Them Yourself.

bomwerk/nightmare is a small, deliberately nasty C repository: a vendored library with a known CVE, an uninitialized submodule, a forked dependency, prebuilt binaries with no manifest anywhere. It ships with GROUND-TRUTH.md, twelve components checked by hand, and a script, tools/compare_tools.py, that scores any scanner against that list.

We've had numbers on that page for a while. What we didn't have, until this week, was a way for anyone outside bomwerk to generate bomwerk's own row themselves. The repo's own README said as much: bomwerk wasn't publicly downloadable, so its result was ours to claim, not yours to check. That gap mattered more than the numbers did.

The numbers

Recall is how many of the twelve hand-verified components each tool actually found. Over-reported is how many extra, wrong components it claimed on top of that. bomwerk found ten of twelve and got two wrong. That's not a clean sweep, and we're not presenting it as one. Manifest-only scanners found two or zero, which is the expected result against a repo built specifically to hide things from manifests: vendored code, static archives, a forked SDK with a version string that no longer matches its contents.

sbom-tool's zero isn't a bug in sbom-tool so much as a demonstration of what "manifest-only" actually means when there's no manifest to read.

What changed this week

The core went Apache-2.0 and public on 30 September. On 2 October, bomwerk/bomwerk got its first tag, v1.0.0-rc.1: source only, no prebuilt binary, but buildable by anyone who wants to run cmake and wait. Put those two together and the "bomwerk isn't downloadable yet" caveat on the nightmare README is no longer true. It just hasn't been updated yet, which we're fixing.

Running the comparison yourself looks like this:

$ git clone https://github.com/bomwerk/bomwerk.git && cd bomwerk
$ git checkout v1.0.0-rc.1 && cmake -B build && cmake --build build
$ git clone https://github.com/bomwerk/nightmare.git ../nightmare
$ cd ../nightmare && python3 tools/compare_tools.py --bomwerk ../bomwerk/build/bomwerk

You'll need syft, cdxgen, and sbom-tool installed separately if you want their rows too; the script only runs what it finds on your PATH. Nobody has to take our word for any row in that table anymore, including ours.

What this doesn't prove

One synthetic repo with twelve hand-picked components is not a market benchmark. It's a stress test built to surface one specific failure mode: tools that only read manifests, applied to a codebase designed to have none worth reading. A real C/C++ project won't look exactly like nightmare, and a scanner that does well here could still miss things nightmare didn't think to test for. We built the repo, we wrote the ground truth, and that's a conflict of interest worth naming plainly rather than hoping nobody notices.

What it does prove, now that it's reproducible, is narrower and more honest: against this specific, adversarial, publicly inspectable test, bomwerk finds more of what's actually there than three tools that read manifests for a living. Ten out of twelve, not twelve out of twelve. Two wrong, not zero.

Why bother publishing a benchmark where you don't win cleanly

Because a 10/12 you can rerun is worth more than a 12/12 you have to trust. The two misses and two false positives are exactly the kind of detail that disappears from a chart in a sales deck and survives in a ground-truth file anyone can read. If a parser gets fixed and a number changes, the diff is public too.

Clone the repo, run the script, and if your result doesn't match this table, that's a bug report, not a discrepancy to argue about.

Free repo scan: bomwerk.com