Never Crash Is a Promise. Here's How We Test It.

Last week's post said every parser in bomwerk returns a result instead of throwing, so one bad archive can't take down a whole scan. That's an easy thing to claim in a blog post. It's a different thing to actually know it's still true after the fortieth commit touches that code.

So here's the part that doesn't show up in a design doc: how we check it.

A diff across 50 repos, not a handful of unit tests

Unit tests check the pieces you thought to write a test for. They don't catch what happens when a real project has a Cargo.lock with a dependency graph nobody would design by hand, or a vendored tree three levels deeper than any fixture we made up ourselves.

So alongside the unit tests, we keep a golden corpus: real, messy repositories, each with a snapshot SBOM checked into the test suite. Every CI run scans all of them fresh and diffs the output against the snapshot. Any byte of difference fails the build. Right now it's a modest set. We're growing it deliberately toward 50 by the time the harder milestones land. A corpus of five doesn't tell you much about a scanner meant to run on someone else's ten-year-old codebase.

The failure mode this catches is boring, and that's exactly the point: a change to the npm parser that quietly shifts how a nested node_modules tree gets walked, three files down, in a project nobody on the team wrote.

$ ctest --test-dir build -R golden --output-on-failure
Test #12: golden.corpus-07 ......................... Passed  (0.41s)
Test #13: golden.corpus-08 ......................... Failed  (0.38s)
    --- expected/corpus-08.cdx.json
    +++ actual/corpus-08.cdx.json
    @@ component[41] @@
    -  "version": "1.4.2"
    +  "version": "unknown"
Total: 14 tests, 1 failed

That's the kind of failure a human should look at before it ships. Either the parser genuinely regressed, or the expected snapshot needed updating on purpose. Either way, CI won't let it pass silently.

One fuzz target per parser

Golden corpus tests check known inputs. Fuzzing checks the inputs nobody wrote down: malformed archives, truncated lockfiles, a .a file with a symbol table that lies about its own size.

Every parser gets its own libFuzzer target, run for a short smoke pass on every pull request and for longer stretches overnight. When a fuzz run finds a crash, the crashing input becomes a permanent fixture. The bug gets fixed once; the input that found it keeps getting re-run on every future change, so the same class of bug can't quietly come back.

$ ./build/fuzz_nuget -runs=1 crash-8f21ac
==41213==ERROR: AddressSanitizer: heap-buffer-overflow READ 1
    #0 bomwerk::parsers::nuget::read_nuspec_xml()  nuget.cpp:118
    #1 bomwerk::parsers::nuget::parse()            nuget.cpp:41
artifact_prefix='./'; Test unit written to ./crash-8f21ac

That's AddressSanitizer (ASan), which is why the CI matrix runs a Linux job with ASan and UBSan (UndefinedBehaviorSanitizer) turned on. C++ will hand you a crash on a malformed input that a memory-safe language would just reject cleanly, and the crash isn't always where the bug is. A one-byte overflow can corrupt memory that only shows symptoms three function calls later. The sanitizers are what turn "the tool behaved strangely" into a stack trace pointing at the actual line.

This matters more than it sounds like it should, because a scanner's job is to read files it didn't write, from repos it doesn't control. Every parser is, in effect, reading untrusted input. Treating a parser bug as only a correctness issue, something to fix whenever, undersells what it actually is on a tool built to run in someone else's CI.

What this doesn't promise

None of this gets you to zero. A golden corpus only catches regressions on inputs already in the corpus; a repo shaped differently from anything in it can still surprise the scanner. A fuzz target only explores what it's had time to explore. A target added last week has seen far fewer inputs than one that's been running nightly for two months. New parsers start thin on both fronts and get more coverage as releases go by, not on day one.

What you actually get is smaller than that, and more honest about it: crashes we've already found don't come back, and the categories of input most likely to break something (malformed, truncated, adversarial) get exercised automatically instead of only when a real customer's weird repo happens to hit the same edge case in production.

A tool that never crashes on your repo isn't the same claim as a tool that's always right about your repo. Testing for one doesn't test for the other, and conflating them is how a scanner ends up with more confidence than it's earned.