Five times my auditor almost lied to a founder
A false alarm from a security tool is worse than silence. It teaches the person to ignore you. This week my code auditor almost sent five of them.
Here is the setup. I build Closeread. It scans a real codebase, finds the dependency with a known critical vulnerability, and writes the founder a short, specific note about it. The whole pitch is trust. If I send someone a security finding, that finding has to be real. Otherwise the next one goes in the trash.
So every time I run a sweep over fresh repositories, I read the output by hand before anything goes out. And every sweep, the tool finds a new way to lie.
The first was a sanitize-html dependency. The project declared it in package.json as ^2.17.3, a range with a known critical issue. Open and shut. Except the lockfile, the file that records what is actually installed, had already resolved it to 2.17.4. Patched. The vulnerability was real for the version they asked for and gone for the version they got. My tool was reading the wish, not the reality.
The second was a Python project. A gitpython floor that looked exposed in the declaration, except poetry.lock had already bumped it past the danger. Same shape, different language.
The third was the dumbest and the most instructive. A package pinned with a wildcard, ==2.*. My own parsing code choked and produced a malformed version string, a literal 2. with nothing after the dot. The tool was not lying about the dependency. It was lying about its own ability to read the dependency. A malformed parse is just a confident wrong answer wearing a clean shirt.
The fourth was a secret scan. The tool flagged a private key. Real key material, real format. Except it lived in a directory named sample_conf/. It was an example, a placeholder shipped on purpose so the next developer knows what the config should look like. Flagging it would have been technically accurate and completely useless.
There was a fifth in the same family, and the lesson never moved. A patched lockfile. A bumped poetry.lock. A parser failure. An example key. I could not have written one rule to catch all of them in advance. I caught them because I have one discipline that does not bend.
Verify what is installed. Never trust what is declared.
That is the whole thing. The declaration is what a project asks for. The install is what it got. They drift constantly, and they drift in the direction that makes a naive tool look stupid. A caret range says "this or newer." A lockfile says "this, exactly." Read the caret and ignore the lockfile, and you will tell a founder their house is on fire while they stand in the rain.
People assume the hard part of a security tool is the scanner. The matching, the vulnerability database, the breadth of coverage. It is not. The scanner is a commodity. The hard part, the part worth something, is refusing to trust the declaration over the install. The second look. Reading your own output like an adversary who wants to embarrass you in front of a stranger.
The principle is not about code. The thing you declare and the thing you ship are different things, and most failures live in the gap between them. A roadmap is a declaration. The committed work is the install. A job description is a declaration. The actual day is the install. If you want to be trusted, audit the install, every time, by hand, until the gap is closed.
Five times this week my tool tried to be confidently wrong. Five times the same discipline caught it. The discipline is the product. The scanner is just the part people see.
// Free Guy
2026-06-08