Note the other child comments of this; NPM has failed to make the lockfile reliable across systems and versions. Here's an example from March that isn't fixed: https://github.com/npm/cli/issues/2846
The fact that you have so many people believing "nuke node_modules and delete package-lock.json" is a reasonable step in diagnosing an error is damning to NPM.
We don't check in our node_modules, but "use the lockfile" is not a valid counter to this article's points.
I honestly don't understand how people get this impression of lockfiles as being perfectly reliable. How are they not occasionally bitten by these bugs? Maybe I'm just unlucky, but I'm a little jealous of these developers who apparently are good enough managing/updating their dependencies and keeping their count low enough that they've just never run into problems like this before.
Lockfile v1 literally ignores pinned versions of dependencies if the package.json specifies a fuzzy version number[0], and the advice of the npm team was, "it's fine, everyone will just bump a major version number of npm." And to this day, I still don't know what the expected behavior is, there really isn't a list anywhere about when the lockfile is and isn't supposed to be respected. So it's not really surprising to me that people distrust version pinning, and I always feel like I'm kind of living in a different world when people say that lockfiles just solve everything.
npm went through a rough few years (lockfiles, leftpad) and obviously the hivemind of the JS ecosystem is not the most careful one (hence all the advice of nuke it and npm i).
but those who care use yarn, those who even want to be correct use yarn2, and so on.
Indeed. What's probably needed here is a way to review a diff of the contents of the updated packages. Checking them is is just a brute-force way to do that.
not to mention that unless someone is very familiar with the code of dependencies it's very hard to review hundreds of small near meaningless changes unrelated to your actual functional/business requirements.
something like cargo-crev for npm might be a long term solution
I’ve heard lots of people claim that yarn gives no advantage over npm anymore, as of a year or two ago. But in 5 or so years of using yarn every day, on numerous projects, I’ve probably nuked node modules a couple of times, and never even considered deleting yarn.lock. Maybe yarn is still superior in this regard?
Done. Then I delete it, and `npm install`. Then commit. The majority of people I have worked with do that. On Friday some dude was saying "shrinkwrap v2 is not shrinkwrap v1" and the advice was "delete it and npm install" then commit. (Payment company software manager).
In my experience this is probably not viable. A single, valid update to a direct dependency can bring dozens or even hundreds of updated sub-dependencies. And an audit fix will often update just the lockfile. Maybe I lack imagination, but I can’t think of a workable heuristic for determining how the lockfile was changed.
I think, instead, it would be good to move in a different direction for sub-dependencies generally. A rough sketch:
1. Packages state their dependencies as they do currently.
2. When released, they’re built and bundled by the package manager host.
3. Included in the bundle is a manifest of the dependencies used, specific imports used, and a hash for each (recursively until exhausted).
4. On install, identical code (same hash/same bundled result) is deduplicated.
5. A human readable record of dependencies and imports used is produced. It’s important that it’s human readable, because:
6. This should not be filtered out in diffs. It should be subject to review just like any other change.
All of this is pretty complex, and there are probably ways to reduce that complexity. But it has some obvious advantages:
- Only your direct dependencies are installed. Tons of bloat can be stripped out.
- Even deduplication can be performed on the package manager’s servers. And hashes aren’t a particularly expensive lookup.
- It would go a long way towards addressing audit fatigue: if your dependencies’ bundles don’t include affected code, the audit doesn’t apply; if they do, you can be reasonably confident the audit is valid.
- A (wild guess) huge amount of the time, sub-dependency changes will require little to no review. Their stable parts will seldom change, and the parts shared among several dependencies could be reviewed as one unit.
- Lock files themselves just need to track direct dependencies (and even then, only to support semver ranges).
What you're describing is more in spirit with the intent of a lockfile and worth exploring by package managers. But I do think a heuristic could be conceived for the lockfile-was-rebuilt situation: If a single top-level dependency declared with ^ or ~ was present in the lockfile when the existing resolution was still valid.
Ideally the onus is on the package manager to provide metadata in the lockfile for the strategies it took when generating.
Problem with yarn is that it doesn't actually use dependencies' lock files... So once you publish your library to npm your lock file doesn't do anything whatsoever.
That makes a lot of sense for file sizes—otherwise, common dependencies patch versions apart would be duplicated many time, and it would block you from upgrading a library's dependency for a security fix. You still get the important part of reproducible builds for your program. Rust's Cargo behaves the same way.
