> I'd like to read more on better examples of dependency management.
I'll answer this before we do some Python history. Since there are a range of paradigms, if you really want a good sense of the main ways things are done and the tradeoffs involved in mature systems, take a look at: dnf/RPM as it's done on current Fedora, Gentoo Portage, Guix, Cargo from Rust, Flatpak, macOS' .apps, and Distri. Each of those represents a different approach.
(You could get away with just looking at dnf and Portage, which also represent modern iterations of the paradigms Python could have looked to in the early days of its own software distribution tooling, if not the earliest days of the language itself.)
SOME PYTHON HISTORY
That's a great question, in part because Python has been around so long! It's easy to collapse the past in a way that's uncharitable to projects like Python, so let me expand on this a little and cross-reference some dates to get it into perspective for both of us.
The best examples to consider today for dependency management are probably Linux distro package managers, *BSD ports systems, and Rust's `cargo`. But Python is older than Linux or any of the major surviving free Unices! So one legitimate difficulty Python has faced here is its age— what Python folks could have looked to for inspiration in 1991 was, compared to today, extremely limited and much less publicly available. They were not positioned, as the designers of Rust were, to really think through how code in the language would be distributed on day zero, before the language would see any real adoption.
There have, however, been opportunities for the Python community to rethink things in light of the work and experience of projects (like Linux distros or ports systems) where dependency management is a very central problem. There have also been occasions on which some breakage was already expected or permitted, and such a change might have been feasible. But I think the focus has been elsewhere, and the Python community and its leadership have had to face the very real risk that too many breaking changes in their language's tooling could result in shrinking or dividing the community in a way that Rust hasn't yet. So the issue isn't some unkind nonsense like ‘Pythonistas are idiots’ or whatever.
That said, by the time distutils rolled out as part of Python's standard library in 2000, apt, Debian's second-gen high-level package management tool was out and had undergone a couple years of battle testing. The *BSD ports systems had also been around for years by then, and Gentoo would hit the scene leveraging a similar approach to build a whole Linux distro that same year. I'm not sure what Linux package management tools were like, but that would certainly have been enough to tell already that static metadata for source packages was a must (and so something like setup.py is not appropriate).
Similarly with the creation of setuptools in 2004— by then there were lots of examples of mature package managers in two major paradigms: source-based (like Homebrew), and binary (like most Linux distro package managers). Similarly again with the rollout of Python 3.0 in 2008. When pipenv came out in 2018, the answer to questions like ‘why does pipenv take so long to freeze dependencies into a lock file’ (‘it has to download every package and pretend to try to build them’) revealed that the Python packaging world had failed to meaningfully incorporate lessons on package management from the outside world in the intervening 10 years.
Homebrew was a much later arrival, so its case is a little more baffling. But some of its choices that now seem naive or questionable should also be credited with its widespread adoption. For example, the choice to depend thoroughly on the underlying macOS system instead of isolating dependencies makes Homebrew pretty brittle in the face of upgrades between macOS releases, but it also made it much more convenient for getting things installed quickly from a blank slate state than other source-based package managers (i.e., MacPorts) for macOS at the time, since it meant less compiling from source. Similarly, while the simplicity of Homebrew's build recipe DSL is a current source of problems, that same simplicity (paired with embedding it in a popular, interpreted language) has made Homebrew feel approachable and helped to draw in the huge number of contributors that have worked on Homebrew's formulae.
I'll answer this before we do some Python history. Since there are a range of paradigms, if you really want a good sense of the main ways things are done and the tradeoffs involved in mature systems, take a look at: dnf/RPM as it's done on current Fedora, Gentoo Portage, Guix, Cargo from Rust, Flatpak, macOS' .apps, and Distri. Each of those represents a different approach.
(You could get away with just looking at dnf and Portage, which also represent modern iterations of the paradigms Python could have looked to in the early days of its own software distribution tooling, if not the earliest days of the language itself.)
SOME PYTHON HISTORY
That's a great question, in part because Python has been around so long! It's easy to collapse the past in a way that's uncharitable to projects like Python, so let me expand on this a little and cross-reference some dates to get it into perspective for both of us.
The best examples to consider today for dependency management are probably Linux distro package managers, *BSD ports systems, and Rust's `cargo`. But Python is older than Linux or any of the major surviving free Unices! So one legitimate difficulty Python has faced here is its age— what Python folks could have looked to for inspiration in 1991 was, compared to today, extremely limited and much less publicly available. They were not positioned, as the designers of Rust were, to really think through how code in the language would be distributed on day zero, before the language would see any real adoption.
There have, however, been opportunities for the Python community to rethink things in light of the work and experience of projects (like Linux distros or ports systems) where dependency management is a very central problem. There have also been occasions on which some breakage was already expected or permitted, and such a change might have been feasible. But I think the focus has been elsewhere, and the Python community and its leadership have had to face the very real risk that too many breaking changes in their language's tooling could result in shrinking or dividing the community in a way that Rust hasn't yet. So the issue isn't some unkind nonsense like ‘Pythonistas are idiots’ or whatever.
That said, by the time distutils rolled out as part of Python's standard library in 2000, apt, Debian's second-gen high-level package management tool was out and had undergone a couple years of battle testing. The *BSD ports systems had also been around for years by then, and Gentoo would hit the scene leveraging a similar approach to build a whole Linux distro that same year. I'm not sure what Linux package management tools were like, but that would certainly have been enough to tell already that static metadata for source packages was a must (and so something like setup.py is not appropriate).
Similarly with the creation of setuptools in 2004— by then there were lots of examples of mature package managers in two major paradigms: source-based (like Homebrew), and binary (like most Linux distro package managers). Similarly again with the rollout of Python 3.0 in 2008. When pipenv came out in 2018, the answer to questions like ‘why does pipenv take so long to freeze dependencies into a lock file’ (‘it has to download every package and pretend to try to build them’) revealed that the Python packaging world had failed to meaningfully incorporate lessons on package management from the outside world in the intervening 10 years.
You can watch/listen to a fairly non-technical overview of the history of Python packaging here: https://www.youtube.com/watch?v=AQsZsgJ30AE
Homebrew was a much later arrival, so its case is a little more baffling. But some of its choices that now seem naive or questionable should also be credited with its widespread adoption. For example, the choice to depend thoroughly on the underlying macOS system instead of isolating dependencies makes Homebrew pretty brittle in the face of upgrades between macOS releases, but it also made it much more convenient for getting things installed quickly from a blank slate state than other source-based package managers (i.e., MacPorts) for macOS at the time, since it meant less compiling from source. Similarly, while the simplicity of Homebrew's build recipe DSL is a current source of problems, that same simplicity (paired with embedding it in a popular, interpreted language) has made Homebrew feel approachable and helped to draw in the huge number of contributors that have worked on Homebrew's formulae.