It’s worth mentioning that there is a bit more overhead in using multiple processes as opposed to multiple threads, in addition to it being more difficult to share data.
There’s probably a whole generation of programmers (if not two) who don’t know the feeling of shooting yourself in the foot with multithreading. You spend a month on a prototype, then some more to hack it all together for semi-real world situations, polish the edges, etc. And then it falls flat day 1 due to unexpected races. Not a bad thing on itself, transferrable experience is always valuable. And don’t worry, this one is. Enough ecos where it’s not “difficult to share data”.
This. Multi-threading is very prone to nasty, hard to reproduce, bugs if used liberally in a codebase. It really should be used with care, and compartmentalized to areas where it demonstratively brings critical performance improvements.
That is not all what I did. But there are several languages/ecosystems which are very prone to such issues - notably C and C++. It is critical important (in my opinion) that Python does not regress to anywhere near that state. This might seem like an invalid concern - after all Python is a high-level, memory-safe language - right? The problem is that use of extensions (mostly in C or C++) are very common, and they rely on particular semantics - which are now changing. A vast amount of Python programs make use of such extensions, probably the majority of programs (even if excluding the standard library). Some of the biggest challenges around nogil and multi-threading in Python are mostly around extension/C related stuff. It was notably also on of the main challenges faced by PyPy. So maybe it's actually a little bit tricky to get right - and not just the developers being stupid or lazy ;) I mean in addition to the the usual general trickiness of major changes making to the core of an interpreter relied by thousands of companies/projects, in a wide range of fields, built over decades, with varying level of quality and maintenance....
Right, I should have phrased it better - did not intend to make it sound like criticism of your reply. Was just aiming at the tendency to dismiss valid concerns with "it's actually a good thing we don't have it" or "it can't be done well".
Of course changing the concurrency guarantees the code relies on and makes assumptions about is one of the most breaking changes that can be made to a language, with very unpleasant failure modes.
Understood. There has been some amount of that in the past. And probably this kind of pushback will rise up again as the work starts to materialize. I think some are a bit fearful of the potential bad consequences - and it remains unclear which will materialize and which will be non-issues.
And of course some have other things they wish to see improved instead, cause they are satisfied with current state. Actually many will be quite happy with (or at least have accepted) the current state - cause those that were/are not probably do not use Python much!
What I see from the development team and close community so far has been quite trust building for me. Slow and steady, gradual integration and testing with feature flags, improving related areas in preparation (like better/simplified C APIs), etc.
So is Turing completeness. But we can’t just put our heads into the sand and ignore these, as we require both Turing completeness and multi-threading in most cases. This is part of our jobs, and is what makes the whole field an art - both of these properties make reasoning about arbitrary programs harder than what is possible with math.
These two are not equivalent in practice. We’ve created a good set of tools that help one to stay in parts of TC that we agreed to consider relatively sane. TC is much bigger than we actually use. It is an enormous landscape, and we only dare to walk very specific paths on it, even our hardware specializes on methods we tend to use.
A similar set of tools and guardrails has yet to be seen for MT, at least in python-ic category. I think we can agree that relative success of MT in some X doesn’t automagically translate everywhere, because it depends on first principles which are different.
There’s probably a whole generation of programmers (if not two) who don’t know the feeling of shooting yourself in the foot with multithreading. You spend a month on a prototype, then some more to hack it all together for semi-real world situations, polish the edges, etc. And then it falls flat day 1 due to unexpected races. Not a bad thing on itself, transferrable experience is always valuable. And don’t worry, this one is. Enough ecos where it’s not “difficult to share data”.