At a first approximation, no one should write their own memory allocator. Just as no one should write their own garbage collector, and no one should write their own filesystem.
All these subsystems have one thing in common: they are incredibly complex and very hard to get right, but seem easy on the outside.
These are complex to make robust, but they are not hard concepts. I think everyone should write their own GC, write a filesystem, and handle their own memory layer, at least once.
Treating lower levels of the stack as "too complex, there be dragons, just write some Javascript that hits RESTful API for JSON" is a great way to ensure you never progress as a software dev or an engineer.
I agree that reinventing the wheel poorly - without understanding geometry or physics - is a terrible idea. But I also think that we don't have enough people nowadays making toy wheels to learn about geometry and physics.
For each of the things you mentioned, there are concrete and good reasons why they grow complexity: filesystems need to be robust to media corruption and power faults; memory allocators need to be highly tuned to OS and architecture memory layouts; GCs need to be optimized for the code patterns of the particular language they serve, as well as understand the same OS and hardware memory constraints as allocators.
But much of the complexity of these layers is not intrinsic to the problem, and not related to the above. In fact, most of the complexity and bugs of production code at these layers have to do with legacy compatibility, or code inheritance. (If you look at what Theo says about this specific OpenSSL bug, it arises because the OpenSSL team wrote their own allocator that doesn't use malloc() protection bits because those are not compatible across some platforms.)
As the old saying goes: in science, we advance by standing on each others' shoulders; in software, we stand on each others' toes.
> I think everyone should write their own GC, write a filesystem, and handle their own memory layer, at least once.
Of course. But not in production. Which is a detail missed by the snarky one-liner, maybe because one-liners suck. Still, the point isn't that people shouldn't ever write them, just that they shouldn't actually use the ones they developed while not being a part of a team of experts specializing in the issue at hand.
I think the important part of the previous comment is "At a first approximation" - of course there is nothing wrong in writing code with the primary goal of personal understanding, but re-inventing the wheel when there are potentially very serious consequences for others is hubris (unless, of course, you have a really good reason why, NIH not being a good reason in my book).
[NB Ken Shirriff's blog articles on Bitcoin are a superb example of someone coding in an effort to understand a system - but I don't think he is attempting to write a real Bitcoin library]
> These are complex to make robust, but they are not hard concepts.
Yes, yes, I didn't quantify my statement with "for production use".
Your sentence I quoted above was exactly my point: they aren't necessarily hard (or at least they don't seem hard), but they are very, very difficult to get right (robust). Getting systems like these to work well in practical production use is more than 80% of the effort.
I wasn't saying that no one should learn how a garbage collector works by implementing one. My point was that no one should implement their own garbage collector for a production system unless they already became an expert in the field of garbage collectors by implementing them for the last 10 years or so. Same goes for memory allocators. If someone thinks these are simple systems, it means they don't know what they don't know.
Indeed. The proper thing to do is: write your own malloc, learn an enormous amount from implementing malloc, never ever under any circumstances allow your malloc implementation to enter into production use.
Hopefully the act of writing your own malloc implementation will be enough to convince you not to use it in anger, but it's probably best to be clear on the subject.
The discouraging of everybody who wants to provide an alternative only locks us in to supershitty software that got there first. Worst is Better reigns.
I mean, there is an extremely small chance that anybody will adopt your library if it is unquestionably better than the dominant alternative. We don't actually need to discourage people, we need to do everything to encourage them. Sure, 99 of 100 will be no better than our present crap. What does it matter? Few will adopt it anyway. The tragedy is that the better alternatives will probably meet the same fate.
The idea is to leave each component to the experts. If you're a concurrency/memory expert, then write a malloc library. If you're a cryptography expert, write your crypto code and use someone else's malloc.
"nobody" is not an absolute here. It's a reminder that the vast majority of people who get the idea to write their own memory allocator, garbage collection or filesystem will do an even worse job at it than the people who implemented the ones we typically use did, but it will end up going mostly unnoticed and possibly make it into some critical piece of software.
I don't understand this attitude, if something is incredibly complex and very hard to get right, this can indicate that there is a mismatch between the tools you use to implement it and what you should use.
With the right language it might be much more pleasant to implement:
Consider for example https://github.com/GaloisInc/halfs a filesystem implemented in Haskell, I think it has ~10000loc
It can also mean that existing implementations are poorly documented and research papers on the subject place no importance on implementation details.
Of course you should not use your first implementation in production, but that is as obvious as not letting someone do brain surgery who has no prior training in it.
10kLOC of Haskell for a filesystem seems like an awful lot, especially when considering how much more concise and higher-level Haskell is supposed to be compared to languages like C. Maybe filesystem code just don't fit too well into the functional paradigm?
For most uses, you don't really have to get them right, though. The problem is knowing when yes, you absolutely do need to get this right.
The cost in effort is highly non-linear. It may be fairly straightforward to get the 80% solution. I agree that it's a problem when devs bring their home-grown 80% solutions to the table when you need a 99% solution.
I make games. No one dies if there's a bug in the AI. No one may even notice.
> All these subsystems have one thing in common: they are incredibly complex and very hard to get right, but seem easy on the outside.
And have teams of operating systems engineers whose entire existence is predicated on them, rather than them just being an afterthought so you can get some other stuff done. If the OS does it, you shouldn't.
Au contraire, crypto developers probably should write their own memory allocators (just, you know, not quite so badly,) otherwise they're open for timing attacks.
There is a difference between complex and complicated. Crypto is a complex issue, with some subtleties that elude too many people. Low-level crypto, like this, has many small little details (such as allocating and deallocating buffers) that make it very complicated.
