"Most internal documentation only begins to make sense to a developer after they’ve developed an internal mental model of how it all hangs together. Most code documentation becomes useful after you have built the theory in your mind, not before. It operates as a mnemonic for what you already know, not as a tool for learning."
This reveals the secret goal of documentation: To build that model, not to document every last detail. The code can do every last detail, it's the model that is hard.
Set out with that goal and... yeah, I can't guarantee success for you either. It's a fundamentally challenging task. But at least you'll be off to a better start than if you think the primary purpose of documentation is to document what happens if you set this combination of flags. That has its place too, certainly, but if I can only pick one, I'd take the high-quality high-level overview over the stuff I can read the code for in 30 seconds.
The mental model really is not in the documentation, it is in the source-code modules how they interact with each other. Therefore I think OOP is a great way for building software. you need to create a mental imaginary world where the objects exist and interact. Then it makes sense.
But like you say it is challenging, even finding good names is hard. But basically, can you write a story about the software working, that would allow readers to understand it? Create user-stories but not (only) about the users but understanding that any module is a "user" of its components.
The thing is every project has mistakes. Things that aren't where they're supposed to be, that somebody means to tidy up sometime later, but just hasn't gotten around to yet. Every project has at least a little bit of a mess somewhere. When all you have to go off of is the code itself, then can be hard to tell the difference between the mess and the design.
A junior dev who comes in without much coaching, might just reference an anti-pattern from the "mess" section and replicate it throughout the code base. And then do that enough times, and a senior dev who gets brought in might conclude that there's no design at all, and that what it really needs is a rewrite.
Good point. And I'm not saying documentation is not needed. But that documentation should document the "theory" that already exists, or should exist in the code. When code is documented we do that in terms of the names of modules and classes and methods that exist in the implementation. But if those don't make much sense then neither does the documentation.
Maybe a good metaphor is to view the code as a simulation of something, and the documentation as the description of what that something is and how it behaves.
For instance say the software implements matrix multiplication. It is easy to document if its "units" correspond to the well-known concepts from the theory of linear algebra. But if the code implemented matrix multiplication but used random names for its modules classes and methods it would be difficult to explain what the code does.
Any explanation must assume that whoever reads it has a basic understanding of the basic concepts the software calculates and manipulates. But as said it is challenging to come up with such a theory and that must be done when the code is written, so the code itself reflects that "theory".
This reveals the secret goal of documentation: To build that model, not to document every last detail. The code can do every last detail, it's the model that is hard.
Set out with that goal and... yeah, I can't guarantee success for you either. It's a fundamentally challenging task. But at least you'll be off to a better start than if you think the primary purpose of documentation is to document what happens if you set this combination of flags. That has its place too, certainly, but if I can only pick one, I'd take the high-quality high-level overview over the stuff I can read the code for in 30 seconds.