NoWeb can support multi-thousand page documents which can compile to tens of thousands of lines of code.
I used it at a deep tech startup I worked a number of years ago to document the theory behind why the code was doing what it was. Doubly useful since I could just use a regular bibtex citation system for papers which had done some part of what we were trying to do.
My code became the defacto technical onboarding document, still in use today, despite the fact that none of the code in it has been updated since I left.
Thanks for the insight. Just to be clear, writing documentation after the fact with lots of code snippets is obviously good and is standard practice.
You can take the next step and ensure that the entirety of the code is in your documentation as snippets. This usually doesn’t make much sense, there’s lots of code that it is not worth explaining in a literate style. And what’s the point of the documentation containing the whole program if it is written after the fact and you already have a standard more maintainable codebase as the ground truth? The fact that your literate code didn’t get touched says a lot.
To me the name Literate Programming implies that you write the code in a literate style from the outset. If you make it literate after the fact it is just normal documentation with snippets isn’t it?
>The fact that your literate code didn’t get touched says a lot.
The fact it's still used 5 years later without any of the code still being in production says more.
>To me the name Literate Programming implies that you write the code in a literate style from the outset.
This seems like a fundamental misunderstanding of what it means to write. You should perhaps look at how people who do it for a living write books or articles. The final document has little to do with what you spend most your time editing.
I absolutely hacked on the tangled source code of the program when trying to fix bugs or extend capabilities. Once I knew what I wanted to do I put it back in the literate program, usually finding a lot more bugs in the process.
> I absolutely hacked on the tangled source code of the program when trying to fix bugs or extend capabilities. Once I knew what I wanted to do I put it back in the literate program
Does NoWeb automate this "untangling" process in any way? I sometimes use Weave.jl [1] when I'm thinking out loud through code, and at times it would be nice to just work on the tangled code, refactor and reorganize things, and have it all untangle back into the original in some way. I have no idea how that would work though, and it would likely be pretty limited even if it existed, but I'm curious what the usual approach you take to this is.
No, in noweb programs you insert chunks of code in multiple places and have conflicts when you try and automatically merge the code back too often.
Org mode has a function which does this, but they didn't allow for arbitrary chunk nesting the last time I looked.
Emacs has a number of very useful features in the modes for noweb/tex, one of which is jumping to the chunk which the code came from in the tangled source code on the pretty printed PDF. This follows the spirit of what you want. In fact SyncTeX support comes pretty much out of the box for noweb files and makes their editing a breeze, either as text or code.
Of course if you're not on Emacs than god help you.
Would you please help me understand your workflow for
> jumping to the chunk which the code came from in the tangled source code on the pretty printed PDF
Do the codeblocks in your pdfs contain hyperlinks back to the org file where they came from?
No I'm using noweb. There is an option in noweb to add comments in the tangled code with the line and file from which they originate. Then there's an Emacs mode that let's you jump to that code. I wrote a little function that let you instead jump to the line in the same like in pdf using SyncTeX.
Perhaps I came across as too critical, I have a lot of respect for what you did and for the craft of technical writing. And I definitely understand that writing is not done linearly and is very iterative.
Correct me if I'm wrong, but it sounds like you were documenting existing code and that the result was very valuable as documentation, but not necessarily as code. You were acting as a technical writer not a programmer, it's a bit of a disconnect to call it Literate Programming, even if you were using Literate Programming tooling (NoWeb).
This kind of documentation is common practice all over industry and it is valued, but I don't think Literate Programming is considered to be widely adopted because of that.
I'm having a hard time even understanding what the question is here.
You seem to be confusing the tools with the work being done.
You can write a prototype of a C function in Python to see if you understand the requirements before you commit to the much harder task of writing it in C. That doesn't mean you're not writing a C program.
The same is true for literate programming. I can write code outside the main literate program when I'm not sure it's meant to do before I put it back in.
What I'm saying is simply that as you describe it, you are first writing the code normally, and then separately writing some documentation about it accompanied by code snippets for context.
But if that's Literate Programming then everyone is doing it and it's not a very meaningful label, it's just documentation.
I do get it, the distinction is that you are using NoWeb and you can convert between the documentation and the code, and that the documentation contains the entirety of the code. I suppose that's neat.
At some point, this boils down to a pointless discussion of semantics (my fault). "Literate Programming" as you describe it does not sound like a style of "Programming". Actually, when you reverse the Programming/Writing emphasis, it simply becomes "Technical Writing", which is what everyone does, because that's what's actually needed. And it is done by great writers rather than great programmers (which may describe Knuth, with the upmost respect).
I always interpreted it as writing the text and the code together, logging your thought process, thinking of code like a piece of literature as it is written, rather than adding some documentation to it later. The notion that writing it like this will yield better code, regardless of its value as documentation. I suppose that's why it was unproductive for me, it is a rather romantic interpretation (again, my fault).
I don't know how you code, but the first draft of code is never what ends up in the code base. Neither is the first draft of the documentation. You can write both together, but until you have an idea of what the structure of the code would look like, and how to split it up then you're better off doing multiple drafts.
As always code is read much more often than it is written and literate programming is used for the reading part, not the writing part. The efficiency comes in not having to guess what 0x5F3759DF is there for.
I have skimmed the TeX literate PDF (I did a number of times in the past too). Frankly isn’t it just like normal code with verbose comments? I have seen lots of code like that and it is not referred to as literate. The only difference is that this is a PDF, which makes it less practical and it is still not particularly readable as a book.
It might have great book-like typography but not the "narrative" structure that helps you properly understand how it works without getting bogged down in details first. There's no coherent outline, no chapters or sections for major systems or design decisions, no overarching overviews, no relating different parts and giving context. There's also no story of how it was built or a log of his thoughts throughout problem-solving process, that would have been another good angle. Instead it's just the code from top to bottom with embedded very local commentary. The code itself is actually rather hard to parse visually by modern typographic standards.
The issue is that probably I am misinterpreting what Knuth intended. The Literate Programming concept was a product of its time, and it has evolved into more practical modern documentation standards that are not so tightly linked to the code and don't exhaustively cover every line. The only problematic thing about it might just be the grandiose name Literate Programming, without that it's mainly good common-sense advice for quality documentation, but not necessarily a practical programming paradigm like the name implies.
Again, I'm having a hard time understanding what the issue is. It seems like you are deeply confused about what literate programming is and how it works.
All of the navigation issues are taken care of by using <<chunks of code>> in a nested structure. You follow the numbers in those, like a follow your own adventure game, to find out whatever you need.
The index has a listing of everything used in the program along with where it was defined and where it was used in case you want to find something specific.
More modern tools, like NoWeb, turn all of this into hyperlinks so you can jump around the pretty printed version without having to loop up page numbers.
I have read the paper in the past, I am well versed about WEB, and I believe I have done literate programming at length for a number of non-trivial projects.
I have explained my thinking in a separate comment (apologies for creating two branches). In short, I do think you are right and that I had an overly romantic notion of Literate Programming in mind.
I used it at a deep tech startup I worked a number of years ago to document the theory behind why the code was doing what it was. Doubly useful since I could just use a regular bibtex citation system for papers which had done some part of what we were trying to do.
My code became the defacto technical onboarding document, still in use today, despite the fact that none of the code in it has been updated since I left.
For examples you can read: http://brokestream.com/tex-web.html TeX is written entirely in WEB.