When it comes to web archiving, I've found that Markdown has some real limitations. Sure, it's great for basic text, but it struggles with things like embedded content and non-standard layouts. Try archiving a Twitter thread or an app-style webpage in Markdown, and you'll see what I mean. It just doesn't capture the full picture.
That's why I've come to prefer formats like webarchive, mhtml, or single HTML files for archiving. They're incredibly faithful to the original content - you get almost perfect rendering of the original page, complete with styling and layout. Plus, they can capture stuff behind paywalls or on logged-in pages, which is a huge plus.
The real challenge, though, isn't just about saving the content. It's about making that saved content useful. These archive formats are great for preservation, but they can quickly become a mess of unorganized files that are hard to search through or make sense of.
I think the key is finding ways to organize and interact with these archives more effectively. Things like full-text search across all your saved pages, the ability to add notes or highlights directly on the archived content, and smart tagging systems could go a long way. And it'd be really powerful if we could integrate these archives with other knowledge management tools we use.
I develop a tool called HamsterBase that seems to address a lot of these issues we've been discussing. t's a local-first app. That means all your data stays on your own device - no need to worry about your personal archives being stored on someone else's servers. There's no sign-up or registration required, which is refreshing in today's cloud-centric world.
> [Markdown] struggles with things like embedded content and non-standard layouts.
I don't share that experience. I typeset all these documents using Markdown with pandoc's div extension, transformed into XHTML, and then passed to ConTeXt:
From XHTML, the document is transformed into TeX statements, which opens a world of possibilities. In the following video, custom styling is applied to nested contents:
Alternatives for authoring PDFs include LaTeX or similar markup languages, or GUI-based tools.
For many works, Markdown is more than sufficient for producing book-length texts (I've done this numerous times myself, either authoring my own works or transcribing/modifying books for improved access/readability). Markdown's benefit is that it is extraordinarily lightweight, and removes overhead from the authoring process.
Even where one ultimately chooses to migrate from Markdown to some more capable authoring format, Markdown remains useful for creating the original rough form of the work. Complex elements (figures, formulae, tables, etc.) can be indicated and, after document conversion from, say Markdown to LaTeX, fleshed out in full.
With tools such as Pandoc (see my earlier comments on it), it's trivially possible to create multiple outputs (I usually refer to these as "endpoints") of a document. I've used Makefiles to drive this process, such that I write source in Markdown and generate partial or full HTML documents,[1] other LWMLs,[2] PDF, ePub, straight ASCII/UTF-8/Unicode text, word-processing formats, etc., as I want. The set of Markdown + Pandoc makes this trivial in ways that, say, LaTeX alone isn't entirely suited.[3]
It's of course possible to use another LWML as the source format. Markdown has its limitations, but is most widely known and implemented, and limitations workarounds are typically reasonable.
________________________________
Notes:
1. A partial HTML doc may be useful for dropping into a larger document, and doesn't require global HTML elements such as the <html>, <head>, <body> tags, or others such as <nav> or <aside> in most cases.
2. Lightweight markup languages such as bbCode, AsciiDoc, RST, MediaWiki, OrgMode, etc., etc., see: <https://en.wikipedia.org/wiki/Lightweight_markup_language>. Useful when inserting the document into systems based on these formats.
"Markdown-formatted document should be publishable as-is, as plain text, without looking like it’s been marked up with tags or formatting instructions."
Any text editor (Notepad, TextPad, (neo)vi(m), Emacs, TextMate, Apostrophe, GhostWriter, Typora, etc.) will do. Markdown-specific editors have either a real-time preview or the ability to edit as WYSIWYM:
When it comes to web archiving, I've found that Markdown has some real limitations. Sure, it's great for basic text, but it struggles with things like embedded content and non-standard layouts. Try archiving a Twitter thread or an app-style webpage in Markdown, and you'll see what I mean. It just doesn't capture the full picture.
That's why I've come to prefer formats like webarchive, mhtml, or single HTML files for archiving. They're incredibly faithful to the original content - you get almost perfect rendering of the original page, complete with styling and layout. Plus, they can capture stuff behind paywalls or on logged-in pages, which is a huge plus.
The real challenge, though, isn't just about saving the content. It's about making that saved content useful. These archive formats are great for preservation, but they can quickly become a mess of unorganized files that are hard to search through or make sense of.
I think the key is finding ways to organize and interact with these archives more effectively. Things like full-text search across all your saved pages, the ability to add notes or highlights directly on the archived content, and smart tagging systems could go a long way. And it'd be really powerful if we could integrate these archives with other knowledge management tools we use.
It's an interesting problem space, and I think there's a lot of room for innovation in how we approach personal web archiving and knowledge management.
I have a read-it-later tool (https://hamsterbase.com/) that I've been maintaining as a hobby for 2 years, supporting Mac, Linux, Windows, iOS, Android, and self-hosted Docker.
All these versions share the same codebase.
VSCF: I extracted part of the source code from VS Code and developed a framework called VSCF. It includes commands, themes, dependency injection, key bindings, IPC.
Local server: On top of VSCF, I developed the underlying business logic, and file IO, SQLite, and the logic related to synchronization will be placed here.
Frontend: On top of VSCF, I used TypeScript and React for front-end development.
nodejs-mobile: This is an open-source project that allows me to use Node.js on mobile devices.
Self-hosted version = local server + frontend, using WebSocket for communication.
Desktop version = Electron + local server + frontend, using Electron's IPC for communication.
Mobile version = nodejs-mobile + self-hosted version. Users use webview to access the UI. It looks just like a native application. They can even use their phones as servers to access data on their phones from a computer.
I suggest trying out HamsterBase (HamsterBase is not open source).
1. It supports direct binding with SingleFile, enabling one-click web page saving. Because it saves in the browser, all other plugins will take effect.
2. It provides an open-source plugin https://github.com/hamsterbase/hamsterbase-highlighter, allowing you to annotate directly in the browser, and it automatically saves a snapshot of the web page when you annotate. When you visit the page again, it automatically displays the previous snapshots.
3. All data is stored on your local device, with both a docker version and a desktop version available. Different versions support P2P synchronization.
4. Provide full-text search function, which can search all the articles on the webpage.
Even though I have been developing for over a year, I still feel that it's not perfect. Currently, no account system has been developed in the software, only a donation website address is left on the official website. Users can leave their email after paying, and I promise to send them activation codes in the future.
The software price is 60, and the income in the recent 30 days is 300 dollars. The total income is 420 dollars.
In terms of PKM applications, believe CRDT is the future。
In response to this, I've developed a "read later" app based on CRDT. The following are my expectations for knowledge management software, as well as the goals I had in mind when designing this application:
1. I want my data to be stored locally so that I can access it anytime and anywhere, thus eliminating the need for expensive hard drives.
2. I hope the software can be purchased for a one-time fee, rather than by subscription. Since all the data is stored locally, this indicates that there are no additional costs for the developers. Therefore, I believe a one-time payment would suffice. At the same time, I'm open to the game DLC model, where enhanced features can be obtained through additional payments.
3. I hope the software can support synchronization so that I can access the most recent data on any device.
