Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Over the last few months I tried to think of a clever way to set up a legacy site for a dictionary that I serve on a VM just because I also need to run sqlite. Since I want to make sure it'll run for longer than me paying for the VM this is the best possible solution. At some point no more updates will happen and it's going to be a static website. So bundling it like this is incredible. I can run multiple backups on different hosts with no additional costs.


If you just do an occasional key/value lookup, you don't need 1.2 MiB of WebAssembly. [1] That might already exceed your total database size.

I'd solve it via sharding: divide the database into N pieces via range- or hash-sharding. [1] Choose an N that's large enough for each piece to be reasonably small. When you look up a key, fetch the shard of interest.

You can put each piece into separate files (a little simpler to code, and most static servers will use pre-gzipped files for "Content-Encoding: gzip requests" easily, but you waste more disk space due to internal fragmentation) or one file (with range serving and an index of the byte range offset for each piece).

The format for each piece can be anything, eg json (simple) or an sstable-like format (more efficient). [3]

[1] Content-Length of https://phiresky.github.io/youtube-sponsorship-stats/sql-was...

[2] hash-sharding means: piece[i] has all the keys where hash(key) % N = i.

[3] https://github.com/google/leveldb/blob/master/doc/table_form... although they just say "formatted according to the code in block_builder.cc" instead of describing the most relevant part.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: