Hey congrats on the Show HN. Local, browser based data exploration works for a lot of uses cases and is so much faster thancloud based tools. We've implemented something similar at https://addmaple.com/ - but with a graphical interface designed for rapid exploratory data analysis of large datasets.
Memory per tab can be an issue for really big files (1gb+) but we're exploring a transform to CBOR which allows us to free up JS memory, i.e. when parsing CBOR we can leave row level data as Uint8Array and it doesn't increase the JS memory overhead.
This is quite interesting - we've not explored really large files so far and being honest, we haven't thought that far either. Didn't know about CBOR! I will have to look deeper into how this can save on memory. I was wondering though, since WASM memory is limited to 4GB, if I have sufficiently large memory on my compute device, at least one tab should be able to handle 1gb+ files too, correct?
I've not done much profiling on DuckDB and what the overhead is - i.e. after the data is parsed how much memory is used. Would be really interesting to push it to the limit - or to explore not loading the entire file in, but only reading the relevant parts, but again that probably requires a conversion first, e.g. to parquet or some other column based storage format.
Memory per tab can be an issue for really big files (1gb+) but we're exploring a transform to CBOR which allows us to free up JS memory, i.e. when parsing CBOR we can leave row level data as Uint8Array and it doesn't increase the JS memory overhead.