Creator here. I maintain https://onthegomap.com as a side project. I originally used Google Maps API but after their price hike switched to OpenStreetMap data. I self-host GraphHopper for routing and tried to self-host OpenMapTiles but gave up and used Stadia Maps since it would have taken over 100 days [1] to generate a map of the world myself. Stadia Maps has been great and I highly recommend them, but the problem continued to nag me, so earlier this year I started prototyping a new way to generate vector tiles faster. The idea worked and ended up being able to generate a ~100GB planet.mbtiles file in as little as 59 minutes on a single c5ad.16xlarge instance with 64 cpus and 128GB RAM (although smaller machines appear to be more economical).
I’m open-sourcing Flatmap today. Let me know what you think, or if you have any suggestions!
Looks great! Potentially a game changer for projects that need a basemap for a smallish area (like one country), get a fair bit of traffic, but don't have budget for ongoing tile subscriptions to Mapbox or MapTiler.
This is really sweet, awesome work. I really love onthegomap too.
I maintain an openmaptiles pipeline, and we've got our planet runtimes down to about ~4 days on AWS.
Would be excited to see if we could use this tool, but we really heavily depend on PostGIS data manipulation to ensure that the data we want ends up in our output tiles. How difficult do you think migrating this SQL to your tool would be? It seems like you can support modifying the data fairly heavily, both before and after MBtile creation.
Flatmap calls into user-defined profiles in 2 places: first when processing each input element to map it to a vector tile feature, then a second time right before emitting all vector features in a layer.
That second call lets you manipulate vector features on each tile using JTS geometry utilities (i.e. merge nearby polygons or lines with the same tags). PostGIS uses GEOS which is ported from JTS so you have access to pretty much the same geometry utilities - often with the same name.
It's definitely not as flexible as a PostGIS based solution - especially if you join faraway features that don't appear on the same tile, but in practice it was enough to port the entire OpenMapTiles schema.
There's an ongoing effort to research vector tile solutions for openstreetmap.org. I wonder how your software stack compares to tilemaker, which has seen some optimization lately.
Thanks! Tilemaker is a great tool and probably the closest alternative (no intermediate db). The most recent benchmarks I saw though were around a day for the planet, so roughly 10x longer than flatmap. If
I hacked around with OpenMapTiles to generate vector tiles in Arctic, Antarctic and WGS84 projections (the latter with 2×1 tiles at zoom 0) [1]. The time spent waiting for PostGIS to do reprojections etc made development very slow, and I've put off updating the tiles.
This tool is reprojects from WGS84 to Web Mercator while reading the tiles, so optimistically changing that would be much of what's needed.
[1] See the map at the bottom of gbif.org, there's a button to change the projection
A bit off-topic, but do people who host their own map tiles worry that other apps will use/abuse their tileserver? I would worry that I'd come in one Monday morning and discover that I've got a $20K bandwidth bill because someone else got popular.
Nothing is foolproof, but for web-based traffic, proper CORS configuration or rejecting traffic based on the referrer (or lack thereof) would go a long way.
We self-host OpenMapTiles, but are super lazy about replacing them with updated tiles (and we don’t love the watermark). I’m really glad that there are options for generating them ourselves! Starred, and will give it a try soon.
at that cost it almost feels like something internet archive or aws open data could provide on a regular basis. the cost of pre-made vector tiles is huge, and making them yourself has been a huge ordeal before this...
The main driver of memory usage (~1.0x the input file size) is storing node locations so we can convert the list of node IDs on each way into lat/lon coordinates. By default nodes are stored in a memory-mapped file so it can run with less than 1.5x the input file size, just slower because of all the page faults. Try running with -Xmx=16g to leave 16GB free for memory-mapped file. it will print % complete as it works through the ways in pass2 so you can decide whether it's worth it to keep waiting.
Also you could try switching from the default --nodemap-type=sortedtable to --nodemap-type=sparsearray. Sorted table uses 12 bytes per node and is more compact for extracts, sparse array uses 8 bytes per node but wastes some storage when there are gaps in the ID space.
The benchmark says the whole world can be done on a "DO 16cpu 128GB" machine in 3 hours, so about $3 from Digital Ocean. Call it $5 to allow some time for transferring data.
Yep, that first one will do it! And you probably want to rip out the polygon merging code and minimum zoom on the attrs. You can use --only-layers= or --exclude-layers= if you don't want all the others.
I’m open-sourcing Flatmap today. Let me know what you think, or if you have any suggestions!
[1] https://github.com/openmaptiles/openmaptiles/issues/654#issu...