Host your own OpenStreetMap Map Tiles(altilunium.my.id)
altilunium.my.id
Host your own OpenStreetMap Map Tiles
https://altilunium.my.id/blog/osm_selfhost/
44 comments
That's vector tiles, right? I'm still looking for an easy-ish way to render and host the old style of pixel tiles. Does anyone here have good pointers for that?
This very much depends on your use case. I recently did the planet for zoom level 12 as I mentioned elsewhere on this article. I can go into details if you're interested.
maplibre-gl-native can render vector tiles to the image format of your choice. tileserver-gl is a nodejs server which still uses the very old mapbox-gl-native but hopefully soon will use maplibre-gl-native to render raster tiles. It requires a mbtiles dataset (which you can either download from somewhere, or create yourself with Planetiler) and a map style. (like osm-bright)
i think we are hugging the webserver to death
Another easy shot at it is to install your proxy server for OSM tiles [these are served with a simple API from OSM site] and cache the images locally with periodic purge every six months, for example.
For giving back to the community, you can share your proxy server with others.
For giving back to the community, you can share your proxy server with others.
OSM does not serve vector tiles, so if you want to style the map this will not work.
The way I've done this is to use planettiler to generate vector tiles for the whole world and then pack that into a squashfs archive. That way I don't need an additional program to serve them, and I can regenerate the whole archive every couple of months or so (I don't need the latest data).
An exciting new standard here is pmtiles; a standardized way to store map tiles that can be used via http range requests. I believe planettiler has plans to add pmtiles output support as well.
https://github.com/protomaps/PMTiles
https://github.com/protomaps/PMTiles
Nice and looks very interesting! How would PMTiles handle compression? IIRC content-encoding and range requests don't play nice together, currently my server supports uncompressed, gzip and brotli for my tiles (compressed on-demand once and then cached).
For the next revision of the JS decoder I'm including `fflate` as a dependency so clients can decompress gzipped tile data using JavaScript.
https://github.com/protomaps/PMTiles/blob/master/js/package....
https://github.com/protomaps/PMTiles/blob/master/js/package....
Would be much better if this involved hosting on a normal webserver, rather than requiring a specific one. Another piece of software to keep up-to-date, just for maps.
If you skipped the "convert to MBTiles" part, you could host it on a normal web server... the issue is that requires each tile to be a separate file, and copying hundreds of millions of tiny files is brittle, slow, and expensive. Having a specialized web server lets us use a more effective file format for the data.
Having JUST done this for zoom level 12 for the planet it's tedious. It took 9 days, to export to mbtiles, another 2 days going through pngcrush, and about 40 minutes to make it into a squashfs resulting in a 2.6G squashfs file which houses the 22,368,460 PNG files which are then served up directly via nginx in our case.
It would be cool if somebody could crowd-source an EBS volume (or some other block storage) template of this that anybody else could easily attach to their cloud instance.
Indeed this is why MBTiles was created — especially for USB/device transfer and offline use.
You can host .mbtiles files with PHP very easily. I wrote a tiny piece of code 9 years ago that I used for a high traffic map server and it worked just fine. https://github.com/infostreams/mbtiles-php
I'm sure there are newer solutions out there, which might have more features and are actually maintained. But this did (and still does) the job.
I'm sure there are newer solutions out there, which might have more features and are actually maintained. But this did (and still does) the job.
The highest zoom levels are generally generated on demand, as there’s so many tiles.
Zoom level 20 is over a trillion tiles, most of them worthless blue sea tiles. We’re probably talking hundreds of terabytes just for the file system metadata, to say nothing of the tiles themselves.
Of course, you could forego the ability to zoom in very far.
Zoom level 20 is over a trillion tiles, most of them worthless blue sea tiles. We’re probably talking hundreds of terabytes just for the file system metadata, to say nothing of the tiles themselves.
Of course, you could forego the ability to zoom in very far.
With vector tiles you don't need to generate up to z20 - just to z14 is enough, and that can easily be done in a single .mbtiles (which is an SQLite database of vector tiles).
Here's a little bit of code I wrote that lets you host from a normal webserver, where that "normal webserver" is Apache: https://github.com/systemed/mod_mbtiles
Oh nice! I didn't know there was a Desktop version of MapTiler to make mbtiles! Instead I have followed pretty much the same but with tilemaker, which allows you to process your osm files through a LUA script to generate mbtiles
Wish you could self-host the tiles for your area in a tesla, say on USB.
Many car brands support that since many years. I think some cars had compact discs containing tiles. On my old BMW you can update the maps through USB. I rather get online maps with a cache to be honest.
Great idea
Thanks for sharing this! I have a hobby project that has hit the paid tier of my tileserver (thunderforest, and they're great) and I'm trying to figure out if I should be figuring out a business model or DIY'ing my tiles. Tileservers see to be expensive in general.
I had the same issue and was considering setting up my own tileserver. Fortunately I found www.tracestrack.com. Much less expensive that other options I considered and service has been good so far.
Thanks! This looks much more affordable.
I generated my own vector tiles using OpenMapTiles. Planetiler looks very promising too.
I serve the tiles using TileServer GL Light. It works but it requires Node.js v10.
OpenStreetMap is planning to make their own vector tiles style to convert their raster tiles. Looking forward to that! https://github.com/openstreetmap/operations
I serve the tiles using TileServer GL Light. It works but it requires Node.js v10.
OpenStreetMap is planning to make their own vector tiles style to convert their raster tiles. Looking forward to that! https://github.com/openstreetmap/operations
Related concept (recently submitted to HN) https://github.com/headwaymaps/headway
Does this do basically the thing as osmand on phones? Why the relatively huge requirements of 64G of ram and 10T of nvme?
Thats to build the server, any you are basically building the data for hosting approximately Google Maps for the entire planet here so seems relatively small really.
It’ll do it on what is basically a beefy dev machine. You can host a single city with less.
Also the requirements are for build not hosting. And can be reduced by using ZFS. Apparently.
To use the map in the browser requires nothing special.
It’ll do it on what is basically a beefy dev machine. You can host a single city with less.
Also the requirements are for build not hosting. And can be reduced by using ZFS. Apparently.
To use the map in the browser requires nothing special.
But using it in a browser presumably downloads image tiles from the server.
Osmand downloads vector data and renders it on the fly. It's both the "server" and the "client".
Osmand downloads vector data and renders it on the fly. It's both the "server" and the "client".
The beefy server part is for taking the gigantic OSM world database and turning it into vector data that can be consumed by the client. I believe Osmand runs their own beefy servers to pregenerate vector data for download, though it’s also possible to use Osmand Map Creator to generate your own vector data on your desktop.
Headway maintainer here. You don't need 10T of nvme storage unless you want to try and build tiles, a routing graph and a search index for the full planet without using a deduplicated filesystem. Eventually I hope to eliminate the need for deduplication but that's not currently a huge priority, since most Headway users don't run full-planet instances anyway. To my knowledge I'm the only one who does. Most people just run a smaller instance for their city.
It's not the same as OsmAnd obviously because it runs in the browser but it does try to fill the same niche. Also 64G of RAM for building the planet is a tiny machine! We don't know how good we have it these days. This used to be completely intractable for mere mortals. :)
It's not the same as OsmAnd obviously because it runs in the browser but it does try to fill the same niche. Also 64G of RAM for building the planet is a tiny machine! We don't know how good we have it these days. This used to be completely intractable for mere mortals. :)
Did the post just cut out mid way through?
That's the Extra Notes section, the actual hosting steps end just before that.
There is a project called Peermaps that is building a full P2P stack for OSM-based maps:
They convert Planet OSM and Natural Earth OSM dumps into the storage format used by their geospatial DB [1] and then distribute them using P2P file storage tools like IPFS [3] or hyperdrive [4].
Because those storage tools make use of content addressing [5] and clever chunking [6], you can download only changes to the (converted) OSM dump.
Because those tools are inherently P2P-focused and come with pluggable transports, the data representing the changes in the (converted) OSM dump don't have to be sent via the internet. For example, Mapeo [7] allows syncing "local" changes to OSM via USB sticks [8].
[1] https://github.com/peermaps/eyros [2] https://github.com/peermaps/data [3] https://ipfs.io/#why [4] https://hypercore-protocol.org/guides/modules/hyperdrive/ [5] https://docs.ipfs.tech/concepts/how-ipfs-works/#content-addr... [6] https://blog.gopheracademy.com/advent-2018/split-data-with-c... [7] https://www.digital-democracy.org/mapeo/ [8] https://docs.mapeo.app/complete-reference-guide/mapeo-deskto...
They convert Planet OSM and Natural Earth OSM dumps into the storage format used by their geospatial DB [1] and then distribute them using P2P file storage tools like IPFS [3] or hyperdrive [4].
Because those storage tools make use of content addressing [5] and clever chunking [6], you can download only changes to the (converted) OSM dump.
Because those tools are inherently P2P-focused and come with pluggable transports, the data representing the changes in the (converted) OSM dump don't have to be sent via the internet. For example, Mapeo [7] allows syncing "local" changes to OSM via USB sticks [8].
[1] https://github.com/peermaps/eyros [2] https://github.com/peermaps/data [3] https://ipfs.io/#why [4] https://hypercore-protocol.org/guides/modules/hyperdrive/ [5] https://docs.ipfs.tech/concepts/how-ipfs-works/#content-addr... [6] https://blog.gopheracademy.com/advent-2018/split-data-with-c... [7] https://www.digital-democracy.org/mapeo/ [8] https://docs.mapeo.app/complete-reference-guide/mapeo-deskto...
I'm a big fan of IPFS, but the "clever chunking" hasn't been very smart in my experience.
Peer maps seems to store the entire dataset in a single 107 GB file [1]
It's unclear to me that that eyros produces a file that doesn't change dramatically when data is inserted or removed.
It seems they are doing full imports, which likely means the output file could be completely different, meaning no shared chucks across generations.
In order to make IPFS actually create sharable chunks takes considerable effort, and is not as magically as their documentation and marketing would lead you to believe.
[1] https://bafybeidf5yn56cxk6zkyjmay4wigu2o7ynqh7q62z3kppag5v7j...
Peer maps seems to store the entire dataset in a single 107 GB file [1]
It's unclear to me that that eyros produces a file that doesn't change dramatically when data is inserted or removed.
It seems they are doing full imports, which likely means the output file could be completely different, meaning no shared chucks across generations.
In order to make IPFS actually create sharable chunks takes considerable effort, and is not as magically as their documentation and marketing would lead you to believe.
[1] https://bafybeidf5yn56cxk6zkyjmay4wigu2o7ynqh7q62z3kppag5v7j...
Is there a good way to host tiles for only a small part of the world? I have some apps that are only useful in my county - so I would like to have very detailed tiles for areas in my county, but there's no reason for me to generate or serve street-level tiles for somewhere in California or Tokyo or wherever.
Because it's hard to lock down tile servers for public-facing apps, I have always been a bit nervous putting anything online in case someone swipes my key and uses it and one morning I wake up to a huge bill.
Because it's hard to lock down tile servers for public-facing apps, I have always been a bit nervous putting anything online in case someone swipes my key and uses it and one morning I wake up to a huge bill.
Another option I've used, self-hosts vector tiles from an MBTiles database: https://github.com/akhenakh/kvtiles
Few things are easier! This is a piece of cake with https://github.com/onthegomap/planetiler#to-build-the-map
https://protomaps.com/ by u/bdon might be relevant