A new way to make maps with OpenStreetMap(protomaps.com)
protomaps.com
A new way to make maps with OpenStreetMap
https://protomaps.com/blog/new-way-to-make-maps/
101 comments
I’ve had success using tippecanoe to generate mvt tiles from geojson sources, which in turn came from various sources (osm and otherwise) from QGIS. I didn’t look into this project too much but it appears to be bundling mvt tiles into a single file? Either way, straight up mvt tiles are easy to generate and can be easily hosted statically on any web server/s3 bucket/cdn.
Thanks for sharing your experience. Much of this vocabulary is still foreign to me but you've given me some the relevant keywords to Google.
One of the main challenges when entering a highly specialised field is finding the vocabulary to express yourself.
One of the main challenges when entering a highly specialised field is finding the vocabulary to express yourself.
I'd also recommend having a look at https://openmaptiles.org/. They have workflows and editors for working with MVT and Maplibre. You can combine a base map from maplibre with other sources using Openlayers [0]
[0]: https://openlayers.org/en/latest/examples/mapbox-layer.html
[0]: https://openlayers.org/en/latest/examples/mapbox-layer.html
What kind of project can this tool be used for?
Right now it's best suited for mapping applications focused on some part of the world - neighborhood, city, country scale - especially ones that might avoid using a hosted service. Think visualization projects built for clients, newspaper site interactives, etc.
I think the author would explain best. Personally, I'm working on casting live earth shadows on a tiled map (link in HN profile) and lack an off-the-shelf tile set that contains all the data I need and in some cases the available tiles I use contain more data than I want. Also, the tiles cost $$$ for each tile load.
So long-term I am looking into generating and hosting my own map tiles.
So long-term I am looking into generating and hosting my own map tiles.
Depending on your needs, tilemaker (https://github.com/systemed/tilemaker) may be the least painful way for generating tiles from OSM data with modifications to it. Or at least it seems to be the least painful way of doing that that I've found so far. The input data is manipulated by Lua code that you provide, and some reasonable default configuration for it exists. Your custom Lua code could pull additional information from other information sources should you wish to do so.
[deleted]
If you need to create your own tiles, I'm not sure how this changes anything. I can't seem to find how one would go about generating one's own tiles for this platform in their documentation. Did you find something to that effect in it?
Author here. You should be able to first download OpenStreetMap tiles in PMTiles format (http://github.com/protomaps/pmtiles) and then write out a new augmented tileset in the same format. It's a use case I want to have first-class support for, but the tooling doesn't exist yet.
> You should be able to first download OpenStreetMap tiles in PMTiles format
To elabore on my comment, I was more wondering about how I generate those tiles from non-tiled raw PBF country extracts on Geofabrik.
To elabore on my comment, I was more wondering about how I generate those tiles from non-tiled raw PBF country extracts on Geofabrik.
The tile generation from PBF (actually OSMX) is not one of the parts that is open source; for that task I would look at https://github.com/systemed/tilemaker or https://github.com/openmaptiles/openmaptiles
I've already known about those, but thank you anyway.
Edit: replied to the wrong comment
This is very nice! Happy to see a few of my libraries in the dependencies :)
Also check out https://github.com/kothic/kothic-js, an older but similar attempt at making a full-fledged map renderer using 2D Canvas. It has a pretty good label rendering in particular which you might find useful.
Also check out https://github.com/kothic/kothic-js, an older but similar attempt at making a full-fledged map renderer using 2D Canvas. It has a pretty good label rendering in particular which you might find useful.
This looks great.
I am super excited about the parent tool, but unsure whether/when I can generate these bundles programmatically. I tried the web based tool and it works great, but what if I need to do it many, many times?
Can your tool do this easily? Is there a way to generate tiles (tiles.pmtiles) that are imported in the same way?
If I had stumbled on your project, I would not have been enthusiastic about it with just the writeup as-is. Seeing the parent project makes me see the possibilities: a completely independent mapping solution, free of any server/service tie in and additional costs. That is super exciting, and I would love to know if and how your project compares.
I am super excited about the parent tool, but unsure whether/when I can generate these bundles programmatically. I tried the web based tool and it works great, but what if I need to do it many, many times?
Can your tool do this easily? Is there a way to generate tiles (tiles.pmtiles) that are imported in the same way?
If I had stumbled on your project, I would not have been enthusiastic about it with just the writeup as-is. Seeing the parent project makes me see the possibilities: a completely independent mapping solution, free of any server/service tie in and additional costs. That is super exciting, and I would love to know if and how your project compares.
I'd like to understand what you mean by doing it many times - do you mean for different areas of the world? I haven't thought about API access to the bundles in detail yet, but you can contact me via email at [email protected] if you have some idea of what you need.
Thank you! Really appreciate the commitment to simplicity among the libraries I'm using like potpack, rbush, and polylabel. I did take a look at Kothic and agree the labeling is great; I did feel like I needed to write yet another Canvas renderer to take advantage of async/await and also eventually expose runtime-extendable custom Symbolizers.
This is really interesting to me. I do a lot of mapping of natural surface single track trails (for hiking and mountain biking) along the way be sure to properly name things, add intersection markers, and create relations (routes and superroutes) to document the systems.
The biggest downside thus far is a good way of displaying the data. mtb.waymarkedtrails.org gets close, but I'd really like something that displays relations in the specified color, has sane intersection markers, etc.
I feel like there should be something out there which effectively lets me design a style sheet and then apply it to a map which I embed somewhere (maybe even in an app).
This... feels like it? Or if not this, can any of you recommend another system?
Ideally I'd love something I can self-host. I tried getting a copy of waymarkedtrails going and then modifying that, but it's a bit too specialized and I kept running into quirks getting the toolchain set up.
EDIT: Looks like this might not include the highway=path data that I need, much relations. Alas.
The biggest downside thus far is a good way of displaying the data. mtb.waymarkedtrails.org gets close, but I'd really like something that displays relations in the specified color, has sane intersection markers, etc.
I feel like there should be something out there which effectively lets me design a style sheet and then apply it to a map which I embed somewhere (maybe even in an app).
This... feels like it? Or if not this, can any of you recommend another system?
Ideally I'd love something I can self-host. I tried getting a copy of waymarkedtrails going and then modifying that, but it's a bit too specialized and I kept running into quirks getting the toolchain set up.
EDIT: Looks like this might not include the highway=path data that I need, much relations. Alas.
Have you used GaiaGPS at all? I use it along with ridewithgps for looking at trail conditions and mapping points of interest on my rides and hikes. You can import your own custom map layers as well.
The developers are fairly consistently adding new content as well, might not hurt to reach out.
The developers are fairly consistently adding new content as well, might not hurt to reach out.
I have! It's great, but it doesn't show route relations for paths.
Author of OP here. Is this information that you're interested in contributing, and is appropriate for, OpenStreetMap?
The end goal is to build a system that lets you customize the tileset to include any node, way, relation, or tag in OSM. This gets quite tricky when considering automated cartographic generalization (how such features will appear when zoomed out) but it's one of my next steps.
The end goal is to build a system that lets you customize the tileset to include any node, way, relation, or tag in OSM. This gets quite tricky when considering automated cartographic generalization (how such features will appear when zoomed out) but it's one of my next steps.
Hey there! And yes, it is spot-on with the scheme for mapping trails.
As an example, take a look at this: https://mtb.waymarkedtrails.org/#route?id=6109588
That's kinda what I want but... better. My OSM username is the same as here if you'd like to see my contributions; I've mapped most of these systems.
As an example, take a look at this: https://mtb.waymarkedtrails.org/#route?id=6109588
That's kinda what I want but... better. My OSM username is the same as here if you'd like to see my contributions; I've mapped most of these systems.
waymarkedtrails.org is a great tool! Glad to see Canadian content in there!
A groomed snowmobile trail addition would also be great!
A groomed snowmobile trail addition would also be great!
Really interesting project/product! It's really nice to see new ways of rendering OSM data, and alternative map services.
I am developing another way to run your own map server (without mapnik!) from a low-end computer (https://github.com/jamesrr39/ownmap-app in case you're interested), but a little bit different to you in that I'm currently focusing on rendering raster tiles. I'm really happy I saw this thread, however, I hadn't heard of the OSM Express, MBTiles or PMTiles formats before and was/am rolling my own format, so it's really interesting looking at some ideas from these.
Thanks again for sharing your project!
I am developing another way to run your own map server (without mapnik!) from a low-end computer (https://github.com/jamesrr39/ownmap-app in case you're interested), but a little bit different to you in that I'm currently focusing on rendering raster tiles. I'm really happy I saw this thread, however, I hadn't heard of the OSM Express, MBTiles or PMTiles formats before and was/am rolling my own format, so it's really interesting looking at some ideas from these.
Thanks again for sharing your project!
Does anyone have a really good (maybe illustrated?) guide/short reference to the world of GIS? Every time I see GIS stuff mentioned the problem space itself, solution jargon, and various movements in the ecosystem (projects being unsupported, stewards/companies changing etc) always throws me for a loop.
It's certainly the case that I just haven't built enough mapping-related software to be comfortable but if there's a resource out there that's like "baby's first GIS application" I'd love to read through it and try to ram the concepts into my gray matter again for the day when I do work on one.
It's certainly the case that I just haven't built enough mapping-related software to be comfortable but if there's a resource out there that's like "baby's first GIS application" I'd love to read through it and try to ram the concepts into my gray matter again for the day when I do work on one.
I didn't know any GIS when I started OpenStreetMap and I think it was for the better. A fresh approach often yields a lot of progress.
> when I started OpenStreetMap
> Created by Steve Coast in the UK in 2004[0]
Thanks for starting this way back then, the amount of value created directly and indirectly is immeasurable.
That said, I'm very doubtful of the likelihood of me contributing something novel and useful to the GIS world -- I'm keep to learn from and stand on the shoulders of giants that got us here for at least the first bit of my foray into GIS before trying to contribute something fresh, RTFM and all that. To be fair it has taken 15 years to whittle down this particular wheel, starting from scratch feels almost disrespectful -- I'll leave that for younger explorers.
[0]: https://en.wikipedia.org/wiki/OpenStreetMap
> Created by Steve Coast in the UK in 2004[0]
Thanks for starting this way back then, the amount of value created directly and indirectly is immeasurable.
That said, I'm very doubtful of the likelihood of me contributing something novel and useful to the GIS world -- I'm keep to learn from and stand on the shoulders of giants that got us here for at least the first bit of my foray into GIS before trying to contribute something fresh, RTFM and all that. To be fair it has taken 15 years to whittle down this particular wheel, starting from scratch feels almost disrespectful -- I'll leave that for younger explorers.
[0]: https://en.wikipedia.org/wiki/OpenStreetMap
Author of the OP here. I don't have a GIS background in particular; I do have a decade of experience building web mapping applications, and here's some of the foundational software I use:
http://leafletjs.com https://shapely.readthedocs.io/ https://github.com/libgeos/geos https://rasterio.readthedocs.io/ https://gdal.org http://www.angusj.com/delphi/clipper.php
http://leafletjs.com https://shapely.readthedocs.io/ https://github.com/libgeos/geos https://rasterio.readthedocs.io/ https://gdal.org http://www.angusj.com/delphi/clipper.php
Thanks for the pointers -- Leaflet I know as the visualization layer that is basically not-mapbox-GL. The rest are new to me, and will crawl through them.
One of the nice out-of-the-box systems I've come across is Pelias[0], what are your opinions on it? It bundles and provides the API so it's one of the things I've thought to self-host if I'm going to. It looks like it doesn't do tiles so I'd need to find something for that?
From deep in my bookmarks I also have:
- https://github.com/tidwall/tile38 (in-memory search+geofencing+other)
- https://openlayers.org/ (visualization, alternative to leaflet?)
- https://download.geofabrik.de/ (where to actually download the OSM data, to hand it to something like PostGIS)
- https://github.com/maptiler/tileserver-gl (A tile server... which presumably needs OSM data & takes coordinates with bounding box parameters?)
> I do have a decade of experience building web mapping applications > I don't have a GIS background in particular;
It's only on HN that you get statements like this, haha, what do you think 10 years of experience building mapping applications is? I building mapping applications is generally what people consider "GIS background" -- if you did GIS but never produced any maps... would the trees make a sound?
[0]: https://pelias.io/
One of the nice out-of-the-box systems I've come across is Pelias[0], what are your opinions on it? It bundles and provides the API so it's one of the things I've thought to self-host if I'm going to. It looks like it doesn't do tiles so I'd need to find something for that?
From deep in my bookmarks I also have:
- https://github.com/tidwall/tile38 (in-memory search+geofencing+other)
- https://openlayers.org/ (visualization, alternative to leaflet?)
- https://download.geofabrik.de/ (where to actually download the OSM data, to hand it to something like PostGIS)
- https://github.com/maptiler/tileserver-gl (A tile server... which presumably needs OSM data & takes coordinates with bounding box parameters?)
> I do have a decade of experience building web mapping applications > I don't have a GIS background in particular;
It's only on HN that you get statements like this, haha, what do you think 10 years of experience building mapping applications is? I building mapping applications is generally what people consider "GIS background" -- if you did GIS but never produced any maps... would the trees make a sound?
[0]: https://pelias.io/
It might sound funny, but making slippy maps in Web Mercator is emphatically not what people call GIS - GIS being:
* The technical part of the academic Geography field
* Data collection and mapping work done at agencies like the USGS and municipal governments
* Work done with ESRI software (see https://joemorrison.medium.com/esri-cant-be-stopped-3b063f3d...)
Yes, Pelias is a mature project backed by a great team at https://geocode.earth - definitely one of the best options if you need an autocompleting geocoder based on OSM+other data like http://whosonfirst.org
About OSM data extracts, I also run an extracts service at http://protomaps.com/extracts that provides on-demand data updated once a minute.
* The technical part of the academic Geography field
* Data collection and mapping work done at agencies like the USGS and municipal governments
* Work done with ESRI software (see https://joemorrison.medium.com/esri-cant-be-stopped-3b063f3d...)
Yes, Pelias is a mature project backed by a great team at https://geocode.earth - definitely one of the best options if you need an autocompleting geocoder based on OSM+other data like http://whosonfirst.org
About OSM data extracts, I also run an extracts service at http://protomaps.com/extracts that provides on-demand data updated once a minute.
Where do folks use tile38?
I don't really know what is throwing you for a loop. I have a Certificate in GIS from UC-Riverside, very respected at the time and some of my professors worked for ESRI by day and taught part-time for this certificate program.
Some basics:
Software-wise, GIS is a database hooked to mapping software so you can query the database and spit out maps as your answer. It's valuable because "A picture's worth a thousand words" and most data can be connected to a mappable bit of data (address, lat-long, etc).
A Geographic Information System has the following components:
1. Hardware
2. Software
3. People
4. Procedures
5. Data
6. Network (added more recently and not part of what I was taught, https://learn.canvas.net/courses/464/pages/unit-1-dot-2-iden...)
Data is the most expensive part of that. Good data is hard to come by.
People with the right skills and talents are the most critical part.
You can start a GIS project by surveying your in-house software, hardware, data flows, etc and putting together a use case. You probably already have some of the pieces you need to create a proper Geographic Information System where the data you already track can be hooked into a system for making it query-able and mappable so you can up your decision-making game.
ESRI is probably still the world leader in GIS software and they put out a certain amount of free info.
Some basics:
Software-wise, GIS is a database hooked to mapping software so you can query the database and spit out maps as your answer. It's valuable because "A picture's worth a thousand words" and most data can be connected to a mappable bit of data (address, lat-long, etc).
A Geographic Information System has the following components:
1. Hardware
2. Software
3. People
4. Procedures
5. Data
6. Network (added more recently and not part of what I was taught, https://learn.canvas.net/courses/464/pages/unit-1-dot-2-iden...)
Data is the most expensive part of that. Good data is hard to come by.
People with the right skills and talents are the most critical part.
You can start a GIS project by surveying your in-house software, hardware, data flows, etc and putting together a use case. You probably already have some of the pieces you need to create a proper Geographic Information System where the data you already track can be hooked into a system for making it query-able and mappable so you can up your decision-making game.
ESRI is probably still the world leader in GIS software and they put out a certain amount of free info.
Check out mapschool.io
Question that maybe HN can answer:
If I want to build a free open-source web application with a map view that can be annotated, works on mobile etc and uses Open Street Map data, what are the best options right now?
If I want to build a free open-source web application with a map view that can be annotated, works on mobile etc and uses Open Street Map data, what are the best options right now?
Great question, GeoDjango with leaflet seems popular.
For annotation UIs on the web I would check out how http://geojson.io works, the implementation is on GitHub: http://github.com/mapbox/geojson.io
Also, there is this comprehensive resource available on OSM.
https://switch2osm.org/
https://switch2osm.org/
Very interesting, I'm currently using Google My Maps as a dead-simple custom mapping tool. I'm using it to collect layers of information as I conduct a nationwide housing search. It would be great if there were a solution that was as simple to get started with as My Maps but also had the flexibility to easily add custom data layers programmatically.
> It would be great if there were a solution that was as simple to get started with as My Maps but also had the flexibility to easily add custom data layers programmatically.
I believe _with some trickery_ you can use Google Earth for that
I believe _with some trickery_ you can use Google Earth for that
Curious about your tool chain for real estate metadata. Is there a way to get a more retail-like experience and not have to deal so much with agents?
Paying a guy in Kenya to scrape data for me. Swear I tried to pay every national real estate API out there and no one is interested in talking to you at less than 7k a month and a full on lead gen business.
I'm curious what sorts of data you are collecting and how it helps to have it in map form rather than some other file format.
Towns across the country that meet our criteria overlayed with climate change maps. The existing layers around satellite and topo views are also super helpful to get an idea of if the area has good tree cover and interesting landscapes.
Thank you.
When we were city and house shopping, I kept wanted to be able to query houses for things that the major providers don’t provide query terms for. Even proximity to parks was difficult to discern or searching by walkscore, crime rates, etc, was difficult. Each house has that data, and you can search for some of it visually, but as a developer, you just want to run queries...You seem like the sort of person to crack that code when you do choose a city. I’d be curious if you do (or have) come up with anything :)
I am, but you really need to be at business scale to get data access, for an individual doing a 1-off search the economics don't make sense and the incentives of data providers arn't aligned.
That said there is a company (nextburb.com) who's CEO I've been in regular contact with who is trying to dramatically scale what kind of searches people can do. I've been talking to them about our needs and use-cases and they've been quite responsive.
That said there is a company (nextburb.com) who's CEO I've been in regular contact with who is trying to dramatically scale what kind of searches people can do. I've been talking to them about our needs and use-cases and they've been quite responsive.
> Existing formats like MBTiles are based on SQLite. This limits the ability to self-host maps to those confident running a server in production.
Is that comical, or is there a deeper meaning I don't understand?
Is that comical, or is there a deeper meaning I don't understand?
Originally, MBTiles was designed for mobile offline. Then, Mapbox started serving directly from the SQLite for web & other clients, creating the Node.js SQLite bindings in the process. Eventually, they had an unpacker for MBTiles uploads that would stick the tiles in CDN. So, really no more complex than old school static content hosting.
Source: I created MBTiles ;-)
Source: I created MBTiles ;-)
Author of OP here, I use the tile unpacker as well! I didn't point this out in the OP, but I found the ergonomics of uploading tiles to S3 etc get bad once you're over a couple tens of thousands of tiles, as each upload has some overhead. I've tested PMTiles on S3 successfully with hundreds of millions of tiles.
Deduplication is also important to save space on redundant ocean/earth tiles; MBTiles can deduplicate via views, PMTiles deduplicates by reference, but unpacked Z/X/Y tiles on S3 can't.
Deduplication is also important to save space on redundant ocean/earth tiles; MBTiles can deduplicate via views, PMTiles deduplicates by reference, but unpacked Z/X/Y tiles on S3 can't.
When I was playing around with hosting rendered Minecraft slippy maps, I found the upload overhead to S3 to be pretty poor due to the “lots of tiny files” problem. (It renders as hundreds of thousands of rasterized image files for the various zoom levels). You could parallelize it, but it was still poor, and made me want to come up with a way to pack them into fewer files and fetch portions of them at render time so that the upload could be faster. It sounds like that’s what you did for the the PMTiles format?
That's exactly what PMTiles is for, and there's nothing in it specific to maps or geographic coordinates, just a Z/X/Y pyramid structure. I chose to support Leaflet for now because it is very popular; there's other viewers like OpenSeadragon for which a decoder plugin would be possible.
Fantastic. The renderer I was using was indeed leaflet based, so this sounds drop-in easy.
I just read through https://github.com/protomaps/PMTiles
Nice work!
The only downside would be that these maps are often re-rendered periodically and only have a few tiles that actually get updated. In that case, it’s nice to have to update only a handful of files. But I think the benefits would far outweigh the costs. Upload one massive file repeatedly is still gonna be faster I would think. Thanks!
That was the impetus behind MBTiles as well. I was building out an iOS toolkit to render OSM-based maps on iPad, offline, and transferring 1000s of rasters (at the time) even over USB was way too slow and error-prone.
Oh wow - very cool. Thanks for this! I love SQLite, and think this is an excellent use case. (File systems, especially networked ones, don’t handle “lots of tiny files” well). I think using SQLite as the primary, writeable storage mechanism makes sense because I could ship that around easily to rendering nodes and incrementally update it easily. Then, I could export to a static PMTile for upload to S3 for serving purposes.
Neat! I'm hoping to dig into your design more... looks great.
Probably what they mean is that they want to be able to host the maps as static files that are served to the web.
So that you can FTP some html, js, css, images and map data files to a web host and call it a day.
So that you can FTP some html, js, css, images and map data files to a web host and call it a day.
AFAIU MBTiles still need some application server to transform x/y/z into proper tiles to be served, which means provisioning a server, scaling it, maintaning it, and all the fun that comes with it; PMTiles is a simple blob that the frontend will download (only desired parts) and render directly in the browser.
You can use tippecanoe (https://github.com/mapbox/tippecanoe) with the --output-to-directory flag to output the individual .pbf tiles instead of a single MBTiles file. These tiles can then be hosted on something like s3 and your map-rendering client can query the necessary tiles, no server required.
[deleted]
So I fee like it actually missed the mark as to why it's so impactful.
When you get to global scale tiling with building accuracy, you literally end up with hundreds of millions of tiles.
The upload time / cost per upload STACKS. However, the tilesets combined usually sits between 100GB and 200GB.
ONE upload of ~150GB seems is faster and more cost efficient via S3.
For comparison, Cloud Optimized GeoTIFF (COG) is specifically designed to be readable with HTTP Range requests, so the single file can be placed on S3 and be immediately usable.
As others pointed out, the MVTs inside can be unpacked for static serving. But a SQLite database is not itself usable in this way.
As others pointed out, the MVTs inside can be unpacked for static serving. But a SQLite database is not itself usable in this way.
I read it as: existing solutions are too hard for simple people with less technical skill.
Author here; a narrower way to express what I meant:
A frontend developer who wants to put a map in their application and wants control over the hosting and data faces some obstacles when using MBtiles:
* Need a virtual machine and disk space
* Need to choose a particular program to read the MBTiles and serve it via HTTP
* Might need reverse proxy and service monitor (systemd or sysv init) to keep it running 24/7
* Need to deal with SSL termination
The popularity of serverless platforms = evidence there's a strong market for delivering software without all of this.
A frontend developer who wants to put a map in their application and wants control over the hosting and data faces some obstacles when using MBtiles:
* Need a virtual machine and disk space
* Need to choose a particular program to read the MBTiles and serve it via HTTP
* Might need reverse proxy and service monitor (systemd or sysv init) to keep it running 24/7
* Need to deal with SSL termination
The popularity of serverless platforms = evidence there's a strong market for delivering software without all of this.
[deleted]
@bdon Awesome! The geospatial industry is in need of simpler tooling for specific jobs such as generating tilesets.
Currently complex tools such as QGIS and GDAL do everything, but the learning curve is very steep.
An addon you might consider a feature to drop an image (and adjust it to fit) over the OSM data to generate a XYZ tilesets directly on the customer AWS S3 (or equivalent storage).
This might be a good way to monetise your offering... I know my customers would buy it.
Currently complex tools such as QGIS and GDAL do everything, but the learning curve is very steep.
An addon you might consider a feature to drop an image (and adjust it to fit) over the OSM data to generate a XYZ tilesets directly on the customer AWS S3 (or equivalent storage).
This might be a good way to monetise your offering... I know my customers would buy it.
Is the author around?
Curiously, the editor does not render (well, it renders a black screen) in Firefox 78.9.0. I see this error in devtools:
Curiously, the editor does not render (well, it renders a black screen) in Firefox 78.9.0. I see this error in devtools:
Uncaught SyntaxError: expected expression, got '=' protomaps.min.js:1:52669
On props update af913d4d.0fa8ea22.js:1:7975 ReferenceError: protomaps is not defined
Yes, I'll look into this. Can you email me at [email protected] or make an issue at http://github.com/protomaps/protomaps.js ? I may need to make a get a sourcemap-enabled build online as well.
On the off chance the author is still hanging around here:
Do you mean it when you say on the blog that OSMX is faster than a relational database? e.g. faster than PostGIS.
And I presume you mean because of the up front cost of conversion?
Do you mean it when you say on the blog that OSMX is faster than a relational database? e.g. faster than PostGIS.
And I presume you mean because of the up front cost of conversion?
I gave a talk on this subject early on in this project, you can watch it here: https://2019.stateofthemap.us/program/sun/osm-express-a-spat...
I personally avoid blanket statements about a certain design being faster than relational databases; for this project as a whole I try to take a "systems programming" approach; I make things faster by doing less work, in this case because I am designing for one specific use case and no more.
For OSMX vs. PostGIS, this entails:
1. PostGIS is a client-server database, OSMX is embedded (in-process). PostGIS must be written defensively for cases like multiple writers; this isn't important for a system that occasionally has a single writer.
2. PostGIS indexes must be built for arbitrary query plans, OSMX only has two kinds of access patterns: by primary key and by S2 cell. S2 cell indexing is an approximation, but will always return more data than you asked for and never less. This is an acceptable downside.
3. PostGIS must treat geodata as OGC Simple Features like points, lines and polygons, this involves lossy conversion from the OSM data model, which is topological.
4. OSMX is built on LMDB, which is an outstanding and underrated piece of database tech (I think the author posts on HN). It's paired with a message format that does not require deserialization (CapnProto) so certain operations common in working with OSM data, such as accessing the coordinates for an OSM node, do not need to malloc for each node. OSM data can be many times larger than RAM; memory-mapping the entire database means you don't need to write your own caching layer; spatial access patterns exhibit high locality since cells are arranged on space-filling curves.
I personally avoid blanket statements about a certain design being faster than relational databases; for this project as a whole I try to take a "systems programming" approach; I make things faster by doing less work, in this case because I am designing for one specific use case and no more.
For OSMX vs. PostGIS, this entails:
1. PostGIS is a client-server database, OSMX is embedded (in-process). PostGIS must be written defensively for cases like multiple writers; this isn't important for a system that occasionally has a single writer.
2. PostGIS indexes must be built for arbitrary query plans, OSMX only has two kinds of access patterns: by primary key and by S2 cell. S2 cell indexing is an approximation, but will always return more data than you asked for and never less. This is an acceptable downside.
3. PostGIS must treat geodata as OGC Simple Features like points, lines and polygons, this involves lossy conversion from the OSM data model, which is topological.
4. OSMX is built on LMDB, which is an outstanding and underrated piece of database tech (I think the author posts on HN). It's paired with a message format that does not require deserialization (CapnProto) so certain operations common in working with OSM data, such as accessing the coordinates for an OSM node, do not need to malloc for each node. OSM data can be many times larger than RAM; memory-mapping the entire database means you don't need to write your own caching layer; spatial access patterns exhibit high locality since cells are arranged on space-filling curves.
Thank you for the detailed answer.
I have tinkered with PostGIS before, and it seemed great. Nonetheless, you make several good points about how a special purpose tool/format can be faster by avoiding certain work.
As for LMDB, I haven't used it, but I have heard about it previously. A full-text search server called MeiliSearch uses it internally.
---
Separately, I think your broader project is really cool. Simplifying the whole end to end process is incredibly useful. I have previously thought "I would love to host a regional map, but I have no idea how".
I have tinkered with PostGIS before, and it seemed great. Nonetheless, you make several good points about how a special purpose tool/format can be faster by avoiding certain work.
As for LMDB, I haven't used it, but I have heard about it previously. A full-text search server called MeiliSearch uses it internally.
---
Separately, I think your broader project is really cool. Simplifying the whole end to end process is incredibly useful. I have previously thought "I would love to host a regional map, but I have no idea how".
What’s the easiest way to self-host OSM data to use like a offline google maps?
In the past I made a map of train lines in and around Prague, which required quite a complex stack, since the information I wanted was in relations. How does Protomaps deal with relations?
It can deal with relations more flexibly than PostGIS-based workflows, because the tiles are generated from a near lossless representation of OSM instead of having to convert to OGC Simple Features first.
Can you email me at [email protected] with what specific tags and relations you are interested in? I want to support transit layers as part of basemaps and ideally this is an option when generating a PMTiles archive.
Can you email me at [email protected] with what specific tags and relations you are interested in? I want to support transit layers as part of basemaps and ideally this is an option when generating a PMTiles archive.
Got the demo up and running.
I've been using leaflet for my map tiling and it just works. The advantage of promomaps being offline is great.
So, now... how does one add their own POI's, routes, polygons??
I've been using leaflet for my map tiling and it just works. The advantage of promomaps being offline is great.
So, now... how does one add their own POI's, routes, polygons??
Author here, so there are a few options:
1) If your information belongs in OpenStreetMap, you can add it via an editor like the web editor at https://openstreetmap.org - this will also benefit all other OSM users. You can then "refresh" your Protomaps download to get a new map.
2) If there isn't many point and polygons, it may sense to add them as Leaflet layers, especially if you want them to be interactive
3) Other options are creating vector tiles of your own data and merging or displaying them in the renderer (https://github.com/mapbox/tippecanoe is a great tool to do this from GeoJSON) but I don't have much to support this yet.
1) If your information belongs in OpenStreetMap, you can add it via an editor like the web editor at https://openstreetmap.org - this will also benefit all other OSM users. You can then "refresh" your Protomaps download to get a new map.
2) If there isn't many point and polygons, it may sense to add them as Leaflet layers, especially if you want them to be interactive
3) Other options are creating vector tiles of your own data and merging or displaying them in the renderer (https://github.com/mapbox/tippecanoe is a great tool to do this from GeoJSON) but I don't have much to support this yet.
Thanks for the prompt reply (and for the awesome tool).
I'm getting old LOL... my question re adding POIs, etc. was a brain-fart. As you note, I'll just use leaflet layers!!
I'm getting old LOL... my question re adding POIs, etc. was a brain-fart. As you note, I'll just use leaflet layers!!
Hi again Brandon!
Been playing around a bit with the protomap's pmtiles.
I notice that POIs have bbox, geom, and just a couple addt'l properties (e.g. name, amenity).
Would it be possible to have the option of exporting the street address into the properties of each poi?
I'd like to be able to (fully offline) generate a delivery route given a list of addresses (that fall within the specific pmtiles region).
Any suggestions or possibility of including this data?
Been playing around a bit with the protomap's pmtiles.
I notice that POIs have bbox, geom, and just a couple addt'l properties (e.g. name, amenity).
Would it be possible to have the option of exporting the street address into the properties of each poi?
I'd like to be able to (fully offline) generate a delivery route given a list of addresses (that fall within the specific pmtiles region).
Any suggestions or possibility of including this data?
I am working on making the set of information included in the tiles configurable via a JSON definition, kind of like Overpass Turbo if you've used that.
The reason I'm doing it this way is because data completeness and performance - the total size of the tiles - are direct tradeoffs. If I keep augmenting the dataset with tags people request, they'll become too big for everyone!
So I'm exploring this idea of "a la carte" tags in tile sets, but am still gathering requirements. Can you email me at [email protected] with more details of the OSM tags you want, as well as the area you're interested in so we can work together on checking out the data availability there?
The reason I'm doing it this way is because data completeness and performance - the total size of the tiles - are direct tradeoffs. If I keep augmenting the dataset with tags people request, they'll become too big for everyone!
So I'm exploring this idea of "a la carte" tags in tile sets, but am still gathering requirements. Can you email me at [email protected] with more details of the OSM tags you want, as well as the area you're interested in so we can work together on checking out the data availability there?
Congrats for getting to the front page, would be cool to read the story how you are building protomaps as an indie dev on indie hackers or somewhere.
Thank you!
This looks like exactly what I was looking for as the base for a geolocation visualization of photo galleries on my Jekyll site. Thanks!
You can already host vector tiles within S3 that work with Mapbox and other libraries.
Not really sure what problem this is trying to solve.
Not really sure what problem this is trying to solve.
The typical way to build and distribute Mapbox vector tiles has been by packing them into a sqlite database with individual rows for each Z/X/Y quad tree coordinate. This is what tools like tippecanoe typically produce.
The problem with this is it still requires a running server process colocated with that sqlite db in order to service requests for individual tiles, like what you'll receive from a client-side mapping library.
This project has a lot of different parts, but one of them is a spec for serving this type of data at low latency without a server by combining a custom packing format with S3 range-get requests to read individual tiles direct from blob storage. So that is certainly interesting for this kind of use-case.
The problem with this is it still requires a running server process colocated with that sqlite db in order to service requests for individual tiles, like what you'll receive from a client-side mapping library.
This project has a lot of different parts, but one of them is a spec for serving this type of data at low latency without a server by combining a custom packing format with S3 range-get requests to read individual tiles direct from blob storage. So that is certainly interesting for this kind of use-case.
In a typical example isn't this what route 53 solves?
Is this a simple process? Can you share a link to how this is done, I'm interested. I always assumed it was fairly complicated.
I’ve used the utility tippecanoe with success to generate vector tiles from geojson sources and am hosting them statically. It’s been overall pretty easy. The only gotcha I can remember is I needed to pass the no compression flag since mapbox gl can’t read compressed tiles (and the file host will gzip everything anyways).
What issues have you run into with gzipped tiles? I store a lot of gzipped individual tiles for use in Mapbox GL maps on S3 with no problem - the only necessary change was making sure that S3 was emitting `content-encoding` headers.
I couldn’t get the tiles to show up at all when they were compressed by tippecanoe. I’m not sure what the problems is, to be honest.
At least for simple GeoJSON you can use tippecanoe (https://github.com/mapbox/tippecanoe) with --output-to-directory to get individual .pbf tiles. Most rendering clients will have some scheme where you can provide a root url and then the tiles need to be stored in some defined structure beneath that (e.g. root/z/x/y.pbf)
(posted something similar to another response)
(posted something similar to another response)
OP author here; one thing I didn't touch on is the way that most map services license their basemap data explicitly disallows "caching" which would be inclusive of hosting basemaps yourself on your own storage. So for the cases where it makes engineering or product sense to have your maps on S3, it goes against the dominant map business model.
Rooting for this!
As it was said
> "In about five minutes, you can select any area in the world and get a self-contained map that runs locally, offline or serverlessly on S3 - check out the Getting Started guide."
5 mins? It took me a night to build the map system using openstreetmap in the original way
> "In about five minutes, you can select any area in the world and get a self-contained map that runs locally, offline or serverlessly on S3 - check out the Getting Started guide."
5 mins? It took me a night to build the map system using openstreetmap in the original way
I spent part of today on the OSM wiki trying to wrap my head around Mapnik, how tiles are generated, how I am going to host this cheaply because my project is non-commercial, and then the HN gods smiled on me. Thank you for taking this on and I'm coming along for this journey!