Overpass Turbo: A Web Based Data Mining Tool for OpenStreetMap(overpass-turbo.eu)

215 pointsby stefankuehnelvor 2 Jahren45 comments

Overpass Turbo: A Web Based Data Mining Tool for OpenStreetMap

https://overpass-turbo.eu/

46 comments

I've played around with Overpass and GPT a bit. Here are a few examples that I think demonstrate the potential:

Find all buildings that straddle the boundary between Glendale and Burbank in California:

https://twitter.com/lemonodor/status/1636849040548675584

Finding the suspected origin of an explosion, by using the time from seeing the explosion on video to the time the sound of the explosion reaches the microphone:

https://twitter.com/lemonodor/status/1636859983223734273

Someone on twitter asked "Is anyone aware of any 4 way stop intersections in Australia?"

https://twitter.com/lemonodor/status/1516239321094713346

Finding examples of airport runways that cross highways:

https://twitter.com/lemonodor/status/1709598048459132976

Finding banks that might be at risk of robbery:

https://twitter.com/lemonodor/status/1716153200750051332

jjwisemanvor 2 Jahren

Bellingcat's Bellingcat OpenStreetMap search is a tool that wraps Overpass Turbo in an easier-to-use interface for the specific task of geolocation based on image and videos: https://osm-search.bellingcat.com

jjwisemanvor 2 Jahren

Here’s a video showing the sort of geolocation task the Bellingcat tool is intended for: https://youtu.be/GqKNKQ02pjY?si=DyAB3YZzl3gJUzT9

WaxProlixvor 2 Jahren

This is awesome, but a lot of the queries I've tried to make seem to fail on OSM's data itself, which is sad. For instance, "coffee shops within a certain distance of fast EV chargers" would be really valuable, but the underlying data just doesn't have that EV data. So it's cool for a lot of stuff, but mostly explicitly streets; the other kinds of nodes all exist but aren't as well fleshed out as they could be (totally understandable).

Doctor_Feggvor 2 Jahren

That’s because we’re all cyclists ;)

dabbervor 2 Jahren

For anyone else curious:

Source code for the backend here: https://github.com/drolbr/Overpass-API

Found from this link listed in the help modal: https://overpass-api.de/ (also links to two additional frontends.)

shpxvor 2 Jahren

I've been asking ChatGPT to write Overpass Turbo queries and it does a good job.

stevagevor 2 Jahren

Yeah, I've had that experience too. Or at least it gets close enough that I can make it work.

The Overpass QL language is the most...bizarrely constructed language I've ever had to use. It's a bit hard to believe that the person who designed the language thought this was a perfectly reasonable statement:

   (._; >;);

Aachenvor 2 Jahren

I read up on the syntax a few years ago and, by learning only a few of the symbols, it actually starts to make a lot of sense and seemed like a nice shorthand.

I then promptly forgot everything and find the syntax very annoying again nowadays... seriously, it's weird remembering having knowledge but not having the knowledge itself anymore

flexagoonvor 2 Jahren

Overpass syntax is one of those things that I just read the documentation for every single time I have to do it. Last time I used it things like ChatGPT didn't exist yet though, so I'll probably use it from now on if I ever have to write an overpass query

stevagevor 2 Jahren

Yeah me too. No way is my brain going to remember any of that between times that I use it.

dylan604vor 2 Jahren

I really don't understand the point of shortcuts where this is the result.

I'm a really big fan of clear and legible code. I'm not at In-n-Out with a secret menu. I'm trying to make changes/updates to code someone else wrote. I don't care how clever they felt they were by using esoteric shortcuts that might make things short and fit on one line. I want to be able to read the code and follow what it does in the least amount of time. Now, if the compiler makes different/better decisions when reading something like that vs much more legible code, we can maybe have a discussion.

maxericksonvor 2 Jahren

Those aren't shortcuts though, it's composition of 3 basic statements. The `(...);` says to gather the results of the statements contained within it into one result. The `._;` is a reference to the previous result. And then the `>;` is an operator that says to go fetch the children of the previous statement. So you are taking the last result and gathering it up together with any children, which is necessary in the OpenStreetMap data model where lines and areas are built up out of other objects.

Depending on the use case, the shortcut is often to add `geom` to the print statement, so instead a line with the above and a line with `out;` you just write `out geom;`.

e-vvor 2 Jahren

I could just upvote your comment (which I did), but I'd like to tell you explicitly that your comment was particularly informative. Thanks!

vloewevor 2 Jahren

I scraped the these websites https://wiki.openstreetmap.org/wiki/Map_features and gave it to OpenAI's Assistant maker -> This made great results in making valid queries

Could even make prompts like "Can you find the buildings closer than 500m to a café?"

Alifatiskvor 2 Jahren

Gotta agree with you, the QL is strange. I tried using the wizard but it kept spinning. I tried asking Bard and ChatGPT but they provided invalid syntax.

This is sad because overpass is extremely powerful tool.

I’ll give the tutorials other commenters linked a try.

e-vvor 2 Jahren

Lucky you! I tried learning with examples generated by ChatGPT and none of them worked at all. I then learned the basics of the API by reading this excellent tutorial: https://osm-queries.ldodds.com/tutorial/, and realised that, in hindsight, ChatGPT's answers weren't even close to being correct.

Raphaellllvor 2 Jahren

we systematically tested this: https://arxiv.org/pdf/2308.16060.pdf

maelitovor 2 Jahren

Where can we use it ? I couldn't find a link. It is closed-source ?

OK, I had a hard time understanding that the link given above is the demo of this paper.

https://overpassnl.schumann.pub.

Very interesting. Quite surprising too : it can find non trivial searches like "bars with darts" ("bars à fléchette" in French), but not easy ones ("fromagerie" or "dermato").

More complex queries did not work : "cafes close to the river" returned nothing, neither did "touristic site close to a pedestrian area" or "touristic streets".

Raphaellllvor 2 Jahren

Yes, it's far from perfect. We currently work on augmenting the prompt with e.g. relevant key-value pairs extracted from https://wiki.openstreetmap.org/wiki/Map_features

zarazasvor 2 Jahren

There is also this: https://overpassnl.schumann.pub/

webel0vor 2 Jahren

It does. The problem is that I have a hard time determining if the output is, indeed, what I was trying to query in the first place!

maelitovor 2 Jahren

Yes. It was way harder with the other LLMs. Except Claude maybe.

lmeyerovvor 2 Jahren

Related: If any OSM GIS engineering enthusiasts, we have a small OSS repo we are looking for an assist on that should be fun for the right person: Spin up a big box, quickly render planet-scale tiles at a decent level (maptiler), and restart as a more reasonably-sized zoom level for serving. We did the core infra/devops, and are looking for someone to help tweak the perf knobs etc. See my profile on how to reach.

Doctor_Feggvor 2 Jahren

Do you really mean maptiler? tilemaker or planetiler will do a better job of “quickly rendering planet-scale tiles”.

mistrial9vor 2 Jahren

please disclose if you are asking for an unpaid intern or volunteer with certain qualifications

lmeyerovvor 2 Jahren

Sponsor, even!

marklitvor 2 Jahren

I built osm_split to extract OSM features into named GPKG files. It makes it easy to pick features by name and drop them into QGIS. https://github.com/marklit/osm_split

RicoElectricovor 2 Jahren

Isn't the point of GPKG that you can bundle multiple named layers in one file?

marklitvor 2 Jahren

Yeah, you can. The file will be huge though and QGIS won't be performant. Somewhere like Tokyo will be ~400 MB in OSM's PBF format but will contain ~1,300 different types of features.

Seeing these listed in alphabetical order in a file explorer makes it easier to track down only what you need to build your map.

It could be you only end up using 150 MB of GPKG files and QGIS, even on an old laptop, would be performant.

ImaCakevor 2 Jahren

There is a QGIS plugin for this API as well which is excellent.

Note that in openstreetmap.com you can zoom in and query items (cursor with question mark icon) to get the feature information that you then use in Overpass. Very useful!

Freak_NLvor 2 Jahren

openstreetmap.org not .com (!)

flexagoonvor 2 Jahren

openstreetmap.com just redirects to the .org domain. There's also just osm.org, which is much easier to type.

westnordostvor 2 Jahren

Regarding data mining tools for OpenStreetMap, there is also a tool with which one can use SPARQL syntax. This means that one could conflate data with wikidata or other data sources easily, too:

QLever: https://qlever.cs.uni-freiburg.de/osm-planet

Actually, there are two. Sophox is around for longer:

Sophox: https://sophox.org/

vloewevor 2 Jahren

Overpass is great!

Published a ShowHN about Atlas.co yesterday, but I also just want to mention that we have a Overpass Turbo OSM integration that enables you to query data directly in a map and then download it as geojson, kml or shp (obv you can also style it and use it in analysis). Might be useful for some

HumblyTossedvor 2 Jahren

Slightly off topic and maybe better as an Ask HN, but here goes.

There are apps for updating OSM that will let me put in the information for the houses in my neighborhood, for example, but that gets tedious. Are there any scripts (python, shell, ?) that would make that easier?

toomuchtodovor 2 Jahren

https://wiki.openstreetmap.org/wiki/Import

> Imports and automated edits should only be carried out by those with experience and understanding of the way the OpenStreetMap community creates maps, and only with careful planning and consultation with the local community. See Import/Guidelines and Automated Edits code of conduct for more information. Imports/automated edits which do not follow these guidelines might be reverted!

Quotvor 2 Jahren

I used StreetComplete when I had an Android phone. It looks for missing data near you, and gives you a really nice interface to input that data.

They are working on an iOS build, and I can't wait to start using it again.

https://github.com/streetcomplete/StreetComplete/

Edit: Also, OSM has a few wiki pages for editing software on different platforms.

- Android_apps_that_can_upload_changes_to_OSM - https://wiki.openstreetmap.org/wiki/Category:Android_apps_th...

- Android_apps_that_can_record_GPS_tracks - https://wiki.openstreetmap.org/wiki/Category:Android_apps_th...

- IOS_software - https://wiki.openstreetmap.org/wiki/Category:IOS_software

- Mobile_editors - https://wiki.openstreetmap.org/wiki/Category:Mobile_editors

Tempest1981vor 2 Jahren

Mentioned here a few months ago: (was trying to remember where I saw it)

How to find a street in 2 minutes [video] (youtube.com)

https://news.ycombinator.com/item?id=36912880

the_g0d_f4thervor 2 Jahren

Actually saved me in an internship where i needed a complete dataset of adresses. This post remindes me that i had a gdoc where i had i lot of ideas for side projects, thanks !

curtisblainevor 2 Jahren

The wizard isn't working for me. Just keeps spinning.

a_gnosticvor 2 Jahren

Spinning is a nice trick.

guwopvor 2 Jahren

Hmmmm. If i want to do larger scale extractions of OSM data into a layer - what's the move? I remember someone on here posted a way to do it

twpvor 2 Jahren

I wrote https://github.com/twpayne/osm-extract for exactly this.

ikawevor 2 Jahren

I typically download a pbf extract, and then use the osmium cli for local filtering.

Then I use ogr2ogr to convert to whatever output format I need.

rony989vor 2 Jahren