New OSM file format: 30% smaller than PBF, 5x faster to import(community.openstreetmap.org)
community.openstreetmap.org
New OSM file format: 30% smaller than PBF, 5x faster to import
https://community.openstreetmap.org/t/new-osm-file-format-30-smaller-than-pbf-5x-faster-to-import/137151
12 comments
My guess is that one of the fundamental differences there would be that KMZ isn't streamable and needs to be fully loaded into memory and then transformed into whatever structure qgis uses internally, but I'm not totally sure about that and haven't used QGIS in a minute. I feel like I've also had bad luck loading KMZ/KML of any reasonable complexity into any other GIS app.
How's geojson of the same data?
How's geojson of the same data?
Sometimes with QGIS the best thing you can do is load up the stuff into Postgres, just orders of magnitude of perf improvements
Tangentially related question for any of you GIS people who might be lurking in this thread:
Can anyone recommend me a method of meshing LIDAR point clouds? The sparseness of the data on building walls & other near-vertical surfaces combined with a lack of point normals leads to degenerate solutions with all the common approaches (poisson/ball pivot/vcg in meshlab) not to mention extremely slow perf. Tree canopies and overhanging parapets make a simple heightmap approach less-than desirable (though ultimately acceptable if I can't find anything better). I'm trying to turn 90 billion lidar points into maybe 30-50 million triangles, hopefully without spending months developing a custom pipeline.
Can anyone recommend me a method of meshing LIDAR point clouds? The sparseness of the data on building walls & other near-vertical surfaces combined with a lack of point normals leads to degenerate solutions with all the common approaches (poisson/ball pivot/vcg in meshlab) not to mention extremely slow perf. Tree canopies and overhanging parapets make a simple heightmap approach less-than desirable (though ultimately acceptable if I can't find anything better). I'm trying to turn 90 billion lidar points into maybe 30-50 million triangles, hopefully without spending months developing a custom pipeline.
https://3dbag.nl/ might be worth a try. This project reconstructed and maintains building models of 11 million buildings in the Netherlands.
It combines airborne LiDAR and building footprints, it's OS (https://github.com/3DBAG) with the reconstruction pipeline here: https://github.com/3DBAG/roofer.
It combines airborne LiDAR and building footprints, it's OS (https://github.com/3DBAG) with the reconstruction pipeline here: https://github.com/3DBAG/roofer.
I think Meshroom can use LIDAR data as an input now? I used it years ago for photogrammetry and camera tracking for some VFX work and it's an incredibly solid suite of open source tools for these types of tasks.
Does this use the new OSM data model?
https://media.jochentopf.com/media/2022-08-15-study-evolutio...
https://github.com/osmlab/osm-data-model
https://blog.openstreetmap.org/2023/01/04/reminder-call-for-...
Resolving the coordinates to node references in current data model is such a nuisance as it's slow and requires lots of RAM.
https://media.jochentopf.com/media/2022-08-15-study-evolutio...
https://github.com/osmlab/osm-data-model
https://blog.openstreetmap.org/2023/01/04/reminder-call-for-...
Resolving the coordinates to node references in current data model is such a nuisance as it's slow and requires lots of RAM.
does the tiling approach have any trade-offs on random access or is lookup performance comparable to PBF once loaded?
Aside from OSM specifics, performance friendly formats for spatial data that support spatial indexing can make huge impact on usability and productivity of applications. e.g. trying to view a large dataset in QGIS that has been saved as KMZ (zipped XML) can make QGIS basically hang for minutes, while the same dataset saved as something like flatgeobuf [1] can be loaded instantly.
[1] https://flatgeobuf.org/