Hey HN! As I was collecting urban data for my startup, I decided to create a free and open tool to visualize what we call "Urban Chaos": data points in a city that can indicate activity and use historical trends to detect anomalies (e.g: a big concert next to a planned road paving might create some difficult street conditions).
This is currently at a very early stage, where we're only plotting the data on a map and have curated some historical time series of road traffic and bike traffic to get an idea of what the street conditions look like. We have more datasets that we're going to add to the dashboard, including events (concerts, sports) as well as more road work and outages, and we'll run some spatial analyses to detect strong urban activity.
How it works:
- A lot of data cleaning was needed to get the traffic data from NYC Open Data API (selecting the right segments and matching link ids manually)
- Geometry conversions are required as NYC uses NAD83 projection (EPSG:2263) for a lot of its points, using https://proj.org/ (use proj, do not try to do this yourself)
- Data points are queried according to their update frequency (weather will update minutely, road work will update once a day)
- We use goroutines to run this in a single light binary, on our server
- For the data that we're allowed to store, we store it in a TimescaleDB, giving us a powerful timeseries engine to query from and generate the graphs on the dashboard
- Pure HTML/JS/CSS (I've built a mini css framework for this, since I just couldn't find the right one that was primarily designed for data visualization. If anyone actually likes it and is interested, I'm happy to make it open source). For the maps, we use MapLibre-GL and OpenFreeMap for the tiles (we're planning on hosting our own tiles soon as well) and ToastUI for the charts.
I'd love to get some feedback on this, and if anyone is interested in some new features, or enough people want a specific US city added, feel free to contact us and we'll answer promptly! We're adding new datasets every week for the NYC area, so expect new features pretty soon.
(Full disclosure: this does have a mention of my startup at the bottom)
0 comments
—
Show HN: I built a tool to visualize NYC Urban Chaos · HackerTrans
This is currently at a very early stage, where we're only plotting the data on a map and have curated some historical time series of road traffic and bike traffic to get an idea of what the street conditions look like. We have more datasets that we're going to add to the dashboard, including events (concerts, sports) as well as more road work and outages, and we'll run some spatial analyses to detect strong urban activity.
How it works:
- A lot of data cleaning was needed to get the traffic data from NYC Open Data API (selecting the right segments and matching link ids manually)
- Geometry conversions are required as NYC uses NAD83 projection (EPSG:2263) for a lot of its points, using https://proj.org/ (use proj, do not try to do this yourself)
- Data points are queried according to their update frequency (weather will update minutely, road work will update once a day)
- We use goroutines to run this in a single light binary, on our server
- For the data that we're allowed to store, we store it in a TimescaleDB, giving us a powerful timeseries engine to query from and generate the graphs on the dashboard
Stack:
- Go
- Proj (https://github.com/twpayne/go-proj)
- Pure HTML/JS/CSS (I've built a mini css framework for this, since I just couldn't find the right one that was primarily designed for data visualization. If anyone actually likes it and is interested, I'm happy to make it open source). For the maps, we use MapLibre-GL and OpenFreeMap for the tiles (we're planning on hosting our own tiles soon as well) and ToastUI for the charts.
I'd love to get some feedback on this, and if anyone is interested in some new features, or enough people want a specific US city added, feel free to contact us and we'll answer promptly! We're adding new datasets every week for the NYC area, so expect new features pretty soon.
(Full disclosure: this does have a mention of my startup at the bottom)