Pirate Weather(pirateweather.net)
pirateweather.net
Pirate Weather
http://pirateweather.net/en/latest/
92 comments
Previous discussion: https://news.ycombinator.com/item?id=34329988
Yup, it's great but that previous thread spent 20 hours on HN's front page just a couple months ago, which makes this one a dupe (see https://news.ycombinator.com/newsfaq.html).
Actually we put the current post in the second-chance pool (https://news.ycombinator.com/pool, explained at https://news.ycombinator.com/item?id=26998308) — we try not to do that for dupes but sometimes we do anyhow. Sorry—it was just a good HN post!
Actually we put the current post in the second-chance pool (https://news.ycombinator.com/pool, explained at https://news.ycombinator.com/item?id=26998308) — we try not to do that for dupes but sometimes we do anyhow. Sorry—it was just a good HN post!
I love the attitude of this guy. From the "Background" section:
> "This project started from two points: as part of my PhD, I had to become very familiar with working with NOAA forecast results. Separately, an old tablet set up as a "Magic Mirror,” and was using a weather module that relied on the Dark Sky API, as well as my Home Assistant setup. So when I heard that it was shutting down, I thought, "I wonder if I could do this.” Plus, I love learning new things, and I had been looking for a project to learn Python on, so this seemed like the perfect opportunity! Spoiler alert, but it was way more difficult than I thought, but learned a lot throughout the process, and I think the end result turned out really well!"
> "This project started from two points: as part of my PhD, I had to become very familiar with working with NOAA forecast results. Separately, an old tablet set up as a "Magic Mirror,” and was using a weather module that relied on the Dark Sky API, as well as my Home Assistant setup. So when I heard that it was shutting down, I thought, "I wonder if I could do this.” Plus, I love learning new things, and I had been looking for a project to learn Python on, so this seemed like the perfect opportunity! Spoiler alert, but it was way more difficult than I thought, but learned a lot throughout the process, and I think the end result turned out really well!"
To me it's a bit worrying that someone who is still is a student, builds such a complicated and hard to maintain architecture (https://github.com/alexander0042/pirateweather/raw/main/docs...). Maybe it's complicated on purpose, to educate themselves on how to create infrastructure like that, but for a production environment, you want to keep things as simple as possible.
I'd have opted for a setup where basically every single "service" in that architecture diagram is one function in a monolith program. Easier to maintain, more performant, easier to deploy and basically better in every single way except you can't individually scale components. But at the scale that the project currently is, I don't think that's needed for a long time, nor is it hard to scale one machine if needed.
But again, if it's just a educational exercise in "web scale architecture", then it of course is OK. But I wouldn't want other students to see something like that and think that it's necessarily to create such a monster for a relatively simple "moving data around and wrangle it a bit" program.
I'd have opted for a setup where basically every single "service" in that architecture diagram is one function in a monolith program. Easier to maintain, more performant, easier to deploy and basically better in every single way except you can't individually scale components. But at the scale that the project currently is, I don't think that's needed for a long time, nor is it hard to scale one machine if needed.
But again, if it's just a educational exercise in "web scale architecture", then it of course is OK. But I wouldn't want other students to see something like that and think that it's necessarily to create such a monster for a relatively simple "moving data around and wrangle it a bit" program.
Yeah, 13 years ago I wrote a small signup app for tracking pick-up hockey skates I run regularly.
It's a pretty cool system, you have your roster of players and setup skates, and it automatically emails everyone x number of days before the skate. You can see who signs up, pay online, etc and if a goalie who's signed up cancels, the 'Skate Operator' is emailed immediately to take care of that (it's a big deal to not have two goalies). It also has an automatic waitlist, and if someone cancels before the skate, and you're the first person on the waitlist, it moves you to the end of the list and emails you to let them know you're in for the skate. It also sends out a reminder X hours before the skate, day of, etc.
After I started using it for my own skates, a few rink managers contacted me to discuss using it for their rink. I met with 2 different groups / municipalities (because most rinks are owned by the city they're in), and it never went any farther than those meetings. They didn't want to use a system by some guy that couldn't maintain it, wouldn't keep it going for more than a year or 3.
13 years later and I'm writing a new version and have run something like 11k skates in that time. There is one small, privately owned rink near my house, has used it the entire time without issue. I've got a few other friends/individuals that use it, too.
It is possible for a single person to create and maintain something purely because they need it.
It's a pretty cool system, you have your roster of players and setup skates, and it automatically emails everyone x number of days before the skate. You can see who signs up, pay online, etc and if a goalie who's signed up cancels, the 'Skate Operator' is emailed immediately to take care of that (it's a big deal to not have two goalies). It also has an automatic waitlist, and if someone cancels before the skate, and you're the first person on the waitlist, it moves you to the end of the list and emails you to let them know you're in for the skate. It also sends out a reminder X hours before the skate, day of, etc.
After I started using it for my own skates, a few rink managers contacted me to discuss using it for their rink. I met with 2 different groups / municipalities (because most rinks are owned by the city they're in), and it never went any farther than those meetings. They didn't want to use a system by some guy that couldn't maintain it, wouldn't keep it going for more than a year or 3.
13 years later and I'm writing a new version and have run something like 11k skates in that time. There is one small, privately owned rink near my house, has used it the entire time without issue. I've got a few other friends/individuals that use it, too.
It is possible for a single person to create and maintain something purely because they need it.
How is it hosted?
AWS virtual machine (I have an old school windows service processing the emails; the rewrite will use a new approach).
> To me it’s a bit worrying that someone who is still is a student, builds such a complicated and hard to maintain architecture
To me, it is kind of expected. “Perfection is achieved, not when there is nothing more to add, but when there is nothing left to take away,” sure, but perfection – or even distantly approaching it – is the product of expertise and practice.
To me, it is kind of expected. “Perfection is achieved, not when there is nothing more to add, but when there is nothing left to take away,” sure, but perfection – or even distantly approaching it – is the product of expertise and practice.
[deleted]
I don't know if a person pursuing a PhD is a student
They're "graduate students", usually. Still have a student status at most universities.
I'm guessing they are studying? That would make them a student, as far as I know. But English is not my first language, maybe there is multiple meanings of student and not all of them fit someone who is "pursuing a PhD".
This whole comment reads so weird. Why do you or anyone else care how complicated someone else’s architecture is? Why is it only “OK” if it’s an educational exercise?
Are you going to be maintaining it? Are you going to even be using it? Why does it matter to you what someone else did in their spare time for their own amusement? Or even to make a commercial product out of it? Either way, it’s not your architecture to backseat architect, let people do what they want.
Are you going to be maintaining it? Are you going to even be using it? Why does it matter to you what someone else did in their spare time for their own amusement? Or even to make a commercial product out of it? Either way, it’s not your architecture to backseat architect, let people do what they want.
It may be helpful to assume good faith. I think the GP is making the point that if this guy who is "a student" chose that diagram as his architecture for a hobby / learning project, then it's worrying that somewhere along the way we may have inadvertently created the expectation on newbies that this is The Way™
Said differently, it's worrying that people decide to overengineer architectures like that before there's any real need. It's a sysops version of the early optimization problem.
Ten years ago, the student would have probably built something much simpler with a couple cron jobs on a simple EC2 instance, maybe celery/rabbitmq as a message queue if he was feeling fancy and called it a day
Said differently, it's worrying that people decide to overengineer architectures like that before there's any real need. It's a sysops version of the early optimization problem.
Ten years ago, the student would have probably built something much simpler with a couple cron jobs on a simple EC2 instance, maybe celery/rabbitmq as a message queue if he was feeling fancy and called it a day
Any student’s production application can assumed to be educational. I think this is even true for a mid level engineer.
I mean, they are very plainly not concerned about the OP or “state of architecting” but that this successful project coming from a student is going to inspire other students to copy the architecture shown here.
Explicitly,
> A bunch of people take information they come across on the internet and think that's the only way of doing things, so if someone comes across the post, looks at the architecture they might think building it with so many components is a required thing, not just optional for really big projects.
> So basically, I want to save myself and others time.
This seems like not only ridiculous critique, but also just plainly not the author’s problem what others choose to do with his open source project.
> Said differently, it's worrying that people decide to overengineer architectures like that before there's any real need. It's a sysops version of the early optimization problem
And why exactly is it yours or other commenter’s problem what people choose to do with their own time? How does it in any way shape or form impact you what this other person chose to do with their project that you will never maintain, contribute to, or likely even use?
Explicitly,
> A bunch of people take information they come across on the internet and think that's the only way of doing things, so if someone comes across the post, looks at the architecture they might think building it with so many components is a required thing, not just optional for really big projects.
> So basically, I want to save myself and others time.
This seems like not only ridiculous critique, but also just plainly not the author’s problem what others choose to do with his open source project.
> Said differently, it's worrying that people decide to overengineer architectures like that before there's any real need. It's a sysops version of the early optimization problem
And why exactly is it yours or other commenter’s problem what people choose to do with their own time? How does it in any way shape or form impact you what this other person chose to do with their project that you will never maintain, contribute to, or likely even use?
Again, I think you're focused on this one instance and the GP was saying "if this guy chose this approach, countless more people are making similar decisions", which as a trend in the long run could make it bad "for everyone" (himself and others)
You may or may not disagree, but it's arguably a worthwhile comment. Possibly more so than your comment on the merits of someone else's comment... which just derails the conversation into this meta-discussion
> And why exactly is it yours or other commenter’s problem what people choose to do with their own time? How does it in any way shape or form impact you what this other person chose to do with their project that you will never maintain, contribute to, or likely even use?
Critiquing technology referenced in submitted posts is what we do for fun around here...
You may or may not disagree, but it's arguably a worthwhile comment. Possibly more so than your comment on the merits of someone else's comment... which just derails the conversation into this meta-discussion
> And why exactly is it yours or other commenter’s problem what people choose to do with their own time? How does it in any way shape or form impact you what this other person chose to do with their project that you will never maintain, contribute to, or likely even use?
Critiquing technology referenced in submitted posts is what we do for fun around here...
I am replying to a person’s comment so, yes, of course I’m “focused on this one instance” and replying to what was said. That is, literally, the purpose of replies.
I think the critique was not constructive, was dismissive about an entire project, backseat architected, and did all that with absolutely no context on the requirements, limitations, and decisions made.
In short, it’s a shit critique and I called it as such.
I think the critique was not constructive, was dismissive about an entire project, backseat architected, and did all that with absolutely no context on the requirements, limitations, and decisions made.
In short, it’s a shit critique and I called it as such.
His comment was valuable and fostered discussion.
He's allowed an opinion especially one that's valid.
It was constructive and explained how he would want to see it structured.
He did it the right way.
Where as your comment is unconstructive. Doesn't foster discussion.
If the only point you're trying to make is to ask someone to stay quiet why not leave it to the mods or even silently downvote and move on?
He's allowed an opinion especially one that's valid.
It was constructive and explained how he would want to see it structured.
He did it the right way.
Where as your comment is unconstructive. Doesn't foster discussion.
If the only point you're trying to make is to ask someone to stay quiet why not leave it to the mods or even silently downvote and move on?
He completely dismissed the entirety of their architecture as too bloated without any sort of context on the decisions that brought it here.
There was no constructive criticism, no “you should do this differently” it was “this is worryingly large and you should feel bad if this isn’t an educational project”.
If this is “the right way” then I don’t want to be right, because it’s a shit way that lacks empathy or tact.
There was no constructive criticism, no “you should do this differently” it was “this is worryingly large and you should feel bad if this isn’t an educational project”.
If this is “the right way” then I don’t want to be right, because it’s a shit way that lacks empathy or tact.
> dismissed the entirety of their architecture
No he didn't. He had a caveat.
> no “you should do this differently”
That's not the case..he clearly says this is how I would do this differently.
None of this matters. Arguing about arguing is futile.
No he didn't. He had a caveat.
> no “you should do this differently”
That's not the case..he clearly says this is how I would do this differently.
None of this matters. Arguing about arguing is futile.
“Unless this is an educational project” is not an appropriate caveat. Especially about a project that was clearly not submitted as an educational one.
> Background
> This project started from two points: as part of my PhD, I had to become very familiar with working with NOAA forecast results (https://orcid.org/0000-0003-4725-3251). Separately, an old tablet set up as a "Magic Mirror,” and was using a weather module that relied on the Dark Sky API, as well as my Home Assistant setup. So when I heard that it was shutting down, I thought, "I wonder if I could do this.” Plus, I love learning new things (http://alexanderrey.ca/), and I had been looking for a project to learn Python on, so this seemed like the perfect opportunity! Spoiler alert, but it was way more difficult than I thought, but learned a lot throughout the process, and I think the end result turned out really well!
Seems perfectly fine to assume that this might be purely an exercise in order to learn as much as possible, rather than to build something that is trying to be as cost-effective and performant as possible.
> This project started from two points: as part of my PhD, I had to become very familiar with working with NOAA forecast results (https://orcid.org/0000-0003-4725-3251). Separately, an old tablet set up as a "Magic Mirror,” and was using a weather module that relied on the Dark Sky API, as well as my Home Assistant setup. So when I heard that it was shutting down, I thought, "I wonder if I could do this.” Plus, I love learning new things (http://alexanderrey.ca/), and I had been looking for a project to learn Python on, so this seemed like the perfect opportunity! Spoiler alert, but it was way more difficult than I thought, but learned a lot throughout the process, and I think the end result turned out really well!
Seems perfectly fine to assume that this might be purely an exercise in order to learn as much as possible, rather than to build something that is trying to be as cost-effective and performant as possible.
If someone posts their architecture diagram on Hacker News, it's fair game for feedback? Anyone is welcome to ignore said feedback.
As the adage goes: Suppose you have a problem, and you say I know, I'll use microservices! Now you have N problems.
As the adage goes: Suppose you have a problem, and you say I know, I'll use microservices! Now you have N problems.
I kinda agree but it’s also not that complicated. It’s a critique of a craft many of us share. It’s natural for an engineer to have an opinion on the design and share his opinion with others. He’s also not being mean about it or anything.
I would venture as far as saying nearly every personal project built to scale is over engineered.
I would venture as far as saying nearly every personal project built to scale is over engineered.
I think I already elaborated on why? A bunch of people take information they come across on the internet and think that's the only way of doing things, so if someone comes across the post, looks at the architecture they might think building it with so many components is a required thing, not just optional for really big projects.
I'm personally dealing with infrastructure at loads of different places, and the amount of engineers who over-complicate their infrastructure pains me, they all go for a distributed architecture before the products they are building even have any users, because they've read bunch of blog posts about building it that way.
It sucks to spend so much time and effort on maintaining and improving something when it's not really needed in the first place. Like using a Ferrari to deliver food in a traffic-heavy city, instead of just having a moped.
So basically, I want to save myself and others time.
I'm personally dealing with infrastructure at loads of different places, and the amount of engineers who over-complicate their infrastructure pains me, they all go for a distributed architecture before the products they are building even have any users, because they've read bunch of blog posts about building it that way.
It sucks to spend so much time and effort on maintaining and improving something when it's not really needed in the first place. Like using a Ferrari to deliver food in a traffic-heavy city, instead of just having a moped.
So basically, I want to save myself and others time.
Gotta love gate keeping architecture in the interest of “saving others time”.
If someone takes one look at one project and decides to copy it’s architecture word for word, then it’s on them. Not on the person who built an open source project, for whatever reason, and put it out there for the world.
It’s not your architecture to architect. If you want to build your own service to show best practices, then go for it. You weren’t there to make decisions, you don’t know of the reasoning behind them, and it seems ludicrous to backseat architect with context, let alone without any amount of it.
If someone takes one look at one project and decides to copy it’s architecture word for word, then it’s on them. Not on the person who built an open source project, for whatever reason, and put it out there for the world.
It’s not your architecture to architect. If you want to build your own service to show best practices, then go for it. You weren’t there to make decisions, you don’t know of the reasoning behind them, and it seems ludicrous to backseat architect with context, let alone without any amount of it.
I don't know what your expertise is exactly, but if someone new in your subject area intentionally made things more complicated than necessary for the scope of the project and publish it as a public resource for others to take inspiration from and learn from, wouldn't you at least have some sort reaction against it?
See it as wanting to advance my own trade by providing opinion on the current "everything should be a serverless function" trends in the hope of hopefully be able to move the trade into a different direction.
See it as wanting to advance my own trade by providing opinion on the current "everything should be a serverless function" trends in the hope of hopefully be able to move the trade into a different direction.
> I don't know what your expertise is exactly, but if someone new in your subject area intentionally made things more complicated than necessary for the scope of the project and publish it as a public resource for others to take inspiration from and learn from, wouldn't you at least have some sort reaction against it?
If I have no affiliation, or plan to affiliate with a project, why would I give a shit about what they do? It’s their project to figure out.
Did they ask you for an architecture review? No. Then don’t backseat architect. It’s as simple as that. Especially when your critique is “you’re doing it all wrong and you should feel bad if it’s not an educational exercise”.
If I have no affiliation, or plan to affiliate with a project, why would I give a shit about what they do? It’s their project to figure out.
Did they ask you for an architecture review? No. Then don’t backseat architect. It’s as simple as that. Especially when your critique is “you’re doing it all wrong and you should feel bad if it’s not an educational exercise”.
Do you have any affiliation to me or plan to affiliate with me or my projects? Then why you give a shit about what I write?
I'm assuming we two have at least one thing in common, we're here on HN to discuss things we come across, both submission articles and comments, since we're having a conversation right now. That's how this works.
So I came across something that I have feeling and opinion about, and that led me to share my opinion about it, as I care about my trade. You're welcome to ignore things even if you have opinions about it, but to request others to do the same in a forum that is meant for discussion, feels a bit unproductive.
I'm assuming we two have at least one thing in common, we're here on HN to discuss things we come across, both submission articles and comments, since we're having a conversation right now. That's how this works.
So I came across something that I have feeling and opinion about, and that led me to share my opinion about it, as I care about my trade. You're welcome to ignore things even if you have opinions about it, but to request others to do the same in a forum that is meant for discussion, feels a bit unproductive.
> If I have no affiliation, or plan to affiliate with a project, why would I give a shit about what they do?
We critique whatever gets shared on HN regardless of our affiliation with it.
In fact, why was this shared here, if not to ask for discussion and opinions?
We critique whatever gets shared on HN regardless of our affiliation with it.
In fact, why was this shared here, if not to ask for discussion and opinions?
What exactly do you think is the purpose of a discussion forum like this one?
The OP states he wanted it to scale and cost-wise using AWS on-demand resources is much cheaper than running multiple EC2 instances which are over-provisioned to begin with. AWS functions and EventBridge is the same thing as CRON and a monolith. LOL why waste a full VM when you could pay pennies on the same system. Obviously vendor lock-in a bit, but who cares! lol ridiculous.
[EDIT]: Removed comment about user.
[EDIT]: Removed comment about user.
From the guidelines: https://news.ycombinator.com/newsguidelines.html
When disagreeing, please reply to the argument instead of calling names. "That is idiotic; 1 + 1 is 2, not 3" can be shortened to "1 + 1 is 2, not 3."
When disagreeing, please reply to the argument instead of calling names. "That is idiotic; 1 + 1 is 2, not 3" can be shortened to "1 + 1 is 2, not 3."
I am sorry. Fixed
Not sure what you think about this, but I'd wager that this fellow posted his detailed architecture diagram for transparency and maybe even to solicit some constructive criticism or feedback.
It sounds like you'd be well positioned to offer some feedback, but I'd recommend against that without your practicing some empathy for the stage he's in with his education.
It sounds like you'd be well positioned to offer some feedback, but I'd recommend against that without your practicing some empathy for the stage he's in with his education.
This isn't a complicated architecture. It looks like a pretty small and simple cloud deployment. The EC2 looks a bit out of place, but I'm assuming there were technical reasons it couldn't be a lambda instead. Apart from EC2, all of the compute is consumption based so you're just paying for actual utilization. This is a far more efficient approach to building in the cloud than just throwing a monolith on a VM and calling it a day.
I've built lots of things like this working at AWS. I probably would have used ECS instead of Fargate, S3 and S3 events instead of EFS, and CW Events instead of EventBridge for simplicity sake, but the "gist" of what he is trying to do is pretty sane and this is what a real-world combination of building blocks would look like. If it were me I'd also try to find a way to modify WGRIB2 to run entirely in Lambda and get the costs down.
Most architecture look complicated when include everything. He includes the load balancer. It would look simpler as VM but then all the complexity is the things running on VM.
Also, much of the complexity is from pulling from data sources. Lots of apps are just the web site and database, but weather app needs the complexity. I bet EventBridge has some important purpose.
Also, much of the complexity is from pulling from data sources. Lots of apps are just the web site and database, but weather app needs the complexity. I bet EventBridge has some important purpose.
Agreed. Guy above doesn't know how to save money or use resources efficiently in the cloud...
This reads as someone who has never designed a cloud architecture. If he wanted to use a VM he might as well use a VPS. The whole point seems to utilize the resources AWS provides... not some basic VM monolith. That's child's play.
[EDIT]: He's a Phd for chrissakes!
[EDIT]: He's a Phd for chrissakes!
Yes, use a VPS or dedicated instance, one machine is usually enough. This is exactly what I'm saying :)
And yes, if the whole point is to utilize AWS resources, then of course a architecture like that is fine, as it's a learning exercise (to learn how to utilize resources). My comment includes that as well.
As far as I can tell, the author is studying at a university, that makes them a student in my mind. Maybe they do other things than just studying, I didn't look into it very far, hence the "maybes" and other caveats in my initial comment.
And yes, if the whole point is to utilize AWS resources, then of course a architecture like that is fine, as it's a learning exercise (to learn how to utilize resources). My comment includes that as well.
As far as I can tell, the author is studying at a university, that makes them a student in my mind. Maybe they do other things than just studying, I didn't look into it very far, hence the "maybes" and other caveats in my initial comment.
Someone quoted note where he says using AWS for scalability and reliability. Would single VPS have survived being on Hacker News? Would he need to make new VPS to handle the load? I bet AWS scaled up without problems. Also important, it will scale down when spike is done and will only cost what was used.
AWS is more reliable, and from this architecture, probably doesn't have to do anything. If the VPS dies, then the site is down until can build new VM. Or have to run two of them at twice the cost. More since he has to setup database or shared storage.
That might be overkill for this site, but for some upfront work, he gets something that needs less maintenance. If going to learn, then should do it the right way. He may be scaling for the future, but it is tradeoff to design now instead of later.
AWS is more reliable, and from this architecture, probably doesn't have to do anything. If the VPS dies, then the site is down until can build new VM. Or have to run two of them at twice the cost. More since he has to setup database or shared storage.
That might be overkill for this site, but for some upfront work, he gets something that needs less maintenance. If going to learn, then should do it the right way. He may be scaling for the future, but it is tradeoff to design now instead of later.
[deleted]
[deleted]
Apache2 license as well.
No need to worry about them selling out.
No need to worry about them selling out.
I'm not familiar with AWS, but I'm not sure if everything needed to run your own version on your own AWS account is open source?
Lot of people hating on his 'architecture' that are a bunch of 'architecture astronauts'.
This is a kid that saw a need and went out and did it. His PHD is “currently completing my PhD in Water Resources Engineering”. So, this work is pretty amazing as a hobby, a side line to his PhD.
Isn’t that what Open Source is all about, jumping in and filling a need, not “waiting until I’m a grizzled 30 year veteran of the corporate IT wars, then deciding to do something with a perfect architecture”.
This is a kid that saw a need and went out and did it. His PHD is “currently completing my PhD in Water Resources Engineering”. So, this work is pretty amazing as a hobby, a side line to his PhD.
Isn’t that what Open Source is all about, jumping in and filling a need, not “waiting until I’m a grizzled 30 year veteran of the corporate IT wars, then deciding to do something with a perfect architecture”.
It’s hacker news. An engineering audience is going to have an opinion. The author can learn from the feedback. By all means, it would be rude to criticize OP for his design choices but most of what I see doesn’t fit that category.
For the record I agree that his accomplishment with this project is impressive.
For the record I agree that his accomplishment with this project is impressive.
I see no hate, just a critique.
("Architecture astronauts" don't argue for simpler architectures, but instead want to build general, solves-everything kind of overengineered monsters. I don't see this in the critiques)
("Architecture astronauts" don't argue for simpler architectures, but instead want to build general, solves-everything kind of overengineered monsters. I don't see this in the critiques)
I don't see any critiques.
It's the second topmost thread, start with this comment: https://news.ycombinator.com/item?id=35342160
I think the critique is pretty mild and reasonable, regardless of whether you agree with it or not, and it's definitely not "hate".
I think the critique is pretty mild and reasonable, regardless of whether you agree with it or not, and it's definitely not "hate".
Ahh, thank you for the direct link. Part of the confusion with the duped/notduped post. I don't think OP really meant "hate" with the expression "hating on" but it's still easily taken as harsh.
[deleted]
In conjunction with Pirate Weather, to consume/visualize the data I've been using:
https://merrysky.net/
and
https://briefsky.app/
https://merrysky.net/
and
https://briefsky.app/
On android I use NativeAlpha for setting up Briefsky as "app" that opens in its own Android Webview "app" rather than a browser. It's almost like DarkSky minus the notifications.
https://github.com/cylonid/NativeAlphaForAndroid
https://github.com/cylonid/NativeAlphaForAndroid
When I am in the Backcountry and need forecasts for anywhere in the world, there is no data source more reliable than the Norwegian weather service. Their whole globe model is extremely accurate to the specific mountain or pass you are concerned about, with reliable mm and hour precision forecasts. In New Zealand it was way more accurate than any NZ sources I consulted.
They also have many (free) api's for whole world weather. If your life depends on it, they are my recommendation.
https://developer.yr.no/
They also have many (free) api's for whole world weather. If your life depends on it, they are my recommendation.
https://developer.yr.no/
How do you access it in the backcountry?
This is fun actually. Once a week or so when I have connection, I load up the tabs I need for the next week in chrome, i.e. locations ~25km apart on my path. Then, when I'm on a mountaintop (usually at least 3-4 times a day when you are backpacking), I try to refresh them until I've updated the forecasts.
If it loads, I take a screenshot and stop trying for the day.
At night I review the screenshots and mark them up.
After my last thru hike I vowed to turn this into an automatic app that would turn on the radio for these polls at certain altitudes based upon geospatial calculations and then turn off the radio automatically after acquiring a daily update.. but alas I never did.
If it loads, I take a screenshot and stop trying for the day.
At night I review the screenshots and mark them up.
After my last thru hike I vowed to turn this into an automatic app that would turn on the radio for these polls at certain altitudes based upon geospatial calculations and then turn off the radio automatically after acquiring a daily update.. but alas I never did.
> Weather forecasts are primarily found using models run by government agencies, but the outputs aren't easy to use or in formats built for the web.
Maybe, but the Norwegian government does provide worldwide data via API: https://developer.yr.no/
Maybe, but the Norwegian government does provide worldwide data via API: https://developer.yr.no/
I love this resource but, I will say it is 1km step for Nordic countries and 2.5km step for the rest of the world [0].
[0]: https://api.met.no/doc/locationforecast/FAQ
[0]: https://api.met.no/doc/locationforecast/FAQ
I got a kick out of the fact their base data structure is GeoJSON. I just.. wasn't expecting that
https://api.met.no/weatherapi/locationforecast/2.0/compact?l...
https://api.met.no/weatherapi/locationforecast/2.0/compact?l...
> Before going any farther, I wanted to add a link to sign up and support this project! Running this on AWS means that it scales beautifully and is much more reliable than if I was trying to host this, but also costs real money.
https://github.com/alexander0042/pirateweather/raw/main/docs...
That looks like a pretty complex architecture.
I wonder if instead of financial support, the author could use help figuring out a more cost-effective solution?
https://github.com/alexander0042/pirateweather/raw/main/docs...
That looks like a pretty complex architecture.
I wonder if instead of financial support, the author could use help figuring out a more cost-effective solution?
It's not clear that this isn't the more cost-effective solution. Without diving into the pricing of each component shown, I suspect there may be some cost optimizations to make (for one example, I had a hard time discerning from the write-up what the EC2 instance was doing...something related to more weather data collection I think?) but this is approximately "cloud-native" in approach, and the way to minimize costs in the cloud is to design your application to be cloud-native and use managed services and provider-specific capabilities where they make sense (and also tend to offer very low/free per-use pricing). Does that increase vendor lock-in? Definitely. It's a trade-off that needs to be considered for each application. Using the cloud to run a bunch of static VMs is an anti-pattern and a big factor in driving costs through the roof. (If you need a single server, a VPS is a very good approach.)
The architecture is complex from the standpoint of having a lot of icons and arrows, but if it was running on a single server would it be _that_ much less complex? At this level of architectural diagram you probably condense most of these service icons down to one or two icons. But drill down further and you still have storage, cron jobs, an API endpoint with throttling/quota logic, CDN, a shared filesystem, the need to access the S3 buckets with weather data that AWS is providing from its Open Data Registry, plus (ideally) fine-grained permissions to lock everything down. Could you run all of this on a single server in a VPS? Yes! Could you run it on a vendor-agnostic K8s cluster? Yes! Would those options automatically be less expensive? Not necessarily.
The architecture is complex from the standpoint of having a lot of icons and arrows, but if it was running on a single server would it be _that_ much less complex? At this level of architectural diagram you probably condense most of these service icons down to one or two icons. But drill down further and you still have storage, cron jobs, an API endpoint with throttling/quota logic, CDN, a shared filesystem, the need to access the S3 buckets with weather data that AWS is providing from its Open Data Registry, plus (ideally) fine-grained permissions to lock everything down. Could you run all of this on a single server in a VPS? Yes! Could you run it on a vendor-agnostic K8s cluster? Yes! Would those options automatically be less expensive? Not necessarily.
What specifically is too complex?
Probably this whole thing could run on a single $10/mo DigitalOcean instance in a single monolith, with an appropriate caching CDN in front of it.
That's fantastic. Is there any one piece specfically that might run on that?
The author discusses using AWS to get free in network transfers since the source material is published there by the guvment. In your analysis how would you offset that incresed cost by moving off network?
It would also be interesting to try this all on a $10/month commodity hosting and then open it up for user facing apps such as a darksky type service.
The author discusses using AWS to get free in network transfers since the source material is published there by the guvment. In your analysis how would you offset that incresed cost by moving off network?
It would also be interesting to try this all on a $10/month commodity hosting and then open it up for user facing apps such as a darksky type service.
Someone please use this data to make a clone of Weather Spark. Weather Spark came way before Dark Sky. As an ameature professional weather nerd I feel it was the best weather site ever! You could do so much with the data in one unified dashboard style screen! Especially with the integrated historical datasets.
Examples: http://3.bp.blogspot.com/-B1LupTaXvSo/TcBNECP0A0I/AAAAAAAAEi...
https://flowingdata.com/wp-content/uploads/2011/03/Weathersp...
https://media.licdn.com/dms/image/C5112AQEuHVOGOd-5fQ/articl...
Examples: http://3.bp.blogspot.com/-B1LupTaXvSo/TcBNECP0A0I/AAAAAAAAEi...
https://flowingdata.com/wp-content/uploads/2011/03/Weathersp...
https://media.licdn.com/dms/image/C5112AQEuHVOGOd-5fQ/articl...
Wow. Thanks for the blast from the past. WeatherSpark was the best! Forgot all about this.
This could be a really interesting experiment in Open Source Architecture. The design is currently.... complicated. I wonder what kind of process we could use to unpack and map out the entire architecture, such that average contributors could make "PRs" about changes to the architecture, in order to slowly simplify and improve the architecture over time.
I am imagining a process that involves programmatic architectural diagrams, graphs, estimates of resources, DAGs, documentation, links to source code, etc. Almost like a sub-project, where the architecture itself becomes a project. The end result of that project could tell you rough estimates of cost, performance, complexity, and let you visualize changes. And of course it would become the "map" by which you discover pieces of the system, the place all new devs start to learn about the system or find documentation.
(I am not saying the author's system needs to change, I'm just speculating about how to organize a collaborative effort, if change was wanted)
I am imagining a process that involves programmatic architectural diagrams, graphs, estimates of resources, DAGs, documentation, links to source code, etc. Almost like a sub-project, where the architecture itself becomes a project. The end result of that project could tell you rough estimates of cost, performance, complexity, and let you visualize changes. And of course it would become the "map" by which you discover pieces of the system, the place all new devs start to learn about the system or find documentation.
(I am not saying the author's system needs to change, I'm just speculating about how to organize a collaborative effort, if change was wanted)
I've been using this API (and sponsor its creator) and it's really great. Highly recommend it.
Something I miss from Dark Sky that Apple inexplicably dropped failed to include in its Weather app: the ability to visualize changing temperatures across a region.
The radar map allows you to scroll forward in time, but not the temperature map.
Anyone have suggestions for apps/sites that offer that view?
The radar map allows you to scroll forward in time, but not the temperature map.
Anyone have suggestions for apps/sites that offer that view?
Checkout Saildrone, it allows you to do that, and the visualizations are very responsive/fluid.
Thanks.
This has to be, by far, the most detailed “look at our cool app” page I’ve seen that doesn’t actually include a download link, or indeed any description at all of how to find it.
https://www.saildrone.com/forecast
This has to be, by far, the most detailed “look at our cool app” page I’ve seen that doesn’t actually include a download link, or indeed any description at all of how to find it.
https://www.saildrone.com/forecast
windy.com is a good one for that.
I used to go to windy.com as an alternative to weather.gov but they are moving in the direction of subscriptions. Currently they allow a couple weeks forecast. No idea if they will shorten that to a week to get more subs.
Thanks!
Update: neither their site nor their app allow scrubbing. Boo.
Update: neither their site nor their app allow scrubbing. Boo.
Yeah, they do actually. There's a bar at the bottom with the time on it.
Is there a free API for historical data? I wanted to answer questions like: What was the weather like on this day in last x years, or what is the average in some place in a given month based on last x years?
You can try my API: https://open-meteo.com/en/docs/historical-weather-api
The link should be edited to use https:// since the site does not have a 301 enabled on port 80.
I love this, I played with grib2 files a while back to build a surf forecasting widget (and a surf forecast API!). It used now-deprecated libraries and it was a major pain to wrap my head around how everything worked.
The sign-up process is flawed. “Apiable” asks me for an Email address, first name, last name, and password. But on the login page it asks me for username and password. I never set a username.
I've never found the govt provided API's all that difficult.
is it true that let's encrypt will not run your domain if it has a blacklisted word? I imagine "pirate" be it and the reason why this site is not using SSL ?
The site do have SSL, they just don't have the recommended redirect from http to https.
They are using Cloudflare. There is a switch in the settings to toggle on HTTP -> HTTPS redirect which they need to toggle.
I haven't heard of that - they blacklist certain high profile domains, but I don't think they blacklist words. Also, the site is using SSL.
tried to sign up, didn't get a confirmation/activation email.
tried to sign in using the password I set, locked the account.
amazing
tried to sign in using the password I set, locked the account.
amazing
If you try to log in before clicking the link in the conformation email you'll get a "invalid credentials" message. So your account might not be locked, just not activated.