Own a weather station? We want your data(weather.gov)
weather.gov
Own a weather station? We want your data
https://www.weather.gov/iln/cwop
205 comments
Hi, I am building an Open-Source Weather API aggregating open-data weather models from NOAA, ECMWF, DWD, MeteoFrance, JMA, CMA, CMCC and others. I agree that many weather companies basically redistribute NWS data at a premium. There is a free API service available on https://open-meteo.com and all databases are redistributed via an AWS Open-Data Sponsorship. Feel free to reach out if you need help building your weather data broker startup
A sincere personal "thank you" from me! I use Breezy Weather[1] on my smartphone, which gets most of its data from your aggregating service, Open-Meteo. It's the perfect combination of accurate, international and open.
[1]: https://f-droid.org/en/packages/org.breezyweather/
[1]: https://f-droid.org/en/packages/org.breezyweather/
First of all, thank you! What a treasure trove of data.
If I may ask a question, do you have historical air quality data?
If I may ask a question, do you have historical air quality data?
You can access historical air quality data from August 2022 onwards: https://open-meteo.com/en/docs/air-quality-api#start_date=20.... Data is based on the Copernicus air quality forecast. All are references listed the documentation
Thank you!
I am working on this kind of solution - please email - see about
Thank you for doing this!
As my own sidebar, I spent many years at a national lab working with distributed sensor networks (primarily ATC and other radar for detecting non-weather stuff :) ). I thought about using ADS-B as input for weather state and forecasts, but never got around to trying it. Now that I am working on my own startup (self-funded and without revenue so far), so this again will likely languish in my todo list. If someone wants to try it, great, and feel free to reach out as I can probably save you some time selecting and interpreting the right ADS-B fields:
We have a lot of aircraft blasting ADS-B reports whenever they fly. Most reports contain (1) accurate 3D position, including altitude and (2) barometric altitude measurements, which gives you (after some minimal work) air pressure. So you have millions freely available pressure reports not just on the ground, but throughout 0-40000 ft altitude band.
You also get measured airspeed and groundspeed, so in addition to pressure you get wind vectors at thousands of points in the air, updating in real time. I suspect this can provide some non-trivial information and I am not aware of anyone actually using it for this purpose.
As my own sidebar, I spent many years at a national lab working with distributed sensor networks (primarily ATC and other radar for detecting non-weather stuff :) ). I thought about using ADS-B as input for weather state and forecasts, but never got around to trying it. Now that I am working on my own startup (self-funded and without revenue so far), so this again will likely languish in my todo list. If someone wants to try it, great, and feel free to reach out as I can probably save you some time selecting and interpreting the right ADS-B fields:
We have a lot of aircraft blasting ADS-B reports whenever they fly. Most reports contain (1) accurate 3D position, including altitude and (2) barometric altitude measurements, which gives you (after some minimal work) air pressure. So you have millions freely available pressure reports not just on the ground, but throughout 0-40000 ft altitude band.
You also get measured airspeed and groundspeed, so in addition to pressure you get wind vectors at thousands of points in the air, updating in real time. I suspect this can provide some non-trivial information and I am not aware of anyone actually using it for this purpose.
These data are already consumed operationally by the major global weather modeling centers - e.g., check out the AMDAR [1] or ACARS programs. There are commercial agreements which restrict third-party access to these data in real-time, but they are widely disseminate with a 24-48 hour lag in the observational archives that NOAA curates for weather modeling.
These data are a very important input for operational weather forecasting. On a global basis, we've seen how losing these data during the pandemic due to reductions in air travel decreased forecast model skill [2]. Furthermore, ACARS profiles derived from aircraft landing at airports where severe weather is expected are regularly used to complement SPECI weather balloon launches.
[1]: https://community.wmo.int/en/activity-areas/aircraft-based-o... [2]: https://journals.ametsoc.org/view/journals/apme/59/11/JAMC-D...
These data are a very important input for operational weather forecasting. On a global basis, we've seen how losing these data during the pandemic due to reductions in air travel decreased forecast model skill [2]. Furthermore, ACARS profiles derived from aircraft landing at airports where severe weather is expected are regularly used to complement SPECI weather balloon launches.
[1]: https://community.wmo.int/en/activity-areas/aircraft-based-o... [2]: https://journals.ametsoc.org/view/journals/apme/59/11/JAMC-D...
Thank you for the info, I am glad to hear that the data is being collected and used. Getting it from raw ADS-B reports makes all fields available without the lag (and thus more useful for weather reporting and forecasting), but the recorded data is definitely good to have!
you referenced "(1)" and "(2)" but didn't provide the actual links
Those numbers were a form of "verbal highlight" of items in a long sentence, not actual links.
For specific ADS-B fields, ADS-B exchange used to have a great data fields overview. ICAO should have a spec, too.
For specific ADS-B fields, ADS-B exchange used to have a great data fields overview. ICAO should have a spec, too.
It was an inline numbered list of two items. [2]
[2] https://www.youtube.com/watch?v=UTby_e4-Rhg
[2] https://www.youtube.com/watch?v=UTby_e4-Rhg
Don't forget, AccuWeather wanted to make it illegal for you to have access to that data for free from the government.
Same thing in Germany, with WetterOnline ruminating data from DWD (the German meteorological service) and then suing them when they offered their DWD WarnWetter app for free. Unfortunately, they were successful: https://www.heise.de/news/BGH-Urteil-Staatlicher-Wetterdiens... (Couldn't find an English source, sorry.)
>Unfortunately, they were successful
Unbelievable.
Unbelievable.
Alas, there are yet ongoing attempts to break-up and/or privatize the NWS. Project 2025 specifically calls for the break-up of NOAA.
Oh for crying out loud! WTF???
I think someone spilled idiot juice in the water.
I think someone spilled idiot juice in the water.
Swamp Water Kool-Aid for the cultists.
You can always use yr.no
This is the Norwegian government weather service. It's global and free for everyone. Also has fully open apis
This is the Norwegian government weather service. It's global and free for everyone. Also has fully open apis
> weather.com (cough IBM)
Didn't IBM sell weather.com (and all of the assets) to a private equity? The deal was announced last year and closed in Feb 2024
https://finance.yahoo.com/news/francisco-partners-completes-...
Didn't IBM sell weather.com (and all of the assets) to a private equity? The deal was announced last year and closed in Feb 2024
https://finance.yahoo.com/news/francisco-partners-completes-...
Yes, they did. AccuWeather is also privately owned and a major competitor of TWCo. It has never had anything to do with IBM.
Yeah, they did... but being blunt, IBM had 8 years to do a lot of damage. Not speaking of the engineers or the tech behind it, I'm talking about escalating the ad revenue and general junk on their website. MOST of the page load time on weather.com and AccuWeather is ad and tracking. It's an ungodly number of requests and will drain a data plan's usage limits surprisingly quickly.
Just opening weather.com will send almost 1000 requests , transfer 10.3MB. Every 30 seconds or so it will make about 300 requests + 2MB of transfer for new ads. It's... insane
Just opening weather.com will send almost 1000 requests , transfer 10.3MB. Every 30 seconds or so it will make about 300 requests + 2MB of transfer for new ads. It's... insane
Ads and cookies are hardly the worst thing that AccuWeather has done. They have been lobbying the federal government for 30 years to forbid NOAA from issuing forecasts at all. They want the government to pay for all the satellites and supercomputers, and they want all that data for free, but they want you to have to pay AccuWeather for forecasts.
In situations like this I feel that it is important to put a face to a name.
AccuWeather, a corporation like any other is made of people. These shortsighted decisions and shitty behavior are directly attributable to people who do them for selfish reasons.
So let's start talking about these people in the first person, not in the abstract.
Who is directly responsible for these atrocious actions and why do they stand to benefit from them?
AccuWeather, a corporation like any other is made of people. These shortsighted decisions and shitty behavior are directly attributable to people who do them for selfish reasons.
So let's start talking about these people in the first person, not in the abstract.
Who is directly responsible for these atrocious actions and why do they stand to benefit from them?
> So let's start talking about these people in the first person, not in the abstract.
Here are two: Rick Santorum and Jim Bridenstine.
https://www.washingtonpost.com/news/capital-weather-gang/wp/...
Here are two: Rick Santorum and Jim Bridenstine.
https://www.washingtonpost.com/news/capital-weather-gang/wp/...
This same junk happens in Canada. Attempts to disrupt pages like this:
https://weather.gc.ca/city/pages/on-118_metric_e.html
and any info from Environment Canada, being packaged in a way that might compete, with the very same people using Environment Canada's free data for-profit.
Cries of "How can the Government compete with the private sector" are thrown around, always glossing over how that very data is often sourced from the Government. Pathetic. Leeches of the worst kind.
https://weather.gc.ca/city/pages/on-118_metric_e.html
and any info from Environment Canada, being packaged in a way that might compete, with the very same people using Environment Canada's free data for-profit.
Cries of "How can the Government compete with the private sector" are thrown around, always glossing over how that very data is often sourced from the Government. Pathetic. Leeches of the worst kind.
But who? Who specifically in Canada is making these cries?
I was at a wedding where the accu-weather guy was complaining the NFL was a monopoly because they wouldn't let Rush Limbaugh buy a team...
John Oliver had an episode[1] of Last Week Tonight which discussed some of this I believe.
[1] https://youtube.com/watch?v=qMGn9T37eR8
[1] https://youtube.com/watch?v=qMGn9T37eR8
At the time they bought it, I joked that IBM bought the Weather Channel as part of its Cloud strategy. I stand by that claim.
[deleted]
> These private companies have a much much higher detail on the ground than the government by being able to partner with companies that make common internet-connected personal radar stations and reselling that data to each other.
I haven't heard of personal radar stations, and wasn't hitting anything in a quick web search. Are you able to provide an example of these systems?
I haven't heard of personal radar stations, and wasn't hitting anything in a quick web search. Are you able to provide an example of these systems?
I'm pretty sure this was a typo, so s/radar/weather/ .
For reference, a weather radar operates in Doppler mode with return signal coming from Rayleigh scattering of raindrops, so it's on the 3cm - 10cm wavelength. You are talking about something like a 5 meter diameter antenna dish that weighs half a tonne, which is on an elevation-azimuth motorized mount, in a 7 meter diameter radome, with peak transmit power of 250 000 W.
Of course you can buy one yourself, if you have the space, electrical power and money for it - ballpark 1.5 mill. USD.
For reference, a weather radar operates in Doppler mode with return signal coming from Rayleigh scattering of raindrops, so it's on the 3cm - 10cm wavelength. You are talking about something like a 5 meter diameter antenna dish that weighs half a tonne, which is on an elevation-azimuth motorized mount, in a 7 meter diameter radome, with peak transmit power of 250 000 W.
Of course you can buy one yourself, if you have the space, electrical power and money for it - ballpark 1.5 mill. USD.
I've worked on a weather radar system with specs suspiciously similar to those you are describing. 250kW is a pulse power for a C-band stationary radar system, with a typical pulse length around 1us repeated 500-1000 times a second it amounts to 1/1000 duty cycle and 250W average radiated power. These pulsing parameters give about 150-300km of usable range, return signal becomes too noisy on longer ranges anyway and geometry of beam propagation means that you're shooting into outer space above meteorologically interesting part of atmosphere. It doesn't use that much power from the grid either - datasheet specs around 5kW total for all the stuff (transmitter, motorized antenna pedestal and equipment rack with a pretty beefy server to chew all that data coming from the receiver in real time). The cost aspect is pretty much in the ballpark - I've once visited our test site and the engineer pointed on the antenna horn (about paint can sized chunk of metal hanging in front of the dish full of microwave RF magic ) and told that just that piece costs about the same as a new high-segment car.
C-band weather radars existed since forever, first using magnetron transmitters and now solid-state amplifiers, though there are still a lot of new magnetron-based systems being installed, with the downside that magnetron pulses are practically impossible to modulate to perform advanced radar techniques that improve different aspects performance. There are also X-band weather radars, which operate on higher frequencies and use more modestly sized antennas (but still larger than you'd like to have on your house roof). They are more limited in range (100km-ish max) due to high attenuation and mostly used at airports, offshore oil rigs and windfarms and similar installations that mostly interested in precise local weather. They are still several hundred thousand bucks.
C-band weather radars existed since forever, first using magnetron transmitters and now solid-state amplifiers, though there are still a lot of new magnetron-based systems being installed, with the downside that magnetron pulses are practically impossible to modulate to perform advanced radar techniques that improve different aspects performance. There are also X-band weather radars, which operate on higher frequencies and use more modestly sized antennas (but still larger than you'd like to have on your house roof). They are more limited in range (100km-ish max) due to high attenuation and mostly used at airports, offshore oil rigs and windfarms and similar installations that mostly interested in precise local weather. They are still several hundred thousand bucks.
I don't know how much this costs, but it's only a meter, 65 kg, and runs on a 110 volt outlet. I'm thinking high ones of thousands of dollars, maybe low tens of thousands.
https://www.furuno.com/files/Brochure/448/upload/WR110_EN.pd...
https://www.furuno.com/files/Brochure/448/upload/WR110_EN.pd...
They only list agents and the agents don't list prices and few feature the doppler weather radar on ther website ..
That said, I'd guess $10,000 US < price < $50,000 US for that product given the pricing of $8K US for smaller commercial fishing radars from the same company.
More on the two compact weather radar systems: https://www.furuno.com/en/systems/meteorological-monitoring/
That said, I'd guess $10,000 US < price < $50,000 US for that product given the pricing of $8K US for smaller commercial fishing radars from the same company.
More on the two compact weather radar systems: https://www.furuno.com/en/systems/meteorological-monitoring/
Probably another order of magnitude more expensive.
Maybe, maybe not.
There's another recent peer comment that confirms 1.5 million US for a large 250kW is a pulse power for a C-band stationary radar system with 300 km of range.
This is a small doppler with modest power and 70 km range max.
If cost is proportional to the 3D volume of space scanned (as the power requirements likely are) Then this small mini radar might well be less than 100K.
TBH I have no specific knowledge here although I have worked in other sensing domains and seen a wide spread on cost of equipment related to volume and quality of data.
I'd be interested in the flat cost price of the mini system, I suspect that's not going to appear without some inside knowledge or working a dealer, it appears to be a rare bespoke kinf of thing, not like the commercial fishing radars.
There's another recent peer comment that confirms 1.5 million US for a large 250kW is a pulse power for a C-band stationary radar system with 300 km of range.
This is a small doppler with modest power and 70 km range max.
If cost is proportional to the 3D volume of space scanned (as the power requirements likely are) Then this small mini radar might well be less than 100K.
TBH I have no specific knowledge here although I have worked in other sensing domains and seen a wide spread on cost of equipment related to volume and quality of data.
I'd be interested in the flat cost price of the mini system, I suspect that's not going to appear without some inside knowledge or working a dealer, it appears to be a rare bespoke kinf of thing, not like the commercial fishing radars.
Even if you buy one you will almost certainly need to get spectrum approval to transmit, which is non-trivial and comes with many strings.
https://www.meteopress.cz/meteopress-com/ These guys started to produce beefy radars order of magnitude cheaper
I plan on installing one of these at my apartment building so that my tenants can have their own local weather channel.
My friend built one. Details here: https://www.youtube.com/watch?v=fvxZvObr2hw
Since the Weather Research and Forecasting Innovation Act of 2017, Congress is requiring NOAA to start acquiring data via commercial partnerships.
NOAA has already made some contracts with Spire [1] and Saildrone [2]. I am sure there are more but these are the ones most familiar to me.
Your weather data broker startup sounds very interesting!
[1]: https://spire.com/press-release/spire-global-awarded-nationa... [2]: https://research.noaa.gov/2022/08/03/noaa-and-saildrone-team...
NOAA has already made some contracts with Spire [1] and Saildrone [2]. I am sure there are more but these are the ones most familiar to me.
Your weather data broker startup sounds very interesting!
[1]: https://spire.com/press-release/spire-global-awarded-nationa... [2]: https://research.noaa.gov/2022/08/03/noaa-and-saildrone-team...
The entire slate of commercial acquisitions planned or in progress can be found at [1]. It's pretty anemic; NOAA has spent far less than what folks were hoping they would. I think a major part of this is that the private sector really didn't have very many high-TRL observation systems that could readily be integrated into NOAA's assimilation and forecasting systems. Lots of planned constellations and ideas about things to do in the future, but just not that much stuff that was ready to package-up and deliver to NOAA. The most successful acquisitions for GPS/GNSS-RO and buoy/drone data seems strongly bolstered by the fact that these data were already readily assimilated by existing infrastructure.
The private sector has really embellished its capabilities to the detriment of the CDP and other programs. I think too many industry players saw NOAA's expansion here as a potential slush fund to fully subsidize their R&D, but again the TRL of planned observation systems was too low and so the system didn't really work efficiently. Classic policy failure - would make a fantastic case study or Master's thesis for someone studying weather in an STS program!
[1]: https://www.space.commerce.gov/business-with-noaa/commercial...
The private sector has really embellished its capabilities to the detriment of the CDP and other programs. I think too many industry players saw NOAA's expansion here as a potential slush fund to fully subsidize their R&D, but again the TRL of planned observation systems was too low and so the system didn't really work efficiently. Classic policy failure - would make a fantastic case study or Master's thesis for someone studying weather in an STS program!
[1]: https://www.space.commerce.gov/business-with-noaa/commercial...
> Sidebar: I'm a founder of a self-bootstrapped startup to build a better weather data broker that doesn't cost an arm and a leg. If that's something you are passionate about solving, feel free to reach out :)
Would love to hear more about this. I’m a researcher and a lot of my work revolves around machine learning applications to building energy modeling, and one of my projects actually revolves around the difference between using TMY vs AMY EPW files in automatic calibration of models. Would be great to chat more. What’s the name of your startup? I can shoot you an email at the official email.
Would love to hear more about this. I’m a researcher and a lot of my work revolves around machine learning applications to building energy modeling, and one of my projects actually revolves around the difference between using TMY vs AMY EPW files in automatic calibration of models. Would be great to chat more. What’s the name of your startup? I can shoot you an email at the official email.
I have always wanted to build a cheap ESP based weather station I can give to a friend, have them set up their WiFi and it would transmit data to the central server and I do not want to submit data to AccuWeather or wunderground or someone else.
Is there a Foss server around that I can set up on my own ?
Is there a Foss server around that I can set up on my own ?
FWIW, purple air is basically this minus wind information.
This sounds really cool. Have you looked into see if you can get permission from the guys over at aprs.fi to scrape their data from ham radio weather stations running over APRS? [1]
[0] https://aprs.fi/page/api
[0] https://aprs.fi/page/api
My rough understanding is CWOP was started by ham nerds and forwards all data to MADIS via FindU[0]
[0]: <http://www.findu.com/>
[0]: <http://www.findu.com/>
Thanks for mentioning. When I looked up MADIS, Wiki pointed me here: https://www.weather.gov/ncep/
There, selecting 'Weather Prediction Center' goes to https://www.wpc.ncep.noaa.gov/#page=ovw
There I searched in 'Local forecast' for cities and see most of what 'weather com' delivers, with 3 domains in Noscript instead of the couple-dozen from weather.com
There, selecting 'Weather Prediction Center' goes to https://www.wpc.ncep.noaa.gov/#page=ovw
There I searched in 'Local forecast' for cities and see most of what 'weather com' delivers, with 3 domains in Noscript instead of the couple-dozen from weather.com
I became so annoyed at the tracking, advertising and poor notifications of many apps, I wrote my own for severe weather alerts.
My hobby will include a C band antenna this fall, and I’m on the hunt for radar data sources in which to create my own mosaics.
My hobby will include a C band antenna this fall, and I’m on the hunt for radar data sources in which to create my own mosaics.
What is the best option for someone who wants to contribute but doesn’t have a weather station currently or a lot of free time to take on another project?
For the data, do you know meteostat.net?
They get input from e.g. NOAA (US), DWD (DE), YR (NO) and so on.
I'm not affiliated, just needed some historical data and (finally!) discovered them.
They get input from e.g. NOAA (US), DWD (DE), YR (NO) and so on.
I'm not affiliated, just needed some historical data and (finally!) discovered them.
If I may use your expertise a bit: I need air quality data for two cities, one of which doesn't have any air quality monitoring stations. Is there anything I can use as a proxy for air pollution, like satellite data? Also, do you know any sources for that data?
Thanks!
P.S.: I'm passionate about air quality, given I have alergies and was an active member of the bicycle activism movement for about 10 years.
Thanks!
P.S.: I'm passionate about air quality, given I have alergies and was an active member of the bicycle activism movement for about 10 years.
I don't understand why the quality of the specific weather station does not matter. Isn't bad data impacting the overall model quality?
> NOAA/NWS, for example, is extremely underfunded so if they had to privilege to buy it they probably couldn't come to an agreement to buy it. As a result, they can't use that data to improve the accuracy of alerts/warnings/forcasts, the same exact tools that the big weather companies make all their money from. It's a shit cycle and totally unfair IMO.
Huh? This is kind of an odd take for a few reasons. For starters, NOAA isn't "extremely underfunded"; with the possible exception of the current budgeting cycle, NOAA generally does pretty well and has strong bipartisan support. It could always use more money, but I wouldn't call it "underfunded.
The reason NOAA doesn't buy more data is because most of the available data has limited value. Personal weather stations have substantial quality issues and add almost no value in areas where we already have high-quality surface observations. We thin out and throw away a ton of surface observations already during the data assimilation process to initialize our forecast models anyways - data from aloft is far more valuable and impactful from a forecast impact perspective.
For what it's worth, few if any companies use proprietary observations to improve their forecasts. It's an open secret that the vast majority of companies out there are just applying proprietary statistical modeling / bias correction on top of publicly available data. Only a handful of companies actually have novel observations, and there's limited evidence it makes a significant difference in the forecast. At best, it can result in the way that those statistical corrections are applied to existing forecasts and ensembles - you can count on one hand the number of companies that actually run a vertically-integrated stack including data assimilation of proprietary observations and end-to-end numerical modeling.
That isn't to say there isn't unique value in the observations. It's just that the industry flagrantly misleads about how they use them.
Huh? This is kind of an odd take for a few reasons. For starters, NOAA isn't "extremely underfunded"; with the possible exception of the current budgeting cycle, NOAA generally does pretty well and has strong bipartisan support. It could always use more money, but I wouldn't call it "underfunded.
The reason NOAA doesn't buy more data is because most of the available data has limited value. Personal weather stations have substantial quality issues and add almost no value in areas where we already have high-quality surface observations. We thin out and throw away a ton of surface observations already during the data assimilation process to initialize our forecast models anyways - data from aloft is far more valuable and impactful from a forecast impact perspective.
For what it's worth, few if any companies use proprietary observations to improve their forecasts. It's an open secret that the vast majority of companies out there are just applying proprietary statistical modeling / bias correction on top of publicly available data. Only a handful of companies actually have novel observations, and there's limited evidence it makes a significant difference in the forecast. At best, it can result in the way that those statistical corrections are applied to existing forecasts and ensembles - you can count on one hand the number of companies that actually run a vertically-integrated stack including data assimilation of proprietary observations and end-to-end numerical modeling.
That isn't to say there isn't unique value in the observations. It's just that the industry flagrantly misleads about how they use them.
> We thin out and throw away a ton of surface observations already during the data assimilation process to initialize our forecast models anyways - data from aloft is far more valuable and impactful from a forecast impact perspective.
I regularly notice that the NWS forecasts, even in the very short term, get the surface conditions rather wrong. (This is by comparison to a an inadvertent but, I think, quite accurate surface temperature and humidity measurement that I have.)
I fully believe that the measurements aloft do a great job of predicting the conditions aloft, but I wonder whether the results would be further improved by even a fairly simple model to map the forecast results back to detailed surface conditions. After all, many of consumers of weather forecasts, e.g. people caring about personal comfort, climate control energy predictions and pre-heating/pre-cooling of buildings, etc. care about surface conditions more than they care about conditions aloft.
I regularly notice that the NWS forecasts, even in the very short term, get the surface conditions rather wrong. (This is by comparison to a an inadvertent but, I think, quite accurate surface temperature and humidity measurement that I have.)
I fully believe that the measurements aloft do a great job of predicting the conditions aloft, but I wonder whether the results would be further improved by even a fairly simple model to map the forecast results back to detailed surface conditions. After all, many of consumers of weather forecasts, e.g. people caring about personal comfort, climate control energy predictions and pre-heating/pre-cooling of buildings, etc. care about surface conditions more than they care about conditions aloft.
The measurements aloft constrain the entire system - things like vertical profiles of moisture and temperature, as well as the kinematic structure of the atmosphere (e.g. wind profiles) grossly constrain the evolution of the system. Put another way - the _information density_ of these observations is very high and they tend to constrain non-local features of the flow / structure of the atmosphere.
Observations for the surface don't have this effect for two reasons: (1) they can be dominated by local influences (like local topography) that poorly constrain the background atmospheric state, and (2) the majority of numerical weather models do not directly model the planetary boundary layer (the layer of the atmosphere closest to the ground), and instead parameterize processes that occur here. What this means, practically, is that the information content of surface observations is low (1), and even when it isn't, there isn't a mechanism to effectively propagate this information outside of a single grid cell or even column in the actual forecast model (2).
That's why observations are typically used to bias-correct forecast models - it's a form of localization or downscaling.
Observations for the surface don't have this effect for two reasons: (1) they can be dominated by local influences (like local topography) that poorly constrain the background atmospheric state, and (2) the majority of numerical weather models do not directly model the planetary boundary layer (the layer of the atmosphere closest to the ground), and instead parameterize processes that occur here. What this means, practically, is that the information content of surface observations is low (1), and even when it isn't, there isn't a mechanism to effectively propagate this information outside of a single grid cell or even column in the actual forecast model (2).
That's why observations are typically used to bias-correct forecast models - it's a form of localization or downscaling.
Very few companies run the vertically-integrated stack because it is prohibitively expensive to do so with current NWP versus what you can sell it for with only marginal forecast improvements. I know several companies have tried this with integrating their own observation sources and ended up with worse performing forecasts. Oops.
I'm very interested to see how the ML modeling revolution changes this. The ability to perform global forecasts on a single GPU should make it cost competitive for more companies. I know several companies are already deriving their own weights for the forecasting component so that they can sell them. Google appears to be working on the next piece of the puzzle too with using ML for the data assimilation step, or skipping that altogether and using observations to go directly to forecasts.
I'm very interested to see how the ML modeling revolution changes this. The ability to perform global forecasts on a single GPU should make it cost competitive for more companies. I know several companies are already deriving their own weights for the forecasting component so that they can sell them. Google appears to be working on the next piece of the puzzle too with using ML for the data assimilation step, or skipping that altogether and using observations to go directly to forecasts.
There are a few groups working on leveraging observations more directly in the ML forecast models and skipping over the assimilation/analysis step. However, unlike the original ML forecasting problem (which, let's be honest - was grossly over-simplified by the existence of ERA-5, which has been treated as "ground truth" for the atmosphere and used to teach models how to simply go from state at t=1 to state at t=1+\delta t), there's reason to believe that incorporating the observations will be substantially more difficult, given the complexity and bounty of the observations themselves and the challenge of framing a tractable, useful ML problem on top of them.
Referring to the parent comment, the data which NOAA isn’t able to buy is the government’s data, which is freely provided to non-government organizations. The parent comment doesn’t discuss personal data very much. I think this misunderstanding might be the root cause of your disagreement.
Parent comment is mis-informed. NOAA doesn't have to buy any data from other government agencies or organizations. It's all open and publicly available. There are challenges around the reliability and quality of data that NOAA doesn't take efforts itself to curate and maintain which limit their utility, but that's a separate issue.
More importantly, NOAA explicitly funds the National Mesonet Program [1] to actively identify, acquire, consume, and ingest data from a wide variety of state and federal agencies across the country. The NMP itself partners with a major private sector company, Synoptic Data PBC [2] to perform the engineering necessary to acquire all this data. Synoptic actively maintains the infrastructure which consumes and publishes this data to MADIS for use by NOAA and any other stakeholder.
[1]: https://nationalmesonet.us/ [2]: https://synopticdata.com/
More importantly, NOAA explicitly funds the National Mesonet Program [1] to actively identify, acquire, consume, and ingest data from a wide variety of state and federal agencies across the country. The NMP itself partners with a major private sector company, Synoptic Data PBC [2] to perform the engineering necessary to acquire all this data. Synoptic actively maintains the infrastructure which consumes and publishes this data to MADIS for use by NOAA and any other stakeholder.
[1]: https://nationalmesonet.us/ [2]: https://synopticdata.com/
If you see this post, You should go share your story and probably this effort with the Home automation and Home assistant crowd.
As consumers and creators of plenty of weather data you might see a fair bit of traction there!
As consumers and creators of plenty of weather data you might see a fair bit of traction there!
what’s good weather station hw to get? thinking ultrasonic wind instrument and some type of radar
Doesn't encompass radar (and no home weather station will) but NetAtmo sells one with ultrasonic wind, smart rain gauge, and has a full RESTful API available: https://www.netatmo.com/smart-weather-station
I use this and love it.
I use this and love it.
Hi Mike- I am very interested in this and doing some work - please email
"The governments of the world provides these big weather companies (weather.com (cough IBM), Accuweather (cough IBM cough), etc) a metric shit ton of their data completely for free (by law) including data transfer. These are things like radar, satellite, ground station data, forecasts, composite models, etc. These companies profit substantially on it, as in billions of dollars"
Michael Lewis' book The Fifth Risk goes IN DEPTH into how Accuweather/Weather.com and the government were interacting (particularly during the Trump administration).
I highly recommend the book in general and for this particular story in particular.
Michael Lewis' book The Fifth Risk goes IN DEPTH into how Accuweather/Weather.com and the government were interacting (particularly during the Trump administration).
I highly recommend the book in general and for this particular story in particular.
Do you have a source for sites like weather.com and AccuWeather being part of a billions of dollars industry? That is surprising to me.
It's hard to get accurate numbers as many weather sites are privately held. But to give some info:
Weather Company was sold for $2B to IBM in 2015 (and recently sold to private equity for undisclosed amount). Tomorrow.IO has a $1B+ valuation pre-IPO SPAC.
https://www.forbes.com/sites/johnkoetsier/2021/04/08/ibms-we...
Weather Company was sold for $2B to IBM in 2015 (and recently sold to private equity for undisclosed amount). Tomorrow.IO has a $1B+ valuation pre-IPO SPAC.
https://www.forbes.com/sites/johnkoetsier/2021/04/08/ibms-we...
That’s not really indicative of a large industry. If the weather company was bought for 2 billion, at standard valuations they were probably pulling at most 100 million per year in net income.
User name checks out.
> AccuWeather's estimated annual revenue is currently $156.6M per year.
You can't safely fly or sail unless you know and can predict the weather at your destination.
That's not a full explanation for these companies being valuable though. Only flight planning makes use of commercial weather forecasts - the actual decisions (take off or delay, land or redirect to alternate?) are made by the pilots based on reports made by the airports themselves, usually based on their respective national forecaster's reports. Commercial pilots don't just bring up weather.com on their iPad on final approach, although their dispatcher might well have used a more sophisticated version of the same thing hours before the flight set off.
P.S. It's easy find online versions of the reports that pilots get over the radio. Here's an example with explanation for Boston Logan International, USA: https://aviationweather.gov/data/metar/?id=KBOS&hours=0&deco...
P.S. It's easy find online versions of the reports that pilots get over the radio. Here's an example with explanation for Boston Logan International, USA: https://aviationweather.gov/data/metar/?id=KBOS&hours=0&deco...
Big Weather does provide the data for "free" to the weather widgets on millions of phones and computers.
So they do provide a public service, even if maybe they get too much money from it.
So they do provide a public service, even if maybe they get too much money from it.
Why is this unfair? With these private companies I get access to all this data for free. If the government did it I would have to pay for it out of my taxes. Why do we need the government to provide more accurate forecasts when this is already taken care of by the private sector?
You get that data for "free" or your hand over your browser data - and likely location data, to unknown third party brokers that know more about you than you know about yourself. Weather apps in particular are notorious for selling your location data, since everyone does "show weather for my current location".
It's "free" in the worst sense of the word.
It's "free" in the worst sense of the word.
Do you know much about Windy, and whether they have any ties to selling user location data?
All the plots and features seemed nice so I paid for a year of subscription (after Dark Sky was discontinued ugh), but if they do that I’ll just stick to the default apple Weather app.
All the plots and features seemed nice so I paid for a year of subscription (after Dark Sky was discontinued ugh), but if they do that I’ll just stick to the default apple Weather app.
> With these private companies I get access to all this data for free.
These companies are already receiving government data for free. And then enriching it with private weather station data (and ads). So your taxes are already funding some part of it.
As computing power being used for forecasting increases (14.5 petaflops each for both Dogwood and Cactus NOAA supercomputers), having more granular data is going to be useful in improving their models.
https://www.noaa.gov/news-release/noaa-completes-upgrade-to-...
These companies are already receiving government data for free. And then enriching it with private weather station data (and ads). So your taxes are already funding some part of it.
As computing power being used for forecasting increases (14.5 petaflops each for both Dogwood and Cactus NOAA supercomputers), having more granular data is going to be useful in improving their models.
https://www.noaa.gov/news-release/noaa-completes-upgrade-to-...
It is in the public interest to have both weather and forecasts. Maybe you are like me and look out the window to see the weather for that day, but millions need accurate weather and forecasts. They include farmers, those that run power grids, air traffic controllers, and more.
I live in Portugal, and not only do I have a weather station, I’ve set up flood gauges with GSM uplinks along several local rivers - because I live in a watermill and I want to know when we’re about to have a bad time, after getting hit by a centenary flood.
Nobody wants the data, sadly - there used to be a national monitoring system on rivers up and down the country, but after 2008 it was abandoned, and now there’s only limited monitoring of major rivers, and none of tributaries. The IPMA and APA eventually responded and said thanks for your interest, but we have no means to ingest third party data.
I wish it could be useful for someone other than myself - but short of building my own platform and ingesting the APA’s copyrighted data, it ain’t happening.
Nobody wants the data, sadly - there used to be a national monitoring system on rivers up and down the country, but after 2008 it was abandoned, and now there’s only limited monitoring of major rivers, and none of tributaries. The IPMA and APA eventually responded and said thanks for your interest, but we have no means to ingest third party data.
I wish it could be useful for someone other than myself - but short of building my own platform and ingesting the APA’s copyrighted data, it ain’t happening.
> after 2008 it was abandoned
It's scary how easily this happens. Even America briefly reduced funding to the USGS during the Trump years, among other agencies that saw budgets cut for anything that collects data on the environment. The impact was immediate and now certain important time series including stream gauges and ground water levels have years of missing data because those programs were shut down.
It's scary how easily this happens. Even America briefly reduced funding to the USGS during the Trump years, among other agencies that saw budgets cut for anything that collects data on the environment. The impact was immediate and now certain important time series including stream gauges and ground water levels have years of missing data because those programs were shut down.
Ironically, a lot of the gauges here have been destroyed by floods and were never replaced.
Own a national weather service?
I want your API to not return 500 errors for hours at a time, multiple times a day...
Today it was down from 9AM to 2PM, and then from 9:30PM to 11PM.
Yesterday was even worse.
It isn't rate throttling because I only try to retrieve current conditions every 30 minutes.
Also, the load of the API might be a lot lower if it were possible to retrieve just the current measurements, but the response you get back for observations is insanely bloated with 12 hours back of each measurement, plus a bunch of station information that is heavily duplicative of information provided about the station.
Nothing about the station should be returned when you ask for the station's measurements should be returned except a link to API call for the station's information.
I want your API to not return 500 errors for hours at a time, multiple times a day...
Today it was down from 9AM to 2PM, and then from 9:30PM to 11PM.
Yesterday was even worse.
It isn't rate throttling because I only try to retrieve current conditions every 30 minutes.
Also, the load of the API might be a lot lower if it were possible to retrieve just the current measurements, but the response you get back for observations is insanely bloated with 12 hours back of each measurement, plus a bunch of station information that is heavily duplicative of information provided about the station.
Nothing about the station should be returned when you ask for the station's measurements should be returned except a link to API call for the station's information.
They are migrating the api to a new domain and browning out the old one to get folks to switch over and figure out if there are any issues. There is also another infrastructure change going on as well.
Normally those would be posted on https://www.weather.gov/notification/ but that’s having hiccups, but the email list for announcements still works.
A few of the recent announcements: - https://www.weather.gov/media/notification/pdf_2023_24/scn24... - https://www.weather.gov/media/notification/pdf_2023_24/scn24... - https://www.weather.gov/media/notification/pdf_2023_24/scn24...
Normally those would be posted on https://www.weather.gov/notification/ but that’s having hiccups, but the email list for announcements still works.
A few of the recent announcements: - https://www.weather.gov/media/notification/pdf_2023_24/scn24... - https://www.weather.gov/media/notification/pdf_2023_24/scn24... - https://www.weather.gov/media/notification/pdf_2023_24/scn24...
I'm fairly sure weather.gov's api doesn't have an SLA of any kind. I highly recommend going through the push vs polling approach if possible and store the data as it comes in live in your own DB. AWS Open Data + MADIS are good sources that are... for reasons... far more stable
EDIT: sorry, I misread your commend. If it's only once every 30 secs, that may be a bit overkill for your needs. You may be able to get the appropriate text product directly and parse it out. Unfortunately probably the most stable, yet clunkiest, way
EDIT: sorry, I misread your commend. If it's only once every 30 secs, that may be a bit overkill for your needs. You may be able to get the appropriate text product directly and parse it out. Unfortunately probably the most stable, yet clunkiest, way
Let's not forget the time they just turned it off for the winter holidays because they didn't want to be paged about it going down!
> Also, the load of the API might be a lot lower if it were possible to retrieve just the current measurement
/stations/{stationId}/observations/latest ?
/stations/{stationId}/observations/latest ?
That's fair. I assume it's this? https://www.weather.gov/documentation/services-web-api
Strange that they have outage information via radio. What endpoints were failing? I'll email and see if they'll accept some help.
Strange that they have outage information via radio. What endpoints were failing? I'll email and see if they'll accept some help.
Likely what's happening, is someone was put in charge with a purposeful goal of weakening public end points. Similar shenanigans happened with the FCC, and it is a modern trick to denigrating the value of public services.
* First, take an excellent Government service and under-fund it by reducing its budget or hobbling it in some capacity
* Ensure that the prior does not cut the budget in half, or something so overt. If you do that, your political adversaries will surely drum up support to stop you. Instead just never increase the budget. Let aging infrastructure go unfunded for upgrades and improvements, etc. Small, non-apparent cuts.
* Next, put someone in charge who ensures that the service is eating money, yet its output is useless for most tasks, and unreliable.
* Later, complain how the service is clearly not useful, yet "look at all this money the taxpayer pays!!", and suggest changing its scope so that it better aligns with "others providing the same service". Key to this is that you didn't cut the budget extensively, so the service is indeed very expensive without value.
* Either dismantle the service entirely, or turn it into a shadow of its former self
This is an ongoing game. There are those that literally go bananas, at the idea of anything, anything at all being done by Government.
It is ridiculously easy to keep services up, if you're even mildly competent, and just handle things with the correct process. Process is vital here, key, how you develop services, and how you deploy and monitor and maintain them, is key.
The fact that stuff "just goes down" repeatedly means that they literally have no idea what they're doing.
And I suspect that's on purpose.
* First, take an excellent Government service and under-fund it by reducing its budget or hobbling it in some capacity
* Ensure that the prior does not cut the budget in half, or something so overt. If you do that, your political adversaries will surely drum up support to stop you. Instead just never increase the budget. Let aging infrastructure go unfunded for upgrades and improvements, etc. Small, non-apparent cuts.
* Next, put someone in charge who ensures that the service is eating money, yet its output is useless for most tasks, and unreliable.
* Later, complain how the service is clearly not useful, yet "look at all this money the taxpayer pays!!", and suggest changing its scope so that it better aligns with "others providing the same service". Key to this is that you didn't cut the budget extensively, so the service is indeed very expensive without value.
* Either dismantle the service entirely, or turn it into a shadow of its former self
This is an ongoing game. There are those that literally go bananas, at the idea of anything, anything at all being done by Government.
It is ridiculously easy to keep services up, if you're even mildly competent, and just handle things with the correct process. Process is vital here, key, how you develop services, and how you deploy and monitor and maintain them, is key.
The fact that stuff "just goes down" repeatedly means that they literally have no idea what they're doing.
And I suspect that's on purpose.
Recheck their API spec, you can limit observations to current or any number you want for any station.
The problem however, is the 500 or target station readings returning null.
The problem however, is the 500 or target station readings returning null.
There used to be this amazing hobbyist weather station sharing site. Weather Underground.
But like so many they turned commercial and kicked off all the hobbyists that made them big. It was a really nasty move. I have never found a suitable replacement. I don't know what it is about weather that gives people so much greed.
But like so many they turned commercial and kicked off all the hobbyists that made them big. It was a really nasty move. I have never found a suitable replacement. I don't know what it is about weather that gives people so much greed.
The hobbyists are still on wunderground, I still use it to regularly check some in my neighborhood.
The only "bad" thing Weather Underground did was they ended their free API for non-contributors. [1]
I believe personal weather station contributors still get free access to a different API that wunderground moved to. [2][3]
[1] https://archive.md/xe9B1
[2] https://docs.google.com/document/d/1eKCnKXI9xnoMGRRzOL1xPCBi...
[3] https://github.com/cytech/Home-Assistant-wundergroundpws/?ta...
The only "bad" thing Weather Underground did was they ended their free API for non-contributors. [1]
I believe personal weather station contributors still get free access to a different API that wunderground moved to. [2][3]
[1] https://archive.md/xe9B1
[2] https://docs.google.com/document/d/1eKCnKXI9xnoMGRRzOL1xPCBi...
[3] https://github.com/cytech/Home-Assistant-wundergroundpws/?ta...
Well that is pretty bad IMO. What's the point of sharing it if people can't access it? And I don't think there is even an option to pay for personal accounts.
It was a wonderful community-driven service and they basically canned the whole community, and moved to focus on commercial users. I think that's very bad. Even the site is totally useless now. Only crap news and some basic weather reports but it's very hard to find anything community generated.
What's worst though is that no alternative emerged. I had expected that to happen. It's sad that community alternatives always end up selling out. The same with ADS-B Explorer.
It was a wonderful community-driven service and they basically canned the whole community, and moved to focus on commercial users. I think that's very bad. Even the site is totally useless now. Only crap news and some basic weather reports but it's very hard to find anything community generated.
What's worst though is that no alternative emerged. I had expected that to happen. It's sad that community alternatives always end up selling out. The same with ADS-B Explorer.
The web-based “history” tool’s content and usability has been significantly reduced, as well. You used to be able to view and download a lot of detailed hourly data - now it’s reduced to things like daily summaries when selecting long time ranges.
> But like so many they turned commercial and kicked off all the hobbyists that made them big. It was a really nasty move.
Yes, their acquisition by IBM really messed things up. I went from totally loving the app to uninstalling it and never looking back.
Definitely going to feed my local data to weather.gov.
Yes, their acquisition by IBM really messed things up. I went from totally loving the app to uninstalling it and never looking back.
Definitely going to feed my local data to weather.gov.
I have sent my weather data with my Ambient station to wunderground for the last 7 years without issue. It has been used to track local weather patterns due to hurricanes and tropical storms too
What are some other examples? Ultimate Guitar Tabs comes to mind too
This may be a dumb question but: What is a weather station exactly? How much does the hardware cost to install and then set up an interface to send data to this system?
I own this* one and it is set up on a pole off of my deck. I connect it to wunderground.com and anyone can look up data from my station. I'll be sharing with this service as well now that I know about it.
* https://ambientweather.com/ws-5000-ultrasonic-smart-weather-...
There are much cheaper options than that one if you're interested in hosting a weather station. I decided to install it since I live in a relatively rural area and there is a fairly large gap in coverage where I live. Now I can get more local data and my neighbors can benefit as well. If you live in an urban area odds are a lot of the stations you see on the coverage map are just like this. When I lived in Austin I had a lot nearby readings and if you dig into the info tab for each one you'll see they're personal stations people are sharing.
* https://ambientweather.com/ws-5000-ultrasonic-smart-weather-...
There are much cheaper options than that one if you're interested in hosting a weather station. I decided to install it since I live in a relatively rural area and there is a fairly large gap in coverage where I live. Now I can get more local data and my neighbors can benefit as well. If you live in an urban area odds are a lot of the stations you see on the coverage map are just like this. When I lived in Austin I had a lot nearby readings and if you dig into the info tab for each one you'll see they're personal stations people are sharing.
I run an Ambient WS 5000, and I run two Weatherflow Tempests as well.
With the Ambient Weather WS-5000, the little console panel is also the hub, so you'll have to have it within signal range. You can put their new "Weather Window" elsewhere in the home:
https://ambientweather.com/awn-weather-window
It also supports indoor and outdoor sensors for room, soil, lightning, etc.:
https://ambientweather.com/wh31e-wireless-temperature-humidi...
As a simpler device but remarkably rich start, I'd recommend people try the Tempest to get started:
https://shop.tempest.earth/products/tempest
The API is incredibly easy to work with, both as local devices and as a station:
https://weatherflow.github.io/Tempest/api/
I'm using both brands in a hostile environment, and find the Tempests last better.
I'm also very interested in Weatherflow's network:
https://business.tempest.earth/tempest-network
And as discussed here, Weatherflow Networks collabs with unis and NOAA:
http://weatherflownetworks.com/noaa-national-mesonet-program
On Android and iOS, you can use an app called SmartMixin to commingle and compare readings across multiple brands of weather device.
https://smartmixin.io/
With the Ambient Weather WS-5000, the little console panel is also the hub, so you'll have to have it within signal range. You can put their new "Weather Window" elsewhere in the home:
https://ambientweather.com/awn-weather-window
It also supports indoor and outdoor sensors for room, soil, lightning, etc.:
https://ambientweather.com/wh31e-wireless-temperature-humidi...
As a simpler device but remarkably rich start, I'd recommend people try the Tempest to get started:
https://shop.tempest.earth/products/tempest
The API is incredibly easy to work with, both as local devices and as a station:
https://weatherflow.github.io/Tempest/api/
I'm using both brands in a hostile environment, and find the Tempests last better.
I'm also very interested in Weatherflow's network:
https://business.tempest.earth/tempest-network
And as discussed here, Weatherflow Networks collabs with unis and NOAA:
http://weatherflownetworks.com/noaa-national-mesonet-program
On Android and iOS, you can use an app called SmartMixin to commingle and compare readings across multiple brands of weather device.
https://smartmixin.io/
I have some forest land that is remote and I'd like to set up weather sensors on.
I've been stuck on what to purchase, especially since there's no dedicated power or network connections (the site does have cellular access).
The biggest hurdle I have is "here's the full kit of shit you need to buy in that case", any suggestions?
I've been stuck on what to purchase, especially since there's no dedicated power or network connections (the site does have cellular access).
The biggest hurdle I have is "here's the full kit of shit you need to buy in that case", any suggestions?
I had some luck with ChatGPT. The model numbers are out of date but the general descriptions and capabilities of each brand aren't. And in general ChatGPT is good with prompts like "I want to do X, tell me all the things I should consider" - it has been trained on Q&A sites after all.
That's cool of you to share, particularly in a data drought area. I recently upgraded my sprinkler controller to a Rachio because it could use aggregated weather data, then during setup found out it could use individual weather stations, and not only that, but apparently my neighborhood HOA runs one already. Many weather apps can pull from personal weather stations, like CARROT Weather (my favorite), and it's really nice to know the actual temperature as of roughly 100 yards from my door when I take my dog for a walk.
Any good options for bare weather stations with zigbee or wifi connectivity, then let something like HomeAssistant do the monitoring and display?
I'm the wrong person to ask, as I'm fairly new to this. This is my first foray into home automation in general, but this platform has a Home Assistant integration that I'm playing around with. But the included base station is what is aggregating all of the data from the sensors and sending it out to the cloud when connected to wifi. HA is able to poll that base station for the data as well, and I think you can do that purely on the local network. You can keep all of the data local to only the base station and not use their cloud service.
I picked this platform because it has a lot of out-of-the-box functionality as I'm just getting started. It looked like I can still play around outside their walled garden as I do more home automation stuff. Beyond the outdoor weather data a site like WU wants, it has sensors for indoor temp and air quality so I can set up some HVAC routines or monitor if my chest freezer for getting too warm. It has a soil moisture sensor and one day I'd like to set up an automatic irrigation system for my garden. I can expand the connected sensors as I think of more things I want to try.
I think I could have accomplished a lot of the same with bare sensors like you're suggesting but I wasn't ready to take that on. That was too much learning curve for me right now.
I picked this platform because it has a lot of out-of-the-box functionality as I'm just getting started. It looked like I can still play around outside their walled garden as I do more home automation stuff. Beyond the outdoor weather data a site like WU wants, it has sensors for indoor temp and air quality so I can set up some HVAC routines or monitor if my chest freezer for getting too warm. It has a soil moisture sensor and one day I'd like to set up an automatic irrigation system for my garden. I can expand the connected sensors as I think of more things I want to try.
I think I could have accomplished a lot of the same with bare sensors like you're suggesting but I wasn't ready to take that on. That was too much learning curve for me right now.
Does anyone have any suggestions for a personal weather station? Especially one for a beginner?
I’ve had a Weatherflow Tempest [1] for a few years and love its simplicity. No moving parts, solar and rechargeable battery powered.
Their platform is primarily mobile app based but they have a comprehensive API should you want to play around.
[1] https://tempest.earth/
[1] https://tempest.earth/
Thirded. I connected my stations' local UDP broadcasts to Prometheus push/VictoriaMetrics:
https://github.com/willglynn/tempest_exporter
Their central web API is nice too (and the tool above can extract metrics from it) but the local, offline data access is what got me in the door. Tempest could shut down their services tomorrow without breaking my setup.
https://github.com/willglynn/tempest_exporter
Their central web API is nice too (and the tool above can extract metrics from it) but the local, offline data access is what got me in the door. Tempest could shut down their services tomorrow without breaking my setup.
+1. I have Tempest devices at a couple of sites. They were easy to set up, and they've worked well. The API is straightforward, too, so it was easy to write a Python script to sync my data to a local SQLite database and keep it up to date. (If you're interested: https://github.com/tmoertel/tempest-personal-weather.)
Second this; I love mine. And the base station has a UDP port you can listen to for all the real-time data, making integration with Home Assistant and the like pretty straight forward and 100% local.
Hmmm, at first glance Tempest isn't currently compatible with CWOP.
Edit: Nevermind. It looks like Tempest themselves sends the data to NOAA, so it's already taken care of. Nice.
Edit: Nevermind. It looks like Tempest themselves sends the data to NOAA, so it's already taken care of. Nice.
Just did a quick search, looks great; easy to set up, and cost <$500. What's the maintenance like? I see it's solar powered, provided not in complete shade https://help.tempest.earth/hc/en-us/articles/4414460312603-P...
None. I’m three years in with mine in Denver so it deals with hot summer, frigid winter, random hail, occasional high winds.
I’ve never had to touch it.
I wish more home technology was so painless.
I wish more home technology was so painless.
From Dec to Feb mine is shaded by trees (sun doesn't get to high altitude here in Winter at 42N). The battery got pretty far down, but the unit went into low power mode and kept working fine, as far as I can tell. It's back up to full battery since March. I may move it up to a higher and sunnier location.
Put mine up in January 2021, zero maintenance, and it's been working great the whole time. Hopefully I haven't jinxed it. :)
What's your weather like?
Cyclonic (Hurricanes), baking heat, extended sub zero cold .. ?
I have an interest in stations that can last a long time in bad weather.
Cyclonic (Hurricanes), baking heat, extended sub zero cold .. ?
I have an interest in stations that can last a long time in bad weather.
Los Angeles area, so the only issue here is heat, and even that is usually only a few bad weeks a year. The Tempest app tells me the hottest day was when it hit 109F the first week of September 2022, and that whole week had highs above 100F.
Cheers for that.
It can get a little warmer here: http://www.bom.gov.au/climate/dwo/202402/html/IDCJDW6136.202... and even more so in other parts of the state: http://www.bom.gov.au/climate/dwo/202402/html/IDCJDW6079.202...
( 100F == 37.7C , 109F == 42.7C )
It can get a little warmer here: http://www.bom.gov.au/climate/dwo/202402/html/IDCJDW6136.202... and even more so in other parts of the state: http://www.bom.gov.au/climate/dwo/202402/html/IDCJDW6079.202...
( 100F == 37.7C , 109F == 42.7C )
is it accurate? the rain and wind sensors with no moving parts seem like they'd be a significant compromise over transitional moving anemometers, etc.
Ultrasonic anemometers are generally better than cup anemometers, especially if regular maintenance is not guaranteed.
I’m not sure about ran sensors.
But moving parts are generally bad for reliability.
I’m not sure about ran sensors.
But moving parts are generally bad for reliability.
I have a Davis Vantage Vue. It's a nice station, but requires a proprietary data logger to get a serial connection which I loathe. There's tons of good weather station nerd info on wxforums[0].
weewx[1] software supports sending data to CWOP and has drivers for the Davis station among others.
Bonus: you can also send data to CWOP over the ham bands via aprs, and all your data shows up on aprs.fi[2]
[0]: <https://www.wxforum.net/>
[1]: <https://weewx.com/>
[2]: <https://aprs.fi/>
weewx[1] software supports sending data to CWOP and has drivers for the Davis station among others.
Bonus: you can also send data to CWOP over the ham bands via aprs, and all your data shows up on aprs.fi[2]
[0]: <https://www.wxforum.net/>
[1]: <https://weewx.com/>
[2]: <https://aprs.fi/>
I have a Vantage Pro2 with the WeatherLink Live. I'm really happy with it. It has a pretty simple API you can hit on the local network, and it can push to weatherlink.com. It's a bit on the pricey side, but I live in a fairly rural area, so it's nice to have my own data. Plus, I'm on a private airstrip, so other pilots can use see it.
It's been a while, but with a little bit of solder and a slightly-janky connector, you don't need the proprietary dongle to connect to a serial connection (at least with a Vantage Vue from ~2010).
The Vantage Vue hardware is solid; much better than the cheaper kits out there.
The Vantage Vue hardware is solid; much better than the cheaper kits out there.
I spent some time building a data logger when I first got my vue, but they updated hardware circa 2012 specifically to thwart this, it seems[0]. It was a frustrating discovery: had to spend an extra $100 for something that can be made with $2 worth of parts :(
[0]: <https://www.wxforum.net/index.php?topic=10721.275>
[0]: <https://www.wxforum.net/index.php?topic=10721.275>
That's really unfortunate.
Maybe work backwards from
Citizen Weather Observer Program (CWOP) Information: http://www.wxqa.com/cwop_info.htm
If you're going digital, there are several suggested backbones, eg:
Legacy (oops): https://controlbyweb.com/legacy/x320m/
Current model ($450 US): https://controlbyweb.com/accessories/davis-suite-controller
which is a hub that uploads data from attached sensors, which in turn has details of compatable sensors:
Legacy manual: https://controlbyweb.com/wp-content/uploads/2024/01/X-320_um...
Current Integrated bunch of sensors on a stick ( $595 US ): https://controlbyweb.com/accessories/sensor-suite
The CWOP community likely has forums and users experienced in matching suggestions to new member questions.
Depends a lot on your skill | interest levels - the controlbyweb path is one option, and that's $450 pluse extra for sensors ( tempreture, wind speed | direction ), air pressure, etc.
All in ~ $1,100 (US) - controller + base sensors + shipping + some wiring and fitting at your site.
There are likely other products and various levels of kit build, solder your own, etc.
The Complete Tempest Weather System https://shop.tempest.earth/collections/featured/products/tem...
comes in at $339 (US) which is cheaper to get started and promises much - build quality, robustness, etc would have to be compared | asked about.
Citizen Weather Observer Program (CWOP) Information: http://www.wxqa.com/cwop_info.htm
If you're going digital, there are several suggested backbones, eg:
Legacy (oops): https://controlbyweb.com/legacy/x320m/
Current model ($450 US): https://controlbyweb.com/accessories/davis-suite-controller
which is a hub that uploads data from attached sensors, which in turn has details of compatable sensors:
Legacy manual: https://controlbyweb.com/wp-content/uploads/2024/01/X-320_um...
Current Integrated bunch of sensors on a stick ( $595 US ): https://controlbyweb.com/accessories/sensor-suite
The CWOP community likely has forums and users experienced in matching suggestions to new member questions.
Depends a lot on your skill | interest levels - the controlbyweb path is one option, and that's $450 pluse extra for sensors ( tempreture, wind speed | direction ), air pressure, etc.
All in ~ $1,100 (US) - controller + base sensors + shipping + some wiring and fitting at your site.
There are likely other products and various levels of kit build, solder your own, etc.
The Complete Tempest Weather System https://shop.tempest.earth/collections/featured/products/tem...
comes in at $339 (US) which is cheaper to get started and promises much - build quality, robustness, etc would have to be compared | asked about.
The San Diego Rowing Club has an Ambient Weather WS-2902 weather station and live camera feed [1] and the details of how we collect, process, and present the data. [2] If one clicks on the static image you'll get a live feed. The static image is updated frequently and is a better mobile experience. The AW API can be problematic at times so later this summer I'll reconfigure the station to do a GET request to a local device and process the data without a cloud dependency.
1. https://www.sandiegorowing.org/weather/
2. https://www.sandiegorowing.org/weather/tech/
1. https://www.sandiegorowing.org/weather/
2. https://www.sandiegorowing.org/weather/tech/
I have Ambient Weather equipment at multiple locations and it works great. Ambient’s own site and software is also very good lately.
My dad and I both have Davis Vantage Pro 2s.
https://www.davisinstruments.com/pages/vantage-pro2
His has been working for 20 years now with no problems. There are cheaper ones no doubt, but the ability to add senors like a UV and solar wattage are nice. They also have sensors for soil moisture, and leaf transpiration, but it doesn't rain enough in norcal for those.
At some point, Davis changed the data logger, so I had to buy some USB adapter from some guy in Australia to unlock it, but it works fine. I don't remember the guy's name, or if it's still needed.
I use weewx and a raspberry pi to log everything. Integrated my PurpleAir into weewx and calculate AQIs as well.
http://weewx.com/
https://www.davisinstruments.com/pages/vantage-pro2
His has been working for 20 years now with no problems. There are cheaper ones no doubt, but the ability to add senors like a UV and solar wattage are nice. They also have sensors for soil moisture, and leaf transpiration, but it doesn't rain enough in norcal for those.
At some point, Davis changed the data logger, so I had to buy some USB adapter from some guy in Australia to unlock it, but it works fine. I don't remember the guy's name, or if it's still needed.
I use weewx and a raspberry pi to log everything. Integrated my PurpleAir into weewx and calculate AQIs as well.
http://weewx.com/
I have an Ambient Weather WS-2902C feeding to CWOP out of the box. It also lets you configure custom endpoints (REST based), so I have the data feeding into my own system and logging into a db that I can use however I want.
Inside its esp8266 based. Currently sells for less than $200.
Inside its esp8266 based. Currently sells for less than $200.
I also have a WS-2902, and years ago I tried to publish to CWOP. It was not easy without installing a number of Python dependencies and running your own process to constantly poll the weather station and then publish to CWOP. If I remember correctly, Ambient would publish direct to Weather Underground at the click of a button. Alas, WU has sort of deteriorated over the years. My information here is 4-5 years old.
Check the FAQ, mine definitely is sending to AmbientWeather and CWOP without the use of another computer.
https://ambientweather.com/support/ws-2902-weather-station-s...
https://ambientweather.com/support/ws-2902-weather-station-s...
Thanks, will do.
I have a WS-2902D and wasn’t aware it could do either of those things! Do you have a link you could share on how to set it up?
You have to use their WeatherBridge app to configure it. They have some info for CWOP on their FAQ site [1]. I believe that same app is where you set up your custom endpoints (it's been a few years since I did it).
[1] https://ambientweather.com/support/ws-2902-weather-station-s...
[1] https://ambientweather.com/support/ws-2902-weather-station-s...
I have two KestrelMet 6000s, build quality seems very nice and I’ve been happy with them. The one stopped connecting for some reason for a little bit, but it’s been fine since, think the battery just got drained with so many cloudy days.
Something from Davis Intruments or Kestrel would very good. But they are not cheap.
Weatherflow Tempest hands down is the best consumer station out there today.
It would be great if this could be extended with air pollution data. Combined with weather data like wind speed and wind direction, the data could be of great use in forecasting for example air pollution from wildfires.
There are some platforms for air pollution like OpenAQ [1] or sensor.community [2] but as far as I know none really has such an open API like weather.gov and allows individual contributions.
[1] https://openaq.org/
[2] https://sensor.community/en/
There are some platforms for air pollution like OpenAQ [1] or sensor.community [2] but as far as I know none really has such an open API like weather.gov and allows individual contributions.
[1] https://openaq.org/
[2] https://sensor.community/en/
Google Maps AQI plot is very very good
As opposed to purchasing a personal weather station, is there anything in terms of building your own weather station (buying a raspberry pi and sensors)?
https://imgur.com/a/zHAcpzX
I built my own using sensors from Sparkfun (also available elsewhere, it’s a pretty standard kit). It uses an ESP32 to read the sensors and submit the data to MQTT, then Node Red on a Pi to push the data to InfluxDB, and finally Grafana for visualization. I love having direct control over everything, and since I built the whole system I can easily expand it with other sensors like the Geiger counter in my garage (luckily that’s a boring sensor to watch) and simple displays:
https://imgur.com/a/FiygpoS
I built my own using sensors from Sparkfun (also available elsewhere, it’s a pretty standard kit). It uses an ESP32 to read the sensors and submit the data to MQTT, then Node Red on a Pi to push the data to InfluxDB, and finally Grafana for visualization. I love having direct control over everything, and since I built the whole system I can easily expand it with other sensors like the Geiger counter in my garage (luckily that’s a boring sensor to watch) and simple displays:
https://imgur.com/a/FiygpoS
The YT channel Explaining Computers has a few videos building a completely DIY weather monitoring system over time. I know he's done an anemometer and just recently a fully custom rain gauge!
Do you have a link?
I wonder what the impact on accuracy will be. I know a few people who collected rainfall amounts and wanted to share that information with some org (can't remember who) only to be turned away because they didn't use the one approved collection device. The approved collection device was expensive and not attractive, but they were told that all the devices had to be identical to make sure the data collected was accurate and that was for something no more complicated than a measuring cup. What happens when the cheap Chinese weather station I get on amazon gives wildly different data from every other cheap Chinese weather station people got on amazon?
probably they are hoping for some kind of statistical covergence as the number of samples in an area goes to infinity?
What a fascinating topic I knew nothing about. I am so grateful for HN users that share their knowledge so generously, unlike some weather companies. I've never heard anybody mention the obvious fact how s**y all the weather and pollen companies are.
One thing I didn't see anybody else mention, is your local Air Force unit does an expert job at delivering daily commercial-free weather reports.
One thing I didn't see anybody else mention, is your local Air Force unit does an expert job at delivering daily commercial-free weather reports.
I think if weather.com or accuweather need my data they can pay for it.
Weather.gov
Isn't the point that by sharing with weather.gov (which I personally think is a great idea), you're in effect giving it to weather.com?
I wouldn't say you got the entire point of this story but yes, that is a valid domain name.
I have some remote forest land I would love to set some sensors up on. I just need to know exactly what to buy to gather the data, power it with solar, and send the data back via cellular signal.
Unfortunately it's too remote to have any sort of internet signal beyond cellular at the moment.
I'd appreciate any and all ideas folks have, assuming I'm willing to drop a few grand without blinking.
Additionally, it's in an area with wildfire smoke part of the year and id love to have AQI monitoring.
Unfortunately it's too remote to have any sort of internet signal beyond cellular at the moment.
I'd appreciate any and all ideas folks have, assuming I'm willing to drop a few grand without blinking.
Additionally, it's in an area with wildfire smoke part of the year and id love to have AQI monitoring.
Hi yareal - I am starting to do some work in this area. I would like to discuss. See contact in my about
I've long had a theory that aggregating these weather sources to provide higher and more accurate models could create serious alpha for commodities and energy trading. Especially in nodal and ancillary markets.
I'm sure there are people here working at funds that leverage weather data to do exactly that. It would be a really cool job.
I'm sure there are people here working at funds that leverage weather data to do exactly that. It would be a really cool job.
Am I the only one wondering why a governmental website like weather.gov is not responsive and therefore not accessible?
A new one is on the way.
https://beta.weather.gov/
https://beta.weather.gov/
You may also checkout https://weatherxm.com
There's a similar project to aggregate rain stats at https://rainlog.org/
Most participants are located in Arizona (project is run by University of Arizona), and kind of makes sense because rain is important in the desert.
Most participants are located in Arizona (project is run by University of Arizona), and kind of makes sense because rain is important in the desert.
This is an absolutely fascinating topic I never knew about. I am grateful for HN users that share their knowledge so generously, unlike those weather companies.
One thing I didn't see discussed – your local Air Force unit does a very, very detailed and commercial-free weather reports daily.
One thing I didn't see discussed – your local Air Force unit does a very, very detailed and commercial-free weather reports daily.
Semi related, why does it seem like the forecast, current temp, and historical temp, are always 10c lower than reality in summer? Happens across a few locations I monitor, both urban and rural.
I’ll often read 41c outside, 35c in shade, but the weather says it’s currently 29c.
I’ll often read 41c outside, 35c in shade, but the weather says it’s currently 29c.
What's the chances home weather stations have some bias in their signal (e.g. likely to be mounted on a hot roof) and that these data points get mixed into a larger corpus and someone ends up drawing incorrect conclusions from that corpus? Hopefully not high?
Absolutely they do, but that bias can be detected and weighted. More is better.
Even the Galaxy S4 phone was a beast of a weather station: onboard barometer, thermometer and humidity sensor! All can be bias corrected and useful.
I used to own S4 phone and it's one of the best smartphone ever. Actually even as the owner never knew that it has thermometer and humidity sensors built-in ten years ago that even most of the USD1K+ phones nowadays do not have these very useful sensors [1].
[1] What You May Not Know About GALAXY S4 Innovative Technology:
https://news.samsung.com/global/what-you-may-not-know-about-...
[1] What You May Not Know About GALAXY S4 Innovative Technology:
https://news.samsung.com/global/what-you-may-not-know-about-...
Weather Underground actually has guidelines for siting a weather station so that this is consistent --- presumably there are similar guidelines for this program:
https://www.wunderground.com/pws/installation-guide
https://www.wunderground.com/pws/installation-guide
[deleted]
Does anyone know of a good source of historic weather data by lat/long or zip code? I’m personally interested in evaluating different places within the US to live based on my preferred weather conditions.
Zomato in India started a crowd-supported weather system
https://www.weatherunion.com/
https://www.weatherunion.com/
I would like to see historical high and low data for every day and location on Earth for the past 100 years. Does anyone have a dataset containing this information?
Here’s a guide on exploring 120 years of global weather measurements with Clickhouse. I enjoyed following this guide over a weekend. Also it gives you a sense of the dataset, even if you’re not interested in the clickhouse part.
https://clickhouse.com/docs/en/getting-started/example-datas...
ERA5 [1] will get to back to 1940. So not exactly 100 years but darn close.
[1] https://www.ecmwf.int/en/forecasts/dataset/ecmwf-reanalysis-...
[1] https://www.ecmwf.int/en/forecasts/dataset/ecmwf-reanalysis-...
You greatly overestimate how long systematic weather data collection has occurred across even first-world countries, nevermind the rest of the globe. There are certainly many, many places _right now_ where daily weather info isn't logged
"every location on Earth" is asking sort of a lot, but you might start at https://climatedataguide.ucar.edu/climate-data/global-temper...
So for all those with weather stations at home do you have recommendations for a all in one physical weather station?
Does anyone know what type of contractor I could call locally that installs these things?
What are some good datasets to combine with weather data based on a geolocation?
IS there any equivalent for the UK, would anyone want my weather data?
Anyone around the world can help Windy.
http://stations.windy.com/
http://stations.windy.com/
The Met Office has WOW: https://wow.metoffice.gov.uk/support
Wasn’t this basically what Dark Skies did?
For context:
The governments of the world provides these big weather companies (weather.com (cough IBM), Accuweather (cough IBM cough), etc) a metric shit ton of their data completely for free (by law) including data transfer. These are things like radar, satellite, ground station data, forecasts, composite models, etc. These companies profit substantially on it, as in billions of dollars. You as citizens also can get this data completely for free as well! MADIS is a system the government is working on to make that data access easier by bringing together many of these systems together and removing the bureaucratic redundancy and abstracting out the aging infrastructure. This is literally terabytes of data per day you can grab with almost no questions asked. That data is then processed privately and resold and repackaged to the end user, and you probably interact with this privatized data the most.
The frustration I have much of the additional "value" these weather data brokers provide is by linking up with each other with data contracts. These private companies have a much much higher detail on the ground than the government by being able to partner with companies that make common internet-connected personal radar stations and reselling that data to each other. The government doesn't have that privilege to buy limitless data. NOAA/NWS, for example, is extremely underfunded so if they had to privilege to buy it they probably couldn't come to an agreement to buy it. As a result, they can't use that data to improve the accuracy of alerts/warnings/forcasts, the same exact tools that the big weather companies make all their money from. It's a shit cycle and totally unfair IMO.
So please contribute if you can!!
Sidebar: I'm a founder of a self-bootstrapped startup to build a better weather data broker that doesn't cost an arm and a leg. If that's something you are passionate about solving, feel free to reach out :)