Show HN: Sensor data and sky photos for ML: new take on a weather app (Android)(play.google.com)

3 pointsby jacobsheehy8 anni fa4 comments

play.google.com

Show HN: Sensor data and sky photos for ML: new take on a weather app (Android)

http://play.google.com/store/apps/details?id=com.allclearweather.android&120hn

4 comments

I make All Clear Weather (Android, US-only for now, see below for why) because I demand better of weather apps. Many of the top weather apps [1] do nothing but display sometimes-outdated weather data in poorly formatted tables. Frankly the top 20 look exactly the same. I'm making a new weather app because it's 2018 and the category is stagnant. I love the weather and I'm sick of weather apps that don't do anything interesting. Here is what I am passionately dedicated to bringing to a weather app to make it interesting and useful at the same time:

1) Your phone probably has a barometer in it! Most Android phones do because Google likes to use it as an altimeter to give you better indoor directions. So with iPhones also having barometers (for the iPhone health app), there are now billions of phones around the world are carrying barometers that next to no weather apps are using. Yeah WU and Dark Sky have a toggle, but what does it do? Do they talk about how they are using the sensor data? I don't think so. With All Clear you can at least see the sensor data from the weather sensors in your phone and know that it is being analyzed and processed into 'virtual weather stations' so it can be eventually included in the main weather models. You'll know this because you'll be able to see it happen in the app as I make progress. Here is a quick graph I made: https://imgur.com/gallery/IPj6w7N of Hurricane Florence's pressure recorded by phones in North Carolina a few weeks ago. It may take some imagination to see how the data could be useful (it needs cleaning, privacy protection, quality control, etc) but you can see the data in there and I know it can be extracted into useful numerical weather model inputs.

2) All the phones in the world have cameras. These cameras are taking photos of the sky and the weather all day long but nobody is using this data in weather forecasting. That's obviously because it is difficult to extract meaningful numeric weather data from a photo, but I believe it can be done. All Clear has a feature to let users send in photos of the sky with tags, and I am going to use this data as a training data set to a machine learning classifier. If it works, it should be able to automatically tag weather information in any outdoor photo - assuming the training dataset is good and my ML is good.

3) Current conditions in a lot of weather apps are out of date, since the source data is out-of-date. A lot of the US MADIS stations update only hourly, and they can miss updating in intervals when significant weather happens. Sometimes they update and sometimes they don't. But I think it's so frustrating to look outside and see it is raining while your weather app just happily denies the existence of that rain and tells you it's just partly cloudy. So I let you fix it, for yourself and for others nearby. If enough people all submit the same weather condition, contrary to the station reports, the app will start showing that condition as the Truth for current conditions in that area. (safeguards built in to protect against cheating).

4) I'm sick of the same UI in all the weather apps. I get that the UI in All Clear isn't perfect yet (it's a new project! and it's a hobby! and I'm not a UI designer!) but I am absolutely going to try new things in terms of how the optimal weather data can be displayed to a user to maximize quick-check understanding. It's too easy to be misled by the text, icons, and gridded numbers in most weather apps.

5) I am dedicated to open source. While the whole app isn't open source yet (it will be!) the main sensor code that does the barometric pressure data collection along with other sensors, is open source on GitHub: https://github.com/JacobSheehy/AllClearSensorLibrary. I would encourage other weather app developers to enable features in their apps to aide in the crowdsourcing of potentially very useful weather data that is otherwise idle in phones.

6) It's in the details. The US NOAA Forecast API that the app uses only returns its text forecasts with F and mi built-in as units. I wrote an algorithm (regexes mostly) to convert the text forecast units on-the-fly so you can be reading it and see it as C and km if you like, it's seamless. Well there is one error I know about, sometimes it will say "quarter of an inch of rain" which is a bit of a pain to parse, but I'm working on that too!

Another detail worth mentioning is the smart retry. During development I discovered that the API returned errors (or didn't return at all) and so rather than just showing the user a failure screen, the API does a nice backoff-retry to try really really hard to get the data for you. And if it can't do that? It knows that probably that forecast office or station is just down. So it changes the location parameters a bit and tries again, hoping to catch a different station forecast that would still be relevant to you. I don't have the numbers for how often it fails to load vs. other weather apps (I hope it's not worse!). But I know this: I put in some serious effort to try my best to show you the weather no matter what. On that topic...

7) Another feature of the app is that it does a lot of work to attempt to pick accurate icons. I find that the icons shown in a lot of weather apps are misleading at best when it comes to describing the weather. It turns out that it's really hard to show the user the right weather information in just an icon. So this is the main reason for the animations in the app: to make the visuals more correct and more engaging. There is logic in the app to not show the default icon that comes from the weather API: if it did do that, then a 30% chance of thunderstorms shows you just the thunderstorm icon. Most apps do this. But chances are? The weather will be just partly cloudy, no thunderstorms. So the app shows a partly cloudy animation with a faded, flashing lightning bolt to indicate the probability. Further work is being done in future updates to the app to make this even more accurate.

-----

Alright, there's what All Clear can do that other weather apps can't. Maybe it should be your next weather app but maybe it is missing a critical feature for you. What's that feature? What do you need or want in a weather app that nobody is delivering today?

All of that above featureset is free by the way. Part of the problem with not enough original and quality weather apps is that they each cost money to run and users are often unwilling to pay. So for All Clear it is free to use all the features of the app; You can pay $1/month if you like to extend the forecasts from 3 days to 6, and extend other features in the app like the number of visible weather pictures (from 10 to 100). The subscription will help pay for the servers that run all the custom weather data code as well as future development of neat weather experiments using smartphone sensors.

And a note about it being US-only. I tried to make this international from the get-go, and wasted months using IBM's API from WU that barely worked at all. Then they yanked the payment form off the web the week I was going to publish the app and buy their $400/month package. No longer an option. Now you have to wait for a phone call, might be weeks, and the prices are like $3,000 for the equivalent data (or something, hard to say). If you want to read some of the drama about this, check out this WeatherUnderground thread: https://apicommunity.wunderground.com/weatherapi/topics/weat... where people are describing the confusion as WU cuts off their keys and makes the service dramatically more expensive.

So the app is using the US NOAA data for free and I am investigating alternatives for international weather data.

[1] https://play.google.com/store/apps/category/WEATHER/collecti...

matt_the_bass8 anni fa

Cool idea! Would you mind clarifiying some questions?

1. Are you looking to do your own weather prediction or just better package the prediction results from existing weather models? I ask because weather modeling is hard and there are lots of PhDs working on continual improvement. They also use an huge amount of processing power to do this.

2. Are you planning on open sourcing the collected data too or do you hope to monetize the data?

jacobsheehy8 anni fa

Thanks!

1. I'm definitely just starting simple. My first goal that makes sense to target is: a) quality control algorithms for the data that make b) it possible to do data assimilation into WRF. I would not expect this project or myself to do the actual WRF data assimilation, that would probably require outside help.

For example, I know that this process can be done with the barometer data, but it is difficult to do this for a couple reasons. First is user privacy, most methods of using this data require a user id attached to the lat/lng and barometric pressure. This isn't great, it's a log history of user's location. So that's why I'd like to make 'virtual weather stations', which would be my first target.

Perhaps in the distant future this project could span its own predictions/models but I don't have a particular plan for that.

2. Well as I alluded to above the raw data is sensitive to user privacy, so the ideal system deletes that data after a few days of processing/QC control, hashing, etc to preserve as much as we can of the useful information while deleting the user's private location history. This makes the dataset a lot less monetizable, since fewer weather-industry folks who'd like to integrate with WRF would be interested in the data, given how far removed my final product would be from the source barometric pressures (virtual weather stations are a brand-new abstraction and they have not been tested).

I'd say that at first the data that I make in terms of 'virtual weather stations' may be an open API, but it depends on how much that costs to run. The project is supposed to make money from users buying it: it has a $1/month subscription optional to show users how to directly financially support the project. Any future sales of weather data would be a long way away and would require me to prove out its usefulness, something that hasn't been done with 'virtual weather stations' yet (but has with the raw barometer data, darn)

Great questions, thanks!

pstch8 anni fa

> Any future sales of weather data would be a long way away and would require me to prove out its usefulness, something that hasn't been done with 'virtual weather stations' yet (but has with the raw barometer data, darn)

Honest question, as I don't know much on the subject : why should future sales of weather data be a long way away ? Aren't there already end-users interested in low-volume API access ?

Also, just to clarify, by 'virtual weather station', you only mean devices that run the application, excluding other non-official weather monitoring systems ?