How to Determine the Error of an Air Quality Sensor?(airgradient.com)

128 pointsby ahaucnx2 anni fa47 comments

How to Determine the Error of an Air Quality Sensor?

https://www.airgradient.com/blog/how-to-determine-error-of-sensor/

47 comments

When I read the HN article title, I knew which domain it'd be from before I saw it.

I have several AirGradient devices and they've been great, no hassle at all with my own software instead of cloud-connected anything. I just wish there were a really good CO monitor that was also no-pain.

Thanks for making a product that sells on its own merits instead of forcing ecosystem lock-in or trying to obscure what the thing is or does.

nsbk2 anni fa

Same here! Props to Achim from AirGradient, we need more entrepreneurs like them. I built some of their DIY Air Quality Sensors open design boards for myself and was so happy with them that now I build and give them as gifts to people I care about.

I integrate them in HomeAssistant via MQTT, see [1].

More in line with the discussed article, I correct the humidity and temperature by adding/subtracting a fixed deviation that I calculated from observation, but I would like to implement something based on [2].

[1]: https://github.com/nsbk/airgradient_mqtt

[2]: https://www.airgradient.com/blog/slr-temperature-example/

thebruce87m2 anni fa

> When I read the HN article title, I knew which domain it'd be from before I saw it.

Well of course - they are they only company in this space that consistently post these articles to HN, and as the article says:

> This is an Ad for our Own Product

ahaucnx2 anni fa

Just to clarify. We label the banner BELOW the article with this disclaimer. The article itself is purely written from a science perspective and not at all an advertisement.

xandrius2 anni fa

Same! I have one and been playing with it for a while but if they added CO reading + alarms, I would buy one device for me and one for each of my family members.

nicolaslem2 anni fa

I have a Fibaro CO sensor on my z-wave network which is relatively low-pain and lock-in free. In my opinion CO monitors are a bit different from say CO2 or VOC in that they should really show 0 ppm at all times. If they don't you want them to alert you first and optionally record the data/send it somewhere. That's why I am fine with this device being treated more like a smoke detector than an IoT device.

ahaucnx2 anni fa

Achim from AirGradient here. Thank you for your support. Very appreciated.

We are currently working on bringing a lot more features into our open-source firmware [1], for example mDNS, local API etc. to make it even easier for non-cloud connectivity. We are also working on an offical home assistant integration.

[1] https://github.com/airgradienthq/arduino [2] https://www.airgradient.com/documentation/local-server/

guiambros2 anni fa

Thank you for everything you've done. I have a few AirGradients, and they are rock solid.

Native Home Assistant integration would be fantastic. I already have mine connected using ESP32, and it works well, but setup is a bit finicky. Would be great to have something easily flashable.

ck22 anni fa

I really love the spirit of AirGradient and their products.

However they are shooting themselves in the marketing foot by not doing free datastorage for life for their users and therefore competing with PurpleAir which is the closed premium they are competing against.

Google has chosen PurpleAir for their map overlay for air quality, it shouldn't be that exclusive but the reason why is the sheer number of installs.

compare for your area:

https://map.purpleair.com/

vs the emptiness of

https://map.airgradient.com/

I keep telling them they need to just GIVE FOR FREE their units to all public universities in the United States with free data for life reporting, it will change their entire company's presence in the USA and maybe even make Google add them.

Right now many universities, firehouses, stadiums, etc. etc. use WeatherStem for local weather recording/reporting. You know what WeatherStem does NOT have? Air Quality recording/reporting. Imagine if AirGradient gave their units to all the major WeatherStem sites that already have power and internet setup.

look at the pulldown for the WeatherStem open-source data export and imagine AirGradient added for air quality to ALL those hundreds of universities

export pulldown https://forsyth.weatherstem.com/data

breakout (https://forsyth.weatherstem.com/)

riedel2 anni fa

Our lab build and characterized one of the first low cost mobile PM sensing devices and worked with a major manufacturer of reference devices. I think the article is partially over simplifying. This is because also the reference devices do some tricks to derive mass concentration. Real mass concentration can only be measured at a low frequency by using actual filters that absorb the PM in the correct size category. Optical and beta attenuation devices are actually calibrated. The question is if the translation of absorption and count is actually so stable against mass. If you use other types of sensors like nephelometers you can for sure get different readings, however: are they really more wrong? Particularly if due to the measurement pricinciple a higher resolution is in the data. If you would want to measure mass concentration you would need to compare against real mass concentration and aggregate values. I am not sure if high resolution RSME is really helpful as a metric here. Constantly in place calibrated low cost sensors are often quite good. But this does not tell so much about their quality. In the end it is what you want to do with the sensor. For regulatorybpurposes (measuring e.g urban background) the placement of the sensor is often more important than the accuracy. If you want to discover relative and transient effects a well calibrated network of dense low-cost sensors might be a fit. In the end you cannot even easily translate exposure to harm, because it the material of your PM very likely will also matter (we are just assuming the typical mix in most cities when deriving limits)

ahaucnx2 anni fa

These are all excellent points that demonstrate the complexity of air quality measurement -especially for PM.

We are planning many more articles like this one and hopefully step by step we can cover some of these aspects as well.

doodlebugging2 anni fa

I appreciate the focus of the article and the explanations laid out for us. I would like to point out though that the two photos used as the "Less Polluted" and "More Polluted" examples may not fit the descriptions.

Less Polluted shows a point in a city after a rain shower. The passage of a storm front tends to leave the air behind it more clean that it was before it arrived. That example makes the most sense.

More Polluted does not make sense when you look at the photo since it obviously shows a storm front making it's way across a city, maybe the same one in the "Less Polluted" example. That storm is cleaning the air as it passes and the buildings in the foreground look like the storm has already passed. To me that means that the air from the photographer's vantage point should be clean already. It isn't possible to know whether you are looking east at dawn or west at sunset but either way that storm has passed, bringing clearer air to the city.

A better example would be a photo of that same city on a cloudless summer day when the air is choked with smog and those who care are wishing for a nice storm like the one pictured to sweep through and clean it all up.

ahaucnx2 anni fa

Yes that's a good point. We will look for more suitable photos to illustrate this better - ideally from the same city. We could probably use Chiang Mai, Thailand where our company of located and has wide annual swings of air quality.

stavros2 anni fa

I think they just thought the rain was smog.

seanmceligot2 anni fa

I'm glad to see there's an alternative to Airthings. The problem with them is they should be calibrated at least every year but they can't be, so they just deny that they ever go wrong and pretend the problem doesn't exist.

dawnerd2 anni fa

I’ve had mine for over a year and it displays basically the same values my other sensors show. Not sure it’s that big of a problem.

nox1012 anni fa

I hope it's okay to ask this question. For you that have these sensors, what actions do you take based on what they tell you? Open more windows when you're cooking? Put on a mask when you vacuum?

Just curious.

I've dreamed of some future perfect portable sensor like a star trek tricorder or something that would tell me if the air in my office, or shopping center has bad stuff in the air (chemicals, viruses, mildew, etc...) to some point that I can print out a reading and tell my office the building is legally unsafe. I haven't seen anything like that.

I have sensor at home but all it tells me is PPM and I rarely pay attention.

jbm2 anni fa

I live in Canada, where radon is an issue. The item I got from Airwave gave me a half year of data that was close enough to the safety limit that I decided to add radon mitigation to my home. I then lent it to a friend who decided to do the same.

I have multiple CO2 meters in the house that helped me determine that there were no major issues with airflow, and led me to consider other ways to improve our sleep quality. This was also kinda helpful for me during parties; I'm usually not thinking about opening the window during winter, but when the CO2 level passes 1200 or so, I'll usually open it a crack.

gausswho2 anni fa

What measurements from the Airgradient devices specifically indicate radon? I have both the outdoor and indoor one but struggle to get actionable evidence.

jbm2 anni fa

Ah, I have an Airthings Wave Plus (I thought it the parent was asking about air quality meters in general.). The information is in the app; I've never used an airgradient device.

quickthrowman2 anni fa

> I hope it's okay to ask this question. For you that have these sensors, what actions do you take based on what they tell you? Open more windows when you're cooking? Put on a mask when you vacuum?

I don’t get it either, there’s not much you can do with a residential AQI sensor without adding a bunch of other automation controls.

Commercially, these are used in building automation to do things like turn on the exhaust fans and make up air units in a parking structure when the CO levels go up, or to open an outside air damper in an AHU to let more fresh air in if elevated CO2 levels are detected.

I just don’t see the application for residential use. It’s just a PPM number, does anyone actually take action if the number gets too high? Anyone automate their actions?

SECProto2 anni fa

> there’s not much you can do with a residential AQI sensor without adding a bunch of other automation controls.

You don't need automation. I have a portable CO2 monitor (aranet4) and it gives me an indication of how effectively the air is being exchanged. I've learned that my desk location at home is ok during summer and winter, but when temperatures are moderate the furnace isn't running as much, and the CO2 will get quite high. I keep a window cracked when I work there and it's enough to bring it down to reasonable levels. I didnt know this information before getting it. Similar when I was visiting relatives over easter - the CO2 in the bedroom got up to 2500ppm wile I slept (old house, sealed tight but no forced air circulation). I opened the window for the rest of my stay.

Klathmon2 anni fa

I had a similar experience. I got a cheap CO2 sensor from AliExpress, I basically would open a window or the door to my office if it got over 1000 during the day. I don't consciously notice a difference, but it also measures pm2.5 levels and I was surprised at how high they get while cooking.

Then for shits and giggles I brought it with me to the inlaws and after turning it on there was shocked the CO2 levels were 4300ppm!

Suddenly the headaches they would get at home made sense, and they've started opening windows in the mornings to help circulate some air.

Royce-CMR2 anni fa

Super fast reply from someone else - Yes, I have a standing air filter unit. If PPM went up I’d kick it into max gear.

(Now I have it running on very low continuously and it keeps PPM down)

The unit is rated for spaces 2x my square footage so on max it makes a quick improvement.

For CO2, minimize cooking / burning and kick on the air to attempt circulation. Worst case leave for a bit. (Used to be very sensitive to CO2 levels, it’s getting better).

rinron2 anni fa

It taught me that most bedrooms dont have enough air flow, it helped me figure out just how much i need the window open or not closing the door which improved my sleep. Also helped me see the importance and notice the effect of having a air filter, and fine tune location and fan speed. I find that air quality is something you get used to and dont realize the effect it has on you unless you pay attention and experiment. its not something i pay attention to every day i could probably use it 1 week 2 times a year and have 90% of the benefit. That being said a lot of people probably live in better more well designed places with less issues than i have.

billfor2 anni fa

I have outdoor and indoor. For outdoor it was helpful for wildfires and knowing when to go outside. For indoors I sometimes pay attention to it. It gave me some surprising insights into cooking and the pollution it causes - so now I try to remember to turn on the hood vent all the time - even things like making waffles generate a surprising amount of pollution.

jddj2 anni fa

> Yet, you would probably go for the right sensor, true? That’s because this one has a lower relative error.

If the author is about, I have a feeling this should read left?

ahaucnx2 anni fa

I believe you talk about the section about normalized RMSE (nRMSE). The nRMSE on the right side is 6% and thus lower than the nRMSE on the left side with 66%. The lower the nRMSE the better. So I believe the article is correct.

However it's always important to not only look at one performance metric but a bundle, e.g. including R2 for the correlation to make a good judgement of a sensor quality.

We covered some of the basics in an earlier article [1].

[1] https://www.airgradient.com/blog/sensor-performance-precisio...

poulpy1232 anni fa

I recently bought one of their monitor, and I'm pretty happy with it. It was the only one that provided CO2, PM2.5 and VOC at my price point. They were also tested by the agency in charge of monitoring the air quality in Paris, so I can reasonably trust their monitors at the difference of most of the non pro ones. I only wish there was a possibility to store the data on a micro SD card.

m4x2 anni fa

Based on the title, I was expecting this article to have instructions for calibrating an air quality sensor, rather than just explaining different error types.

wezdog12 anni fa

It doesn't mention anything about calibrating

seanmceligot2 anni fa

I was looking for this too, "site:airgradient.com calibrate" brings a lot of results, so it seems like they support it, but I haven't dug in yet.

runjake2 anni fa

See: https://news.ycombinator.com/item?id=40030636

m4x2 anni fa

What you think the purpose of determining the error of an instrument is, if not to calibrate it?

ahaucnx2 anni fa

This post is part of a series of posts. We already wrote a few posts about sensor calibrations, e.g. how to develop a correction algorithm for temperature [1] or how to use R for making a linear regression [2].

In one of the next topics we will dive more into PM correction and the sensitivity to relative humidity.

If you want to know more about the science of air quality monitoring, we are also running webinars that you can watch on our reserach page [3].

[1] https://www.airgradient.com/blog/slr-temperature-example/

[2] https://www.airgradient.com/blog/using-r-for-sensor-correcti...

[3] https://www.airgradient.com/research/

Luc2 anni fa

Ok, so you compare your sensor to a reference sensor. But...

> If you compare two reference instruments, they should ideally measure exactly the same, namely the true pollutant concentration. But they don’t: A study from the US-EPA co-located two commonly used reference instruments (Met One BAM-1020 and Teledyne API T640) and found that, at certain concentration ranges, they diverged by over 25% from each other.

honk2 anni fa

This reference instrument variability is especially true of the BAM, which due to its measurement technology has a high "noise floor" at low concentrations, while remaining a fairly good measurement at high concentrations. I believe two identical BAMs next to each other can easily have more than 25% disagreement at the lower end of the concentration range.

The T640 also recently had its approved calibration algorithm(s) updated, to a new set which often reads about 7% lower. What does that mean for T640 reference data collected in the past? I don't know, and AFAIK neither does the EPA really.

All that to say that reference instruments are still instruments with limitations, even if from a regulatory perspective we treat them as "truth".

Havoc2 anni fa

Have experimented with this too - the accuracy is less of a concern to me than the fact that PM sensors require airflow and most achieve that via shitty whiny tiny fans. bzzzzzz

Closest "solution" I've found is the airsensors built into some airfilters. They seem to steal some of the main airflow rather than having a second fan. Obviously not quiet but at least fan size is decent so bit lower pitch

As for other measurements - for temp/humidity/pressure...any sensor will do. CO2 - the SCD41 are best I've found, BME680 and BME688 are hot garbage.

ploxiln2 anni fa

The noisy fan really varies by device. My Air Graident Basic DIY makes very little noise. I have another device I got earlier, a "Lonicera" (some chinese brand) and it looks like it has an extremely similar Plantower PM sensor inside, which is what has the little fan ... but it is often way noiser and rattle-y, the fan is clearly somewhat malfunctioning. I've taken it mostly apart, but it didn't seem worth the effort to go the last mile to fix or replace it.

beryilma2 anni fa

> BME680 and BME688 are hot garbage.

I agree 100%. What they measure (which is already very non-precise), only vaguely correlates with air quality and there are many factors that would throw off the readings.

Reading from their reference manual, one can already see from all the complicated expressions (regression) and numerical factors that they tried too hard to come up with something that looks like an air quality metric.

HankB992 anni fa

> BME680 and BME688 are hot garbage.

I've got a BME680 on a Raspberry Pi reporting the values it senses. I'm confused about how to interpret the air quality readings as far as what is good vs. what should be concerning. Is that what you mean by "hot garbage" or are you not satisfied with the accuracy readings themselves?

Havoc2 anni fa

The readings coming out of it are basically meaningless & only vaguely correlate with what's happening it the room

beryilma2 anni fa

They are just not accurate. 25% variability is just not an acceptable level of accuracy. And their results are very dependant on temperature and humidity.

ahaucnx2 anni fa

It really depenends on the PM sensors you use. For the Plantower 5003 we use in our monitors, you only hear a faint noise from the fan when you put it directly at your ear. However some other monitors need an extra fan in the enclosure and some of them can get noisy.

For CO2, the Sensirion SCD41 is a good sensor which uses photo accoustic principle. We found NDIR sensors using light spectometrie to be a bit more accurate like the SenseAir S8. Price wise the two are quite similar.

geerlingguy2 anni fa

Can confirm, I have about 8 gradients deployed now, including one in my studio with 31 dBa noise floor, and I can only ever hear the Plantower's fan with my ear nearly touching the device. I've never heard it day to day.

somat2 anni fa

I have to admit I was a bit disappointed that the conclusion was effectively diff your sensor with a reference sensor, I was hoping it was more like "here is how to make your own reference measurement".

Don't get me wrong it was a good article, I appreciate the math on how to make a meaningful diff.

anikra2 anni fa

Hi Anika from AirGradient here (the one who wrote the article).

Unfortunatly, we still depend on reference instruments to figure out the performance of a sensor. But you can find reference stations that publish their air quality data online. If you want to do you own reference experiment, check if you can find a closeby air quality station and compare your data with this one. But keep in mind that, if reference and monitor are a few (kilo)meters apart, the air pollution emissions in the surroundings will probably differ. So you might not be able to compare every peak and squiggle, but I would expect that you can see a general correlation if you compare a sufficient time period (at least a week).