TuM'Fatig – Ads blocking with OpenBSD unbound(8)(tumfatig.net)
tumfatig.net
TuM'Fatig – Ads blocking with OpenBSD unbound(8)
https://www.tumfatig.net/2022/ads-blocking-with-openbsd-unbound8/
30 comments
So I've always been curious, does anybody have a writeup of how DNS servers like unbound etc are able to handle evaluating these massive blocklists? Like the algorithms and data structures they use to evaluate queries against these lists efficiently. My pfsense+pfblocker unbound instance has something like 400K hosts blocked and I've always been amazed that it doesn't even break a sweat scanning that many hosts with every query.
> My pfsense+pfblocker unbound instance has something like 400K hosts blocked
But it is not 400k hosts to search:
> 400K hosts
I bet 30% of those hosts are don't resolve anymore and another 30% resolve to domain parking/404 hosts.
Eg: see my comment about "TOP 10M SITES"
https://news.ycombinator.com/item?id=33150554
But it is not 400k hosts to search:
com
facebook
www
ad
google
ad
ads
notadswepromise
googletagmanager
tags
ads
> 400K hosts
I bet 30% of those hosts are don't resolve anymore and another 30% resolve to domain parking/404 hosts.
Eg: see my comment about "TOP 10M SITES"
https://news.ycombinator.com/item?id=33150554
I suspect parent meant the way the list is processed and how the structure resides in memory.
We'd optimally want, e.g., aaa.ai to resolve as fast as zzz.zx. If we just loaded the file for every query and ran down the list linearly we'd probably see a large disparity with the former being much quicker than the latter.
So, the question would be: how is it done in an efficient way? Is it loaded into a btree, is each domain broken into various parts, are certain domains prioritized, threading, compression?
Plus, there's the queries for dns updates, and how the results get integrated.
We'd optimally want, e.g., aaa.ai to resolve as fast as zzz.zx. If we just loaded the file for every query and ran down the list linearly we'd probably see a large disparity with the former being much quicker than the latter.
So, the question would be: how is it done in an efficient way? Is it loaded into a btree, is each domain broken into various parts, are certain domains prioritized, threading, compression?
Plus, there's the queries for dns updates, and how the results get integrated.
Some combination of tries, B-trees and Bloom filters would get you most of the way there.
I bet 30% of those hosts are don't resolve anymore
I've seen some evidence of this. I periodically dig all the blocked domains and noticed that some of them come and go so I assume people cycle them in and out to evade blocking. A handful of the domains just change the A record so I just block the apex domain which makes it a little harder to keep the block lists optimized but does reduce the size a little.
I should add that blocking the apex of some domains includes risk of dropping legit traffic but if I see more than {n} badware domains on the apex, I write it off as compromised or at very least poorly implemented and poorly secured.
I've seen some evidence of this. I periodically dig all the blocked domains and noticed that some of them come and go so I assume people cycle them in and out to evade blocking. A handful of the domains just change the A record so I just block the apex domain which makes it a little harder to keep the block lists optimized but does reduce the size a little.
I should add that blocking the apex of some domains includes risk of dropping legit traffic but if I see more than {n} badware domains on the apex, I write it off as compromised or at very least poorly implemented and poorly secured.
Here's Tony Finch's discussion of the data structures he's working on for ISC's BIND:
https://dotat.at/prog/qp/README.html
https://dotat.at/prog/qp/README.html
https://news.ycombinator.com/item?id=20148457
Not sure how out of date that is
Not sure how out of date that is
This is really cool, thanks for the write up. I did the same thing in my homelab but much simpler, my shell script only fetched from a few sources and converted it into unbound format so it was much shorter. And I didn't care about statistics, but the idea was the same.
Also, why use ftp instead of curl?[1]
1. https://curl.se/docs/manual.html#time-conditions
Also, why use ftp instead of curl?[1]
1. https://curl.se/docs/manual.html#time-conditions
My guess is that curl is a port/package while ftp comes with the base system.
True but I also run OpenBSD at home and curl is part of my common packages to install list in ansible. How I bootstrap all my nodes.
As others have mentioned, ftp that ships with OpenBSD is a bit more robust then its name suggests. https://man.openbsd.org/ftp
This is what I do. I was trying to figure out what pi-hole did one easy thing was it black-holed domains, so I got the same list put it on my dns relay box and there you go same thing on openbsd.
https://pgl.yoyo.org/adservers/
basicly all it is is
https://pgl.yoyo.org/adservers/
basicly all it is is
include: /var/unbound/etc/yoyo.conf
Glad you're finding the list useful!
Just for the record, but my list is available natively in a format for use with Unbound:
- https://pgl.yoyo.org/as/serverlist.php?hostformat=unbound
Remove the header with &showintro=0 and for plain text, use &mimetype=plaintext:
- https://pgl.yoyo.org/as/serverlist.php?hostformat=unbound&mi...
- https://pgl.yoyo.org/as/serverlist.php?hostformat=unbound
Remove the header with &showintro=0 and for plain text, use &mimetype=plaintext:
- https://pgl.yoyo.org/as/serverlist.php?hostformat=unbound&mi...
As a fun experiment one could set this up on a handful of VM's and leave the Unbound TLS port open or DoH if compiled with libnghttp2 to the world in addition to the graphs. I think it would be interesting to see how many clients each VM could handle. Maybe enable the rate limits in Unbound first. I'm half tempted to do this with a few of mine but I do not have the fancy graphs, just shell scripts to parse the logs which I guess could be written out as text files for nginx to serve.
Unfortunately for me the Unbound package on Alpine is not compiled for DoH, only DoT and I am too lazy to recompile things any more.
Unfortunately for me the Unbound package on Alpine is not compiled for DoH, only DoT and I am too lazy to recompile things any more.
In spite of my entire satisfaction with it, I had to ditch my OPNsense router... Its unbound ad filtering DNS is among the features I miss most from it.
Why did you have to ditch?
Bus factor. I had a very nice home setup (three 24 ports switches and a 6-port router) with multiple VLAN, GPON terminated on my own device (emulating the ISP's hardware), bonded Ethernet links, fully routed IPv6, sane firewalling, ad-filtered DNS, reverse proxy etc. - but I was the only person in my home (six kids + adults) to know how to maintain it. Anything happens while I'm away and Internet access stops. I borked an update and was offline for a couple of days - that was once too many for the users.
So we are back to the ISP's router, no VLAN and soon some Wireguard overlay for my hosting.
I'm sad about this: replacing infrastructure I control with the ISP invading my home, replacing tailor-made with generic consumer products, replacing native Internet with yet another overlay network... But MTTR rules my world.
So we are back to the ISP's router, no VLAN and soon some Wireguard overlay for my hosting.
I'm sad about this: replacing infrastructure I control with the ISP invading my home, replacing tailor-made with generic consumer products, replacing native Internet with yet another overlay network... But MTTR rules my world.
I just realized that OpenBSD unbound was likely named as a counter to ISC BIND. Similar to Sun Java and the Eclipse Foundation.
It is not really openbsd's unbound, it is nlnet labs unbound. openbsd just ships it with their operating system.
https://nlnetlabs.nl/projects/unbound/about/
but yes it was made to avoid a bind monoculture. and comes in two parts nsd is the authoritive name server. and unbound is a recursive name server.
https://nlnetlabs.nl/projects/unbound/about/
but yes it was made to avoid a bind monoculture. and comes in two parts nsd is the authoritive name server. and unbound is a recursive name server.
As somebody who spends some time on this, identifying common infrastructure and targeting that is the best bang for the buck: all kinds of stuff lives in the sewer that you never thought of. OTOH it does require some whitelisting, because all kinds of stuff lives in the sewer.
Be vary of using such sources.
Go to the source [0], make "Display: 100" and scroll to "1,351 to 1,450 of 10,000,000 websites".
You will find a bunch of bullshit domains clearly used for something nefarious (and some no longer exists), but to believe what those sites are as popular as spreadsheets.google.com? NO WAI
Tiny sample for the lazy:
[0] https://www.domcop.com/top-10-million-domains