Shutting Down the BGP Hijack Factory(dyn.com)
dyn.com
Shutting Down the BGP Hijack Factory
https://dyn.com/blog/shutting-down-the-bgp-hijack-factory/
62 comments
The lessons learned section of the article hints at orgs being way too permissive or unresponsive when bad behavior occurs. The thing is though, this isn't a world where that kind of softness and leniency makes any sort of sense.
It makes sense from a not-wanting-to-get-sued-for-breach-of-contract standpoint.
The contract at IXPs almost universally includes a phrase like "will not engage in fraudulent announcement of routes", but proving that happened to the satisfaction of a non-profit's board of directors is difficult. You really have to have completely collinear anatidae.
The contract at IXPs almost universally includes a phrase like "will not engage in fraudulent announcement of routes", but proving that happened to the satisfaction of a non-profit's board of directors is difficult. You really have to have completely collinear anatidae.
> collinear anatidae
I'm stealing this phrase, please and thank you.
I'm stealing this phrase, please and thank you.
Why? It sounds incredibly pretentious. I think most people would appreciate Plain English [0].
[0] https://en.wikipedia.org/wiki/Plain_English
[0] https://en.wikipedia.org/wiki/Plain_English
>>> collinear anatidae
> sounds incredibly pretentious. I think most people would appreciate Plain English [0].
I can't speak for most people, but I didn't know what it meant, learned it after reading it, and found it quite appropriately funny and witty. I also learned a new word. There was nothing I found pretentious about it.
I think a helpful and reasonably objective criterion in deciding this might be whether the use of the phrase was (1) necessary to send the correct message, (2) required to understand the received message, and (3) liable to send an incorrect message. If #1 = no and #2 = yes, or if #3 = yes, then you should probably avoid using it. In this case it's #1 = no and #2 = no and #3 = no, so it's fine.
> sounds incredibly pretentious. I think most people would appreciate Plain English [0].
I can't speak for most people, but I didn't know what it meant, learned it after reading it, and found it quite appropriately funny and witty. I also learned a new word. There was nothing I found pretentious about it.
I think a helpful and reasonably objective criterion in deciding this might be whether the use of the phrase was (1) necessary to send the correct message, (2) required to understand the received message, and (3) liable to send an incorrect message. If #1 = no and #2 = yes, or if #3 = yes, then you should probably avoid using it. In this case it's #1 = no and #2 = no and #3 = no, so it's fine.
Because it's a lovely application of rhyming slang.
https://en.wikipedia.org/wiki/Rhyming_slang
https://en.wikipedia.org/wiki/Rhyming_slang
How exactly is it rhyming with anything?
Could you explain what it means? I read about rhyming slang, but can't figure this out
I'm guessing another way of saying "ducks in a row"?
https://idioms.thefreedictionary.com/have+ducks+in+a+row
https://idioms.thefreedictionary.com/have+ducks+in+a+row
Thanks.
Ducks in a row.
I fail to understand why there is no quick and official way to terminate such bad actors. Isn’t there a task force for monitoring and enforcing some rules? There should be a SPoC for every AS, available 24/7 so that such notorious players are kicked out immediately. We live in an age where everything can be traced and monitored and we allow BGP hijacking and other similar acts. Oh well, my romantic idea for a properly moderated network.
Who would be in charge of the moderation?
This very article describes the closest thing to that: NANOG is a collection of network operators, and they communicate with each other about the overall state and coordination of "the internet", which realistically is just "the total set of a lot of network operators agreeing to connect to each other".
Domain registration and DNS have much more centralization because there exists a root entity for the entire (public) system and an owner of each TLD: if an entity wants to remove example.com from existence, they can go to the .com operator and attempt to compel them to do that. For IP routing, you're talking about BGP between a vast number of different entities. By design, traffic can route a variety of ways between each point.
This very article describes the closest thing to that: NANOG is a collection of network operators, and they communicate with each other about the overall state and coordination of "the internet", which realistically is just "the total set of a lot of network operators agreeing to connect to each other".
Domain registration and DNS have much more centralization because there exists a root entity for the entire (public) system and an owner of each TLD: if an entity wants to remove example.com from existence, they can go to the .com operator and attempt to compel them to do that. For IP routing, you're talking about BGP between a vast number of different entities. By design, traffic can route a variety of ways between each point.
I had a conversation with a friend about this, and the outcome was the idea that BGP could be extended with functionality for this case. There needs to be a way to brand "negative" traffic or routes advertised with some sort of reputation system. In the event of a DDoS attack coming from an AS, you could have intra-AS weight for any given AS such that if an AS reports malicious traffic from a route, it's given a lower weight and traffic is less likely to route to that AS in favor of a less specific prefix. This would encourage any given AS to act in desirable ways, as their actions (or actions coming from within them, e.g. a customer of theirs being the source of a DoS attack) would have consequences.
You might be interested in the DOTS working group at the IETF.
https://datatracker.ietf.org/wg/dots/charter/
https://datatracker.ietf.org/wg/dots/charter/
How would that work in practice? If I compromise a pile of IoT devices running on Comcast users' networks, and use them to launch and attack, all Comcast users on their subnet get marked as uncool? And if we're marking them as "bad", doesn't that mean all of their BGP peers mark them as uncool and then the weights for their prefix are lower but still even, so routing still ends up the same?
The only way they'd be impacted would be if some networks didn't implement your bad-actor-prefix-weight-mod, and then we'd just be penalizing the people who don't use your system along with the attackers, since we'd be routing the bad traffic via their networks.
The only way they'd be impacted would be if some networks didn't implement your bad-actor-prefix-weight-mod, and then we'd just be penalizing the people who don't use your system along with the attackers, since we'd be routing the bad traffic via their networks.
You can see the impact of this kind of thinking in RBLs and blocklists - try to send email via your residential connection and you probably won't be able to.
Yes that’s true, most of the work and moderation is done via mailing lists (like in Linux dev or with IETF/RFCs) but when we’re talking about the infrastructure I would expect something more efficient. Sure someone must notice it and report it, but I don’t get why it takes so long. And just to avoid misunderstandings I’m not talking about censorship, but actual WTF situations as this one described in the post where it’s obvious what is happening with Bitcanal.
How do you decide whether the bad BGP announcement was an accident or malicious?
When someone accidentally announces an incorrect prefix, they're pretty up-front about it, and try to correct it as soon as possible. It's not something you really see that often either, as you don't change the prefixes you announce very often. Really only when you acquire new IP space. Otherwise, the router config is largely set-and-forget as long as the peerings are up.
So if someone does fat-finger a prefix announcement, it's followed by an email to the IX mailing lists and transit providers with an apology and a quick fix.
It starts to become apparent who made an honest mistake here and there and who's acting maliciously.
So if someone does fat-finger a prefix announcement, it's followed by an email to the IX mailing lists and transit providers with an apology and a quick fix.
It starts to become apparent who made an honest mistake here and there and who's acting maliciously.
When the announcer doesn't immediately retract it in embarrassment, continues to announce it, the RIPE/APNIC/ARIN WHOIS data for the block continues to show ownership info other than the announcer, and the announcer doesn't produce a legitimate business relationship and LOA for why they should be announcing that "new" /18. It's usually pretty obvious.
How many hours / days / weeks do you need to give the announcer to produce all that evidence, before considering the action as malicious and undoing the announcement?
Proving a business relationship sounds like a multi-day endeavor, whereas you typically want to undo the damage ASAP.
Proving a business relationship sounds like a multi-day endeavor, whereas you typically want to undo the damage ASAP.
a few days, maximum. If you're $SMALLISP and you have a /22 of space, and your upstream is $MEDIUMISP, you give a LOA (letter of authorization) to $MEDIUMISP allowing them to announce your prefix to their peers and upstreams. If $MEDIUMISP can't produce that LOA on demand and the ARIN/RIPE/WHOIS/APNIC/AFRINIC whois data, email/admin/technical contacts for the /22 owned by $SMALLISP don't respond with "yup that's our block and we are allowing $MEDIUMISP to announce it", something highly fishy is going on.
If you think a requirement to forge a paper document is going to stop a spammer who hijacks IP space...
One more count of fraud don’t mean a thing to these criminal operations.
One more count of fraud don’t mean a thing to these criminal operations.
I would not rely on it at all. At a certain point a bgp hijacker has to have some sort of physical interconnection to get to upstreams or an IX. Either equipment colocated at or near an IX, a transport circuit, something. Showing a forged LOA to somebody's colocation process is something that can be helpful in getting them to physically power off/disconnect a customer's equipment for abuse.
The short answer is no, the internet doesn't have any centralized authority to kick people off, or even to set rules, let alone monitor or enforce them. Each Autonomous System has the autonomy to set its own rules for who it connects to.
I have worked for a medium size ISP for many years (3 upstream Tier-1 provider, presence on 2 IXP) and we sometimes suffer from BGP hijaking.
We had developed a software that every hour checks the BGP prefix assigned to every peer and update the BGP filter automatically.
It takes some time to engineering it and develop but after then, it works like a charm.
That’s a 1-hour attack window though. It should be event driven, something where peers can securely signal changes as they happen
You can run every 15 minutes if you want.
It's a shame that the ISP is bankrupted, it could be a very nice product also to sell. I was in charge of this software, I can reproduce it easily (unfortunately I don't get the source code) and put online then everyone can use and improve.
It's a shame that the ISP is bankrupted, it could be a very nice product also to sell. I was in charge of this software, I can reproduce it easily (unfortunately I don't get the source code) and put online then everyone can use and improve.
Write a protocol for it then and demonstrate its use
Those are very expensive.
According to apnic[1], there are 15,000 update / day in 2016.
For small size ISP, the number is much lower and may be managible. But event driven can't be a general solution for larger isp.
[1] https://blog.apnic.net/2017/01/27/bgp-in-2016/
[1] https://blog.apnic.net/2017/01/27/bgp-in-2016/
15k updates a day? how big is the payload? ...sounds negligible to someone ignorant in BGP finer details.
So... what were they doing with the hijacks? Using it to evade IP reputation bans for spamming?
Based on what I know of their ASNs, yes.
If you hijack traffic you can do wire tapping and sniff all traffic not encrypted.
Send spam mostly.
If they have been bad actors for years why didn't they lose access earlier?
They were booted last year by a German IXP, DE-CIX after reported abuse. Nice to see a company with an abuse handle that works
http://seclists.org/nanog/2018/Jun/370
http://seclists.org/nanog/2018/Jun/370
Bitcanal sounds like an appropriately terrible name as it sounds like root canal... but for bits.
BGP really needs some more organized security, but that's nothing new, and i'm sure not super easy to organize.
BGP really needs some more organized security, but that's nothing new, and i'm sure not super easy to organize.
Not in its native language it doesn't.
* https://en.wiktionary.org/wiki/canal#Portuguese
* https://en.wiktionary.org/wiki/canal#Portuguese
And yet still being peered - https://bgp.he.net/AS197426#_peers
The HE site takes a while to refresh.
Good point, thank you. According to the article, Hurricane depeered them on July 9th; the looking glass says it was updated 5pm July 10th, so unless the 'updated' is incorrect, Hurricane started peering this bad actor again!
If you look up the BGP routes for a Bitcanal IP address (185.215.113.235) on HE's looking glass (https://lg.he.net/) it does not appear any routes are present. I believe HE's BGP page may still be out of date, or the peers are present but not active.
Last time I checked it took multiple days for it to update when routes disappear (I would assume they cache them for a while, in case it's just a temporary change).
For historical record, as of today, Hurricane isn't listed anymore, but Cogent is again, and GTT for ip6... Way to remain united against abuse... just, wow.
No comments about the cookie warning/opt-out modal on the page? Perhaps it's only visible in the EU?
The thing explicitly takes ~2-3mins to send a HTTP POST to each of their advertising partners saying you've opted out (and warns "Some vendors cannot receive opt-out requests via https protocols so the processing of your opt-out request is incomplete")... lovely.
The thing explicitly takes ~2-3mins to send a HTTP POST to each of their advertising partners saying you've opted out (and warns "Some vendors cannot receive opt-out requests via https protocols so the processing of your opt-out request is incomplete")... lovely.
Just came here to post exactly that! What a complete mess... and if you do follow the https link, lo and behold, they re-set the settings slider to the lowest level (advertising is OK), despite having set it differently previously. Took a fair while on the https link, but at least it says it worked...
We have RIPE and other IANA organizations that have routing objects in their databases with information about through which ASN certain classes are announced, there are also LOAs. GTT and Cogent are huge Tier-1 providers, why they do not check which classes their clients are announcing? Am I missing something here?
According to a post by Job on nanog they have been known to submitted false or fabricated IRR information to RADB and RIPE: http://seclists.org/nanog/2018/Jun/379
At the end of the day, BGP is a very trusting protocol and it requires keeping the neighborhood clean and clear. IMO providers should be filtering prefixes their clients shouldn't be announcing (al la BCP38) but keeping up on the various IP blocks being shifted around is a paperwork nightmare I'm sure.
At the end of the day, BGP is a very trusting protocol and it requires keeping the neighborhood clean and clear. IMO providers should be filtering prefixes their clients shouldn't be announcing (al la BCP38) but keeping up on the various IP blocks being shifted around is a paperwork nightmare I'm sure.
BGP is still very much built on trust and reputation... At a local ix level if you were to show up at an ix like the ams-ix and regularly announce prefixes you have no right to, your company name and AS# would quickly develop the reputation of a rancid turd.
The BGP authentication method doesn't seem very secure, so how do you know who you are trusting?
a properly implemented IX has MAC address filtering on ports. This can of course be spoofed. But there is also a level of security at OSI layer 1 for the physical fiber cross connect from an ISP's panel to the IX's panel.
For instance: If the IX is located on the 15th floor of the building. An ISP might be colocated on the 12th floor. Fiber XC from 12.501.P4.D4 (12th floor, row 5, rack 01, fiber patch panel 4, SC duplex port D4) to 15.201.P1.D4, then a fiber cable from D4 to an SFP+ port on the IX's switch. Unless somebody physically hijacks your fiber crossconnect and moves it (which would be noticed as hard down immediately) it's pretty hard to pretend to be another ISP, from the perspective of the switch fabric operator of the IX.
For instance: If the IX is located on the 15th floor of the building. An ISP might be colocated on the 12th floor. Fiber XC from 12.501.P4.D4 (12th floor, row 5, rack 01, fiber patch panel 4, SC duplex port D4) to 15.201.P1.D4, then a fiber cable from D4 to an SFP+ port on the IX's switch. Unless somebody physically hijacks your fiber crossconnect and moves it (which would be noticed as hard down immediately) it's pretty hard to pretend to be another ISP, from the perspective of the switch fabric operator of the IX.
There is no real need for BGP authentication: if you want to create a peer relathionship, it need to be configured on both routers, then there is a native trust relationship.
Many big ISPs are quite diligent about what blocks their downstream customers announce. If you're an NTT ip transit customer every new block you announce to them is vetted. They have a high degree of automation with IRR integration but also humans in the loop.
Any asshole can theoretically make a fraudulent LOA, but by producing one to an upstream a hijack factory opens itself up to criminal charges of fraud and forgery.
Any asshole can theoretically make a fraudulent LOA, but by producing one to an upstream a hijack factory opens itself up to criminal charges of fraud and forgery.
One note: not all IP addresses are handled by RIPE/ARIN/etc. There are legacy allocations which are not subject to any agreements and are the property of the owners.
http://www.bitcanal.com is down.
Did they host it in their AS and now their AS is unreachable?
Did they host it in their AS and now their AS is unreachable?
I'm getting a servfail when attempting to resolve their domain, according to DomainTools[1] the IP for the site was 185.215.113.235 announced from AS42229.
The ASN was mentioned in the article as being listed by Spamhaus ASN Droplist but wasn't mentioned earlier as one of the targeted ASNs.
Edit: reviewed the ASN more and it is the Ebony Horizon mentioned in the article, and it is only peered to BitCanal's primary AS197426, which is subsequently being de-peered, so I'd say that is the main reason bitcanal.com is down :) 1: http://whois.domaintools.com/bitcanal.com
The ASN was mentioned in the article as being listed by Spamhaus ASN Droplist but wasn't mentioned earlier as one of the targeted ASNs.
Edit: reviewed the ASN more and it is the Ebony Horizon mentioned in the article, and it is only peered to BitCanal's primary AS197426, which is subsequently being de-peered, so I'd say that is the main reason bitcanal.com is down :) 1: http://whois.domaintools.com/bitcanal.com
I timeout on getting an IP, but I noticed Opera/Chrome already flagged the url as unsafe despite the dns fail.
I'm not sure what to do improve the situation, but there definitely seems like a need for improvement.