Someone was breaking into Orange Spain RIPE account (and break their /12)(benjojo.co.uk)
benjojo.co.uk
Someone was breaking into Orange Spain RIPE account (and break their /12)
https://benjojo.co.uk/u/benjojo/h/r1zj333N4L6cF7P1xv
46 comments
For those like myself that are somewhat unfamiliar with what RIPE is:
RIPE is the European equivalent of ARIN (the North American regional authority of ISP addressing matters). They are sort of like the postal/zoning agents of the internet in that they oversee the distribution and management of IP address blocks. ISPs like Orange Spain have a RIPE account that they use manage their IP allocations, which is what was compromised.
RIPE is the European equivalent of ARIN (the North American regional authority of ISP addressing matters). They are sort of like the postal/zoning agents of the internet in that they oversee the distribution and management of IP address blocks. ISPs like Orange Spain have a RIPE account that they use manage their IP allocations, which is what was compromised.
The technical term for these is RIRs. Regional Internet Registries. There are five of them, ARIN (North America), APNIC (Asia Pacific), RIPE (Europe), AFRINIC (Africa) and LACNIC (Latin America).
The RIRs are delegated the numbers from IANA, and they in turn delegate numbers to LIRs, Local Internet Registries, such as an ISP or a maybe a large organisation which has vertically incorporated that functionality.
The RIRs represent, and are paid for by, the LIRs in their region and so to some extent they reflect local consensus, local culture, etc. This means in some sense RIPE's security policies reflect what European ISPs and similar wanted, for good or ill.
The RIRs are delegated the numbers from IANA, and they in turn delegate numbers to LIRs, Local Internet Registries, such as an ISP or a maybe a large organisation which has vertically incorporated that functionality.
The RIRs represent, and are paid for by, the LIRs in their region and so to some extent they reflect local consensus, local culture, etc. This means in some sense RIPE's security policies reflect what European ISPs and similar wanted, for good or ill.
> This means in some sense RIPE's security policies reflect what European ISPs and similar wanted, for good or ill.
Traditional routing has always been very, very open. Like a club where anyone is welcome and if new folks mess stuff up it just gets fixed.
Several IRRs used to allow RPSL changes by email to their systems with only maintainer auth and no verification of route ownership (e.g. you register with the IRR, then publish Google routes).
It’s all getting hardened nowadays with things like no more email, layers of verification, RPKI, etc.
It’s not necessarily what folks want but how this stuff had traditionally been handled. Thankfully it’s all changing because of events like this.
Traditional routing has always been very, very open. Like a club where anyone is welcome and if new folks mess stuff up it just gets fixed.
Several IRRs used to allow RPSL changes by email to their systems with only maintainer auth and no verification of route ownership (e.g. you register with the IRR, then publish Google routes).
It’s all getting hardened nowadays with things like no more email, layers of verification, RPKI, etc.
It’s not necessarily what folks want but how this stuff had traditionally been handled. Thankfully it’s all changing because of events like this.
Allegedly, Orange Spain's RIPE password was "ripeadmin", with no 2FA: https://twitter.com/Ms_Snow_OwO/status/1742666456058470739
> ... Orange Spain's RIPE password was "ripeadmin"
if it's a thing at Orange Spain and they're reusing that excellent scheme for several services (the excellent scheme being adding "admin" after the name), they're in for a world of hurt...
if it's a thing at Orange Spain and they're reusing that excellent scheme for several services (the excellent scheme being adding "admin" after the name), they're in for a world of hurt...
A company I worked for had “servername_db” as that DB server’s internal domain, then the credentials for the server were “servername_db_user” and “servername_db_password”. The argument was that it was behind a VPN and bastion server. My argument was that it was almost as easy to have a bitwarden account with 2fa and generate passwords that would never be cracked, and it would look really bad if the company was broken into and we couldn’t spend two extra seconds per login to secure it better and lose customer data.
That's pretty loose protection on 10s of millions of dollars of asset.
if it's true, it's ridiculous BUT the cost is just the cost of downtime. is anyone under the impression that they'd actually be out these IP addresses permanently? obviously RIPE would fix the issue and get the IPs returned
If this really lead to an entire country losing access to GitHub, Twitter, Canva, etc for four hours, I wager that's a pretty substantial economic loss. Of course Orange doesn't suffer this cost directly, but the downtime and the surrounding revelations/rumours about their security posture might cost them a lot of business in the medium term (even outside Spain).
I mean, it didn't affect the _entire_ country. I am in Spain and didn't notice (neither did anyone I know) until I saw the news. Logically it should only affect Orange customers, maybe transiting traffic from other ISPs?
Yes and no, while RIPE can help they can only up to a point. For example, the assets are transferred to someone , up to that point RIPE can help but if another transaction has followed then sorry but they can't do much
This isn’t the blockchain. All of that can be easily reversed
Talking from experience in such a case, malicious actor transfers IP range from entity A to intermediate entity B and then entity B sells that range to entity C.
Entity C obtained the asset on a perfectly legal way as they don't know and care how entity B got the asset. Nothing entity A can do then when they discover it, they can't get assets back plus RIPE also can't do anything since they only offer the tool for tracking such transactions they don't take part on them . If you want follow any legal process you want to be reimbursed from entity B and the malicious actor but you are not getting your IP range back
I don't know the legal system in Spain or the wider EU, but in the US for example stolen goods are stolen goods no matter how many hands they pass through. Washing them through a fence doesn't suddenly make them legitimate
A legal process can give them back the IP range but it's going to take a long time. They'll have to find other IP addresses until it's settled.
“I didn’t know it was stolen” doesn’t hold up well in court.
I didn't mention cost for that reason. It's just silly to put so much into assets and then not even put basic protections on them being accessed by others instead.
Maliciously changing a RIPE allocation would be like going into a pub and moving Geoff’s drink to one side so you and your mate can take his and Frank’s spot at the bar. While you may physically succeed at this, you will also find you don’t get served and you do get thrown out and barred.
I’ve always liked the professional level of collusion that makes the Internet work. One day it will all go wrong and we’ll have to set up something democratic, but until then the technocracy works well.
I’ve always liked the professional level of collusion that makes the Internet work. One day it will all go wrong and we’ll have to set up something democratic, but until then the technocracy works well.
This is one of several attacks that can be used to generate valid TLS certificates for domains you don't own. There are mitigations but they can all be defeated. This will persist until there is reform among internet standards groups, but they are controlled by the browser market, and they aren't responsible for generating the certs, so it's one of those "nobody is responsible so it will never be fixed" deals.
What "reform" do you think would help?
The problem here appears to be that the victim didn't do even a halfway decent job of protecting their RIPE account.
RIPE offers TOTP. TOTP is hardly the best possible security, because it can be phished, but assuming the people using it are competent (and why are your networking team incompetent?) that ought to be adequate. It seems as though Orange didn't bother to enable TOTP and indeed didn't even set a decent password.
The problem here appears to be that the victim didn't do even a halfway decent job of protecting their RIPE account.
RIPE offers TOTP. TOTP is hardly the best possible security, because it can be phished, but assuming the people using it are competent (and why are your networking team incompetent?) that ought to be adequate. It seems as though Orange didn't bother to enable TOTP and indeed didn't even set a decent password.
Nobody does this, but in theory dns-sec combined with a CAA record set to nothing would mitigate this in a way that can't be defeated, right?
In principle key pinning would be the counter measure, but that was so problematic it was removed from browsers. I think chrome still does static key pinning for their own domain, and many mobile apps do key pinning, so its not entirely dead, just mostly.
In principle key pinning would be the counter measure, but that was so problematic it was removed from browsers. I think chrome still does static key pinning for their own domain, and many mobile apps do key pinning, so its not entirely dead, just mostly.
Well the first problem with that scenario is it doesn't work for most of the internet. It requires CAs and domain owners to use DNSSEC, and most do not. It requires strict validation of DNSSEC, and most dns resolvers do not. It requires the use of CAA records, which most domain owners do not. And it requires CAs to properly implement CAA (it was found in 2017 that many did not, and Let's Encrypt messed up 3 million certs due to CAA in 2020). So, a single person with their own domain could do all these things, but the rest of the internet won't be doing it, so most of the internet will be insecure. That's not tenable, because that means the rest of the internet can be used in attacks that appear legitimate. Even most of the top 100,000 websites on the internet don't use DNSSEC and CAA, and if you asked them to, they'd ignore you.
But the second problem is it doesn't really solve the problem. Eventually you have to renew the cert, and when that happens, an attacker can spoof the IP of the validating source and have a valid cert issued. CA validation itself is flawed by design.
But the second problem is it doesn't really solve the problem. Eventually you have to renew the cert, and when that happens, an attacker can spoof the IP of the validating source and have a valid cert issued. CA validation itself is flawed by design.
Potentially yes: Site owners could set a CAA record like…
nocerts.example.com CAA 0 issue ";"
… telling CAs that they shouldn't issue certs for `nocerts.example.com` at all.
That being said, it would make things difficult when it's time to do a cert renewal: You'd have to update the CAA record, wait for it to propagate, do the renewal, and then set it back. And that's a window that could be exploited.
What could work is CAA, along with RFC 8657: Certification Authority Authorization (CAA) Record Extensions for Account URI and Automatic Certificate Management Environment (ACME) Method Binding (https://datatracker.ietf.org/doc/html/rfc8657). RFC 8657 extends the CAA record to say "Only this specific CA account may request certs for this domain" and/or "Only this specific validation method may be used when requesting certs for this domain".
nocerts.example.com CAA 0 issue ";"
… telling CAs that they shouldn't issue certs for `nocerts.example.com` at all.
That being said, it would make things difficult when it's time to do a cert renewal: You'd have to update the CAA record, wait for it to propagate, do the renewal, and then set it back. And that's a window that could be exploited.
What could work is CAA, along with RFC 8657: Certification Authority Authorization (CAA) Record Extensions for Account URI and Automatic Certificate Management Environment (ACME) Method Binding (https://datatracker.ietf.org/doc/html/rfc8657). RFC 8657 extends the CAA record to say "Only this specific CA account may request certs for this domain" and/or "Only this specific validation method may be used when requesting certs for this domain".
The title was confusing at first, with "was breaking into" I thought the attacker didn't succeed, but they did. Can the title be changed to something like "Someone hacked Orange Spain's RIPE account and broke their /12"?
We were affected by the downtime, suddenly our VPNs went down and we started troubleshooting if it was a switch sudden reboot, a physical problem, anything. Odd for some of our VPNs to randomly shut down rejecting all traffic but just at the endpoint. Then we started having some very specific problems with just some FTP addresses and couldn't figure out what was happening.
Until our CIO went to the bathroom and came back saying "Hey, these websites aren't working from our WIFI network but they're working fine from mobile data? Could it be a navigation problem from Orange?"
Then we realized it.
Until our CIO went to the bathroom and came back saying "Hey, these websites aren't working from our WIFI network but they're working fine from mobile data? Could it be a navigation problem from Orange?"
Then we realized it.
What evil could one do with this?
Some years back, Russia hijacked a BGP belonging to a major transit provider. For a few hours, international traffic was rerouted thru Russian networks where it could be cloned (like the NSA does in the states) and examined.
They could have been after something specific in the traffic but my unqualified guess is that it was a test or they were showing off.
They could have been after something specific in the traffic but my unqualified guess is that it was a test or they were showing off.
Curious to speculate about.
If you could temporarily redirect BGP at an arbitrary time, at diplomatic risk, what would be the best use of that?
Surely OpenNet/ClassNet/NIPRNet/SIPRNet/GWAN/JWICS aren't accepting BGP updates, where they even touch public network infrastructure.
And I can't think of anything outside those huge spheres that would be worth burning a capability like that on a lark.
Maybe bulk sensitive data transfer? I.e. some site-to-site backup?
Or you're just looking at the metadata of where-to-where, with the goal of finding future targets.
If you could temporarily redirect BGP at an arbitrary time, at diplomatic risk, what would be the best use of that?
Surely OpenNet/ClassNet/NIPRNet/SIPRNet/GWAN/JWICS aren't accepting BGP updates, where they even touch public network infrastructure.
And I can't think of anything outside those huge spheres that would be worth burning a capability like that on a lark.
Maybe bulk sensitive data transfer? I.e. some site-to-site backup?
Or you're just looking at the metadata of where-to-where, with the goal of finding future targets.
I went back and looked up the event. Rostelecom has been behind a few of these. I might be conflating two of them.
In 2017, Rostelecom grabbed financial & other network data for a few minutes. https://www.thousandeyes.com/blog/rostelecom-route-leak-targ...
In 2020 Rostelecom announced lots of networks they didn't own and got that traffic for an hour. Early analysis felt it was accidental. Not sure about later analysis. https://www.zdnet.com/article/russian-telco-hijacks-internet... https://news.ycombinator.com/item?id=22789754
But 2 months ago, Rostelecom grabbed a chunk of Apple's network (which I missed) https://cybernews.com/apple-network-traffic-went-through-rus...
In 2017, Rostelecom grabbed financial & other network data for a few minutes. https://www.thousandeyes.com/blog/rostelecom-route-leak-targ...
In 2020 Rostelecom announced lots of networks they didn't own and got that traffic for an hour. Early analysis felt it was accidental. Not sure about later analysis. https://www.zdnet.com/article/russian-telco-hijacks-internet... https://news.ycombinator.com/item?id=22789754
But 2 months ago, Rostelecom grabbed a chunk of Apple's network (which I missed) https://cybernews.com/apple-network-traffic-went-through-rus...
> Surely OpenNet/ClassNet/NIPRNet/SIPRNet/GWAN/JWICS aren't accepting BGP updates, where they even touch public network infrastructure.
If i understand, the article is not so much a typical bgp hijack but the attacker gaining control of their account that controls routing. Seems like private networks would be just as vulnerable to that.
If i understand, the article is not so much a typical bgp hijack but the attacker gaining control of their account that controls routing. Seems like private networks would be just as vulnerable to that.
In the case of the article / Orange, possibly.
Though I'd hope that Orange doesn't provide network management services for the Spanish military! Some things are better kept in-house.
Less familiar with how European sensitive networks are architected.
Though I'd hope that Orange doesn't provide network management services for the Spanish military! Some things are better kept in-house.
Less familiar with how European sensitive networks are architected.
As they stated, European gov networks (plus military and universities OFC) had and still have their private backbone. In Spain, RedIris was for scientific research, universities and such. IDK about the rest, but for sure they'd have a similar approach.
> Less familiar with how European sensitive networks are architected.
Some governments have physical tunnel networks to move people who matter around. Computer networks follow pretty much without a loss of generality.
Some governments have physical tunnel networks to move people who matter around. Computer networks follow pretty much without a loss of generality.
Indeed, RedIris in Spain, since late 80's or maybe early 90's, can't remember.
Encryption on higher level protocols (https) can do a lot to reduce potential impact.
Big impacts that come to mind (depending on what is being hijacked):
- dos attack
- pasive eavesdropping (not of encrypted contents, but who is connecting to who)
- hijacking plain http
- hijacking things like ssh where nobody pays attention to the mitm warnings
- potentially creating a new tls certificate to further do a later attack
Big impacts that come to mind (depending on what is being hijacked):
- dos attack
- pasive eavesdropping (not of encrypted contents, but who is connecting to who)
- hijacking plain http
- hijacking things like ssh where nobody pays attention to the mitm warnings
- potentially creating a new tls certificate to further do a later attack
Hijack the ip or dns entry and you break https as it’s trivial to get a new certificate and prove ownership of the DNS address adult the time.
Generally yes, but there is a slight mitigating factor that certificate transparency makes it very public who you are attacking.
In theory, if it was just your site ip that was attacked and not your dns provider, CAA record can protect you. Probably one of the few cases where dns-sec is actually useful. All that though is an edge case that probably doesn't apply 99% of the time.
In theory, if it was just your site ip that was attacked and not your dns provider, CAA record can protect you. Probably one of the few cases where dns-sec is actually useful. All that though is an edge case that probably doesn't apply 99% of the time.
> All that though is an edge case that probably doesn't apply 99% of the time.
The same could be said for many necessary security measures.
The same could be said for many necessary security measures.
Hijacking the IP would indeed allow you to respond to ACME HTTP-01 validations. That can be protected against by using RFC 8657: Certification Authority Authorization (CAA) Record Extensions for Account URI and Automatic Certificate Management Environment (ACME) Method Binding (https://datatracker.ietf.org/doc/html/rfc8657). RFC 8657 extends the CAA record to say "Only this specific CA account may request certs for this domain" and/or "Only this specific validation method may be used when requesting certs for this domain".
Let's Encrypt has supported RFC 8657 for over a year now: https://community.letsencrypt.org/t/enabling-acme-caa-accoun...
Let's Encrypt has supported RFC 8657 for over a year now: https://community.letsencrypt.org/t/enabling-acme-caa-accoun...
But then you show up on certificate transparency logs pretty quickly, no?
Well, the new certificate shows up, but so what? CT logs aren't like "Barry Shitpeas got a certificate for hugecorp.example" they just tell you what was issued, not why, not who in any useful sense was issued it, and they take up to 24 hours to do that.
What is the overlap between the set of people who think "pass1234" is a good password and the set of people who have great oversight of their cert issuance and would flag unexpected issuances ? I'd expect it to be approximately empty.
What is the overlap between the set of people who think "pass1234" is a good password and the set of people who have great oversight of their cert issuance and would flag unexpected issuances ? I'd expect it to be approximately empty.
> … they just tell you what was issued, not why, not who in any useful sense was issued it …
They give you the certificate, which contains the issuer's CPS. If it's an issuer that you (the domain owner) don't recognize, you have at least a starting point for reaching out.
> … and they take up to 24 hours to do that.
Indeed, the Maximum Merge Delay is 24 hours. But in practice, by monitoring multiple CT logs, it takes just a minute from successful issuance to having the certificate show up in at least one CT log (see https://utcc.utoronto.ca/~cks/space/blog/web/WebProbeSpeedNe...).
They give you the certificate, which contains the issuer's CPS. If it's an issuer that you (the domain owner) don't recognize, you have at least a starting point for reaching out.
> … and they take up to 24 hours to do that.
Indeed, the Maximum Merge Delay is 24 hours. But in practice, by monitoring multiple CT logs, it takes just a minute from successful issuance to having the certificate show up in at least one CT log (see https://utcc.utoronto.ca/~cks/space/blog/web/WebProbeSpeedNe...).
I imagine letsencrypt is checking if any certs were issued from the hijacked IP-range during the outage, and is moving to revoke those certs?
Presumably the site owner would be the one monitoring CT. If they see there's a rogue certificate, they can take steps to get it revoked.
Found via Twitter that others had already tried changing DNS servers, and then did a tracepath and found that I couldn't reach the resolved IP's. Thought it would have been a misconfiguration of Orange. Then on Twitter (accessed via Orange mobile, which funnily worked fine -- probably a different network?) I found a thread of the people in Spain complaining about it, where someone later replied with links to the RIPE account take-over tweet.
Took about 2-4 hours for the service to be fixed. Haven't fixed any other issues so far. One of the articles pointed that it could have been due to someone that was not using 2FA, but there were no sources in that article.
EDIT: the article mentioned above https://bandaancha.eu/articulos/secuestran-cuenta-ripe-orang...