ARIN Public Incident Report – 4.10 Misissuance Error(arin.net)
arin.net
ARIN Public Incident Report – 4.10 Misissuance Error
https://www.arin.net/announcements/20251212/
40 comments
thaumaturgy(3)
The original report says
> The incorrect state persisted for approximately seven days before detection
However you're saying you've reached out "within a few minutes" ?
> The incorrect state persisted for approximately seven days before detection
However you're saying you've reached out "within a few minutes" ?
The "incorrect state" being talked about is the IP prefix being misregistered in ARIN's database.
The "hijacking" happened later, when the IP prefix was announced via BGP by the registrant who it was incorrectly assigned to. Those are two different events.
The "hijacking" happened later, when the IP prefix was announced via BGP by the registrant who it was incorrectly assigned to. Those are two different events.
It was re-allocated to the new/wrong ARIN customer for seven days before they started announcing it, at which point the OP detected the issue. Prior to that their prefix was routing to them just fine, just without RPKI protection.
A couple of years ago ARIN increased their fees considerably - way higher than fees paid to RIPE for way less resources - and had a call with their management to express my frustration, not because I was paying from my pocket but because of the high discrepancy of the what they wanted to get and the quantity/quality of their services. Now I can see that their backbone services haven't really improved while their income for sure has.
On a sidenote, what I appreciate in both RIPE and ARIN is that you can have at least a proper discussion when you have valid arguments with their support teams.
On a sidenote, what I appreciate in both RIPE and ARIN is that you can have at least a proper discussion when you have valid arguments with their support teams.
Now ARIN is much cheaper than RIPE for small entities.
fee schedules FYI
- ARIN 2026 PDF: https://www.arin.net/resources/fees/images/2026feeschedule.p...
- RIPE 2026 : https://www.ripe.net/membership/payment/
Enthusiasts, trainees and small orgs are paying a lot more with RIPE.
- ARIN 2026 PDF: https://www.arin.net/resources/fees/images/2026feeschedule.p...
- RIPE 2026 : https://www.ripe.net/membership/payment/
Enthusiasts, trainees and small orgs are paying a lot more with RIPE.
Not necessarily. Many have their RIPE registrations through an existing, “sponsoring” LIR. They’re not paying that 1800 Euro, the LIR is.
A single AS resource and a single PI assignment cost more than the ARIN fee.
Are you sure? For RIPE I see a 50 ASN plus 75 euro PI fee. ARIN is $275. Maybe I’m looking at it wrong.
It’s cheaper as a hobbyist to use a RIPE LIR. Even in the US. That’s what I’ve been doing for years.
It’s cheaper as a hobbyist to use a RIPE LIR. Even in the US. That’s what I’ve been doing for years.
afaik that's +VAT and also for LIRs only. LIRs apply markup, see https://www.lir.services/lir-sponsoring/ they charge 200€ per resource, so ASN + PI would be at last 400€/year that's way above the price of ARIN and you have a middleman.
You must have a sponsoring LIR for your resources or become a LIR yourself. The only exception is LEGACY resources (IPv4, no ASN) but that's a different story.
You must have a sponsoring LIR for your resources or become a LIR yourself. The only exception is LEGACY resources (IPv4, no ASN) but that's a different story.
There are more competitive LIRs out there. Example: https://lagrange.cloud/products/lir
It’s also cheaper for me because I have legacy ARIN space. All I really needed was an ASN. The LIR gives me some PA v6 space for cheap, too.
It’s also cheaper for me because I have legacy ARIN space. All I really needed was an ASN. The LIR gives me some PA v6 space for cheap, too.
Okay, but that is not enough to operate independently. PA v6 is another dependency. With ARIN you get your personal IPv6 assignment.
For a hobbyist, the difference is academic. You can announce PA space with your own ASN, which is what I do. If I change LIRs I’ll have to renumber my IPv6 space.
Companies offering LIR services to hobbyists are probably not going to stay in business forever, as many of them are 1 person companies, too. Also keep in mind that they may change pricing. I understand, that with IPv6 the numbering strategy is almost always automatic and a renumbering can be done in a couple of hours, but it's still an inconvenience, especially when you have to update a lot of AAAA records.
I really think that when you start to operate an AS that you should have a direct RIR membership. And as mentioned above, RIPE has a higher financial entry barrier. I remember they had an object volume based pricing scheme 15 years ago, just like ARIN still has.
I really think that when you start to operate an AS that you should have a direct RIR membership. And as mentioned above, RIPE has a higher financial entry barrier. I remember they had an object volume based pricing scheme 15 years ago, just like ARIN still has.
None of us know what will happen in the future. All I can say is that currently, it is cheaper for a hobbyist to use a RIPE LIR than to use ARIN. If this changes in the future, I'll move to ARIN.
ARIN is lowering their costs gradually. When I first made the RIPE LIR or ARIN decision several years ago, ARIN wanted $500 just to register an ASN, on top of the yearly fees. I see they have removed that requirement.
ARIN is lowering their costs gradually. When I first made the RIPE LIR or ARIN decision several years ago, ARIN wanted $500 just to register an ASN, on top of the yearly fees. I see they have removed that requirement.
All the RIRs are, in my experience, a very consistent and safe set of hands. This sort of things is vanishing rare to the point of borderline inconsequence by many providers of major internet infrastructure. The fact they care enough to take it seriously and publish shows how much they care about getting it right.
I just completed a fairly major reorganisation of resources with RIPE, and I’ve interacted with them for two decades, and my experience is they remain as steady and consistent as ever.
Sure, you may not like a particular policy at some moment, or may not agree with the charging structure at some point in time when it’s not advantageous to you, but they do at least do what they say and say what they do.
I just completed a fairly major reorganisation of resources with RIPE, and I’ve interacted with them for two decades, and my experience is they remain as steady and consistent as ever.
Sure, you may not like a particular policy at some moment, or may not agree with the charging structure at some point in time when it’s not advantageous to you, but they do at least do what they say and say what they do.
I like how frank the report is, no sugarcoating. "We relied on manual error prone verification and made a mistake. We have to automate the process."
As ARIN block owner this situation is kinda scary but reading this actually makes me think it's less likely to happen again .
As ARIN block owner this situation is kinda scary but reading this actually makes me think it's less likely to happen again .
The road to automation is always full of outages.
I'm curious how these fellas took something like IP block allocation and turned it into an Excel based workflow.
“Workflow” is probably a bit generous to describe how they probably use Excel.
Having worked at a mom and pop ISP a couple of decades ago where we used Excel to track a lot of things, I can see how this might have happened.
To actually know who is allocated what is ultimately just a list.
And when there are only a few people who edit the list (and probably no more than 1 person at a time) you can get by with even a plain text file, but Excel is quite a bit nicer as you can do things like filtering and sorting easily, maybe even some formulas to help with things.
Building a program backed by a database might be nice, but hard to justify when the manual system has never been a problem before.
They’ve probably been thinking for a while they should, but it’s just never been enough of a pain point for them to invest the effort.
Looks like they see this incident as justification that they need a system with hard coded rules and constraints, no more manual checking.
Having worked at a mom and pop ISP a couple of decades ago where we used Excel to track a lot of things, I can see how this might have happened.
To actually know who is allocated what is ultimately just a list.
And when there are only a few people who edit the list (and probably no more than 1 person at a time) you can get by with even a plain text file, but Excel is quite a bit nicer as you can do things like filtering and sorting easily, maybe even some formulas to help with things.
Building a program backed by a database might be nice, but hard to justify when the manual system has never been a problem before.
They’ve probably been thinking for a while they should, but it’s just never been enough of a pain point for them to invest the effort.
Looks like they see this incident as justification that they need a system with hard coded rules and constraints, no more manual checking.
It's ARIN, this is essentially their only job
The world's financial systems run on Excel, to a great extent.
I'm more surprised that a single person, apparently without seniority, could delete a block. IME deleting user data is usually a significant event; an IP block would especially be a big deal, especially for the IP block issuers. From the OP:
> RSD has implemented additional process controls that require a dual review for all ticketing type workflows that include a network delete.
> Only a limited set of experienced analysts are permitted to perform this function.
Great that they didn't blame the person who deleted it. ARIN seems to have put them in position where a failure was likely, eventually. Without any inside knowledge, I'd hope the culture would have any engineer leary about pressing that button without a second set of eyes reviewing it carefully and without clear authorization; I don't imagine they delete many blocks each day so it shouldn't interfere with productivity.
I'm more surprised that a single person, apparently without seniority, could delete a block. IME deleting user data is usually a significant event; an IP block would especially be a big deal, especially for the IP block issuers. From the OP:
> RSD has implemented additional process controls that require a dual review for all ticketing type workflows that include a network delete.
> Only a limited set of experienced analysts are permitted to perform this function.
Great that they didn't blame the person who deleted it. ARIN seems to have put them in position where a failure was likely, eventually. Without any inside knowledge, I'd hope the culture would have any engineer leary about pressing that button without a second set of eyes reviewing it carefully and without clear authorization; I don't imagine they delete many blocks each day so it shouldn't interfere with productivity.
They've improved over the decades. At one point the authoritative database was a physical paper notebook.
You don't find this part
> We have to automate the process.
to be ominous?
> We have to automate the process.
to be ominous?
I don’t. The report says part of this process relied on flat files and spreadsheets. Automating that with software is a good idea.
“Automate the process” doesn’t mean feeding everything to an LLM.
“Automate the process” doesn’t mean feeding everything to an LLM.
Certificate issuance was once only possible manually.
Domains too, well into the 90s.
The transparency in this incident report is refreshing. "We relied on manual Excel-based verification and screwed up" - no corporate speak, just honest assessment.
What's scary is that IPv4 allocations are literally internet infrastructure. Having your /24 suddenly reassigned to someone else could be catastrophic for a business.
The fact that RPKI didn't catch this is interesting. The ROA was deleted along with the allocation, so from RPKI's perspective everything was valid. This is a good reminder that RPKI protects against hijacking but not against the RIR itself making mistakes.
Glad they're automating this. Anything involving copy-pasting IP ranges in Excel is an accident waiting to happen.
What's scary is that IPv4 allocations are literally internet infrastructure. Having your /24 suddenly reassigned to someone else could be catastrophic for a business.
The fact that RPKI didn't catch this is interesting. The ROA was deleted along with the allocation, so from RPKI's perspective everything was valid. This is a good reminder that RPKI protects against hijacking but not against the RIR itself making mistakes.
Glad they're automating this. Anything involving copy-pasting IP ranges in Excel is an accident waiting to happen.
I can't remember a screw up by ARIN this bad before. I'm not too concerned about it. I understand that mistakes can happen. That said, I'm a little surprised at how easy it was to make this one.
I'm entirely unsurprised that this mistake involved an excel spreadsheet. Out of all the databases and IP management software they could be using which would have prevented this the first thing the employee reached for was excel. Almost every company I've worked for has employees using excel for data that would be better managed/stored/presented outside of an office document.
I'm entirely unsurprised that this mistake involved an excel spreadsheet. Out of all the databases and IP management software they could be using which would have prevented this the first thing the employee reached for was excel. Almost every company I've worked for has employees using excel for data that would be better managed/stored/presented outside of an office document.
From the nanog thread it seemed like the IP allocations for the IPv6 transition space (4.10) was the only space using this manual Excel process. That's probably how they initially started managing these allocations with the intention to build it into their automated systems but hadn't gotten around to it. And it sounds like they're prioritizing that work now, and have implemented an additional lay of checks in the mean time.
This is a really big egg on face moment for ARIN, but it sounds like they are responding appropriately.
This is a really big egg on face moment for ARIN, but it sounds like they are responding appropriately.
So at least a good chunk of the Internet does indeed operate on a spreadsheet. Good to know.
All data begins life in a spreadsheet and dies in a spreadsheet. Automation is an illusion; databases are illusions. Only Excel is real.
This reads like a joke, but I've known two DBAs who don't use database management tools beyond exporting whole tables to excel, making manual changes, and importing to update the tables. Scary stuff.
I've considered setting up an ASN and grabbing an IPv6 block for myself for a while now, but have never had the gumption, time, and funds at the same time.
This is a bit beyond my paygrade, but... this is as serious as it sounds, right? I'm just a bit surprised/confused by the response in these comments, especially compared to outages like when CF goes down. It's like that Gordon Ramsay meme. Is ARIN the 8 year old in this situation?
Hey NANOG,
After receiving a BGPAlerter notification that one of our subnets (23.150.164.0/24) had been hijacked, I checked and noticed the prefix in question was missing RPKI. Assuming I had fat fingered something and butchered the ROA, I logged into ARIN and found that the prefix was missing from our resource list entirely, and had been reallocated to another organization and announced from their network. I created a ticket in ARIN and called immediately.
They confirmed that our subnet had been accidentally reallocated to another customer, and that they are currently working on returning it to us. After a couple hours, they told us the other organization will stop announcing the prefix, and WHOIS will be returned shortly.
I’m guessing there’s no way to prevent this kind of thing on our side if the RPKI ROA itself is removed along with the allocation? I’m planning on adding checks to look for missing ROAs (in addition to invalid/expiring ones), which I'm guessing would've caught this earlier.
Have any of you had anything like this happen with ARIN or another RIR? I’m especially curious what might have happened if we’d only noticed and reached out a few weeks later instead of within a few minutes.