I see a lot of analysis here that this is good for Ant, but I beg to differ, it's a very bad place to be as a company serving enterprises when deployment risk is now present. This might delay Ant's financial goals in their ability to monetize Fable and other Mythos class models.
huh- I guess there are two HN submissions with meaningful replies...
I said this in the other thread, we got access to our account back, but even with a Account Rep. and a CSM on our account- it still took them a while to figure out what was going on.
I'm sure it could have been worse if we didn't have a rep on our account.
I mean, the pain we have caused our customer ultimately proves you correct. That said, we made our decisions with the information and constraints that we knew in that moment in time. Railway has hosts in AWS/GCP/and co-los, so coordinating those workloads in a fully distributed manner would be ideal but end of the day, we didn't forsee that would just have our project get deleted just like that.
(Even if we did get assurances from them in 2024, that it wouldn't happen again, although we just got auto-rate limited the last time.)
Yea, I mean, that's the whole MO of our platform and we failed at that. So yea, that's disappointing and more so for our customers.
I can provide an explanation about the GCP dependency. Yes, we have host workloads off GCP, and we have been able to build a good business by performing a cloud exit. However, we were worried that we would have a circular dependency on our own cloud. I don't think we expected to get auto-modded out of our own account, hence we left our DB on CloudSQL.
It was never our intent to deceive people that we didn't own our own destiny with our business. The last GCP issue, we were assured that this scenario wouldn't happen (when we got auto-ratelimited, which was bad, but survivable) - but it seems like we have further work to do. Apologies.
I think the intent of Mike Judge's joke was less so an outright promotion of eugenics and more so mocking the upper crust of American society's approach to family planning. (That of which Judge was intimately familiar with during his time in SV when he worked for a graphics card company.)
A lot of his work with KotH analyzed the same dynamics of educated and uneducated America and the interplay and I think Idiocracy is essentially the terminus of the observations he would make where if the idiots got their way. (A semi-common plot point with Hank in KotH where he would be pit against rediculous circumstances.)
We have more info coming soon but I think the best way to frame this is actually working backwards and then explain how it impacted yours and other services.
So Railway (and other cloud providers) deal with fraud near constantly. The internet is a bad and scary place and we spend maybe a third to half of our total engineering cycles just on fraud/up-time related work. I don't wanna give any credit to script kiddies to the hostile nation states but we (and others) are under near and constant bombardment from crap workloads in the form of traffic, or not great CPU cycles, or sometimes more benignly, movie pirating.
Most cloud providers understandably don't like talking about it because ironically, the more they talk about it- the bad actors do indeed get a kick from seeing the chaos that they cause work. Begin the vicious cycle...
This hopefully answers:
> If 3% of your services were affected, does that match your expected fraud rate? That is an awful lot of customers to take down in one go, and you'd want to be very accurate in your modeling. I can't see how you'd plan to kill that many without false positives and negative media.
In our 5 year history, this is the third abuse related major outage. One being a Nation State DDoS, one being coordinated denial. This is the first one where it was a false positive taking down services automatically. We tune it constantly so its not really an issue except when it is.
So- with that background, we tune our boxes of lets say "performance" rules constantly. When we see bad workloads, or bad traffic, we have automated systems that "discourage" that use entirely.
When we updated those rules because we detected a new pattern, and then rolling it out, that's when we nailed the legit users, since this used the abuse pattern, it didn't show on your dash, hence the immediate gaslighting.
Which leads to the other question:
> How or why were customers not notified? I have used services before where if something seemed dodgy they would proactively reach out and say 'tell us if it's legit or in 24 hours it will be shut down' or for something truly bad, eg massive CPU usage affecting other services, they'd kill it right away but would _tell you_.
We don't want to tell fraudulent customers if they are effective or not. For this instance, it was a straight up logic bug on the heuristics match. But we have done this for our existence like black holing illegitimate traffic for example, then ban. We did this because some coordinated actors will deploy, get banned with: "reason" and then they would have backup accounts after they found that whatever they were doing was working. If you knew where to look, sometimes they will brag on their IRCs/Discords.
Candidly, we don't want to be transparent about this, but any user impact like this is the least we can do. Zooming out, macro wise, this is why Discord and other services are leaning towards ID verification. ...and it's hard for people on the non service provider side to appreciate the level of garbage out there in the internet. That said, that is an excuse- and we shovel that so that you can do your job and if we stop you, then thats on us which we own and hopefully do better about.
That said, you and others are understandably miffed (understatement) all we can do is work through our actions to rebuild trust.
First off, super duper sorry. It's sometimes a good/bad thing if I can remember someones handle. ...and I specifically remember the support thread where we did have an outage before your demo :| - the number one goal for us is to deliver a great product. Number two is that we should never embarrass a user, outages do exactly that.
We just wrapped up the post mortem and that'll be published soon where it explains why the dashboard was reporting the state of the application incorrectly and would be more than happy to credit you for the impact to keep your business. That said, totally understand if two is way too much impact for your services.
Heard. Being transparent, usually the delay on ack is us trying to determine and correlate the issue. We have a post mortem going out but we note that first report was in our system 10 minutes before it was acked, to which the platform team was trying to see which layer the impact was at.
That said, this is maybe concern #1 of the support team. Where we want the delta between report and customer outage detected to be as small as possible. The way it usually works is that we have the platform alarms and pages go first, and then the platform engineer usually will page a support eng. to run communications.
Usually the priority is to have the platform engineer focus on triaging the issue and then offload the workload to our support team so that we can accurately state what is going on. We have a new comms clustering system that rolling out so that if we get 5 reports with the similar content, it pages up to the support team as well. (We will roll this out after we communicated with affected customers first.)
Angelo from Railway here, Railway runs our own metal for the sheer reason to preserve margins so we can run for perpetuity.
We're nuts for studying failure at the company and Heroku's margins was one of the things we considered to be one of the many nails in that coffin. (RIP)
I think this acquisition makes a lot of sense and it's good business. Finding good MacOS developers who know the system level APIs more so than the docs is a tough go. It would make a lot of sense that OpenAI would just go ahead and hire out this expertise as they try to get their Mac app and their iOS app to get closer and closer to the system.
Author here, not my intent! My deepest apologies. English is my first language but people do joke that they say I write English like I learned it as a second language.
I have fixed the sentence fragment and connected the two thoughts together. Thank you for keeping me honest.
I am more than happy to add color here, I am sorry, I try my best to write everything but my editor cuts as much as I add. We also tend to hire really autonomous engineers who tend to like just going off on their own to try to solve the issue.
There have been a few times where we would commit to the problem, assign a DRI, and then find out midway that... no we have to hire/consult our way out of the issue. I think that's okay, we then look back at the retro to see what we missed.
If interested, I think we can blog about what happens when a problem gets converted to an RFC and then we have more engineering discussions with the stakeholders but the piece was pushing a 10 min read time as it was...
Hello there! Author there, and surprised/delighted with the response. I don't think we had the issue with the cadence, the quarter is arbitrary, but we think it gives us the ability to just go heads down to focus.
With that said, one thing we did and I don't why we did it was that we would "re-justify" why we would want to work on something every three months which isn't great. There is a world where if we had more eng. resources we could have more people than problems and we could take stuff on board as it arrives, but for us deciding on what to work on is a hard decision.
I also agree that market fit is a key factor. I think Railway was lucky that we didn't have to pivot the product 3 to 5 times to get some latch.
What would be the post-quarterty planning process that you would like to see?
Slight tangent brought up by the article, but usually the greats aren't good at just one thing but have a combination of eccentricities that form the person. I find it heartening that Cormac had these in spades.