We're seeing an ongoing attack against our primary network provider(fastmailstatus.com)
fastmailstatus.com
We're seeing an ongoing attack against our primary network provider
https://www.fastmailstatus.com/
90 comments
This is especially worse for the Fastmail team in Melbourne that were likely going to the pub for the first time in a LONG time.
Without any kind of context, why would this be the first time in a long time? Are they usually heavy drinkers but been on a sober-schedule that just been broken or something like that?
Melbourne is just out of a 260 days lockdown.
No, they’re coming out of a 78 day lockdown. They’ve had around 260 days of lockdown _in total_ since the start of the pandemic. That’s been spread across multiple intervals, though.
https://www.abc.net.au/news/2021-10-03/melbourne-longest-loc...
https://www.abc.net.au/news/2021-10-03/melbourne-longest-loc...
Melbourne had one of the longest lockdowns world-wide and it just got lifted.
From the status post, they use the term "primary" which implies they might have more than one network provider. Unclear whether Fastmail is the target, or if the attack is against someone else network-adjacent to them and is just large enough to congest the provider. If the latter, wonder what prevents just withdrawing the announcement and resuming service using the secondary or tertiary provider?
seems to me MUCH more likely they know it's the former, and their primary implies higher capacity.
your speculation rests on the idea that they are incredibly stupid not to think of using the redundancy they built in.
your speculation rests on the idea that they are incredibly stupid not to think of using the redundancy they built in.
Good luck to the FastMail team! Long time user, there is no better email service! Hope whoever is attacking them gets struck by lightning.
I agree! Never been happier with an email service and hope whoever is attacking them gets struck by a second lightning, some years after the first.
Wholly agreed! I've been with Fastmail almost since the beginning and I hope whomoever attacked them gets EDS (if they don't already have it), they suffer from identity theft, and none of their batteries work forever after...right after they get up off the floor from being hit by lightning multiple times!!!
That’s so specific its almost a curse. :)
Out of curiosity, in situation like these, if you send an e-mail to someone with a Fastmail account, does it eventually get to them or will it bounce back?
Short answer: yes
Long answer: it depends. On the settings on all the parties involved. Mail bounceds after the retry limits and timeouts are exhausted.
Your provider tries 5 times over 24 hours? Then that's how long fastmail has to be unreachable by your provider. Maybe my provider reaches them via their secondary routes and thus there's zero chance of bouncing. Mail goes through right away. At least to their server. But maybe the recipient reached their servers via the attacked primary and thus can't see/fetch that email.
Long answer: it depends. On the settings on all the parties involved. Mail bounceds after the retry limits and timeouts are exhausted.
Your provider tries 5 times over 24 hours? Then that's how long fastmail has to be unreachable by your provider. Maybe my provider reaches them via their secondary routes and thus there's zero chance of bouncing. Mail goes through right away. At least to their server. But maybe the recipient reached their servers via the attacked primary and thus can't see/fetch that email.
SMTP servers queue messages and retry periodically. It’s an old and resilient protocol. Eventually the message will bounce, but it’s up to the sending SMTP server how long to retry before giving up. There are some guidelines here:
https://datatracker.ietf.org/doc/html/rfc5321#section-4.5.4
https://datatracker.ietf.org/doc/html/rfc5321#section-4.5.4
It should eventually get to them.
https://twitter.com/Samuel_SGS/status/1451380974776770565
https://twitter.com/Samuel_SGS/status/1451380974776770565
I have the Fastmail Amdroid app installed. I received push notifications that had contained the beginning text of emails, but I couldn't load the app and actually read the whole email.
Now that the attack is over, I can open the app and read those emails.
Now that the attack is over, I can open the app and read those emails.
Could this be linked to truthsocial.com ? https://builtwith.com/detailed/truthsocial.com
What are you suggesting?
That Truth Social is seeing so much activity that its e-mail use is registered as an attack against Fastmail's ISP?
That someone is attacking Truth Social by attacking their e-mail provider's ISP?
Honestly this is the first time I've even heard of Truth Social and without context and with the utter lack of information provided by their landing page, it looks like yet another Gab-like. I don't see how someone would think they're worth attacking to the point where their e-mail provider's ISP is a useful vector.
That Truth Social is seeing so much activity that its e-mail use is registered as an attack against Fastmail's ISP?
That someone is attacking Truth Social by attacking their e-mail provider's ISP?
Honestly this is the first time I've even heard of Truth Social and without context and with the utter lack of information provided by their landing page, it looks like yet another Gab-like. I don't see how someone would think they're worth attacking to the point where their e-mail provider's ISP is a useful vector.
Donald Trump is behind Truth, which could conceivably make it or it’s service providers targets.
Donald Trump is mostly politically irrelevant outside the GOP at this point. Certainly not relevant enough to warrant going to the lengths of attacking the service's e-mail provider's ISP to attack a service affiliated with him. That's an incredibly contrived but also resource-intensive attack with a very limited effect.
[deleted]
Heh. Could not log in, check HN, it's the top story.
I can log in and have been receiving emails so I guess its back now.
Hugops, attacks are never fun.
From Twitter:
> We're seeing an ongoing attack against our primary network provider. We're working with them to block the attack and restore access.
https://twitter.com/Fastmail/status/1451374471344918533
> We're seeing an ongoing attack against our primary network provider. We're working with them to block the attack and restore access.
https://twitter.com/Fastmail/status/1451374471344918533
First time I can remember them down.
Same, I've been on Fastmail for 14 years and this is the first time I can remember them being down for more than a few minutes.
>I'm a happy Fastmail customer, but the number of outages recently has been a little concerning.
From that thread 9 months ago, who goes on to complain about not having post mortems sent to him via Twitter.
I've been using Fastmail since 2014 and can't remember a time I couldn't log in and access my email. I'm sure the service has gone down in the past 7 years, but I can't tell any difference from Fastmail's uptime vs Microsoft's for Outlook 365 and Google's for GMail.
I just read some emails before opening HN and seeing this, in fact. Their caching works well enough that I didn't even know they were down. Now it's showing that it can't connect to the server. Well done, Team Fastmail--that's pretty slick.
From that thread 9 months ago, who goes on to complain about not having post mortems sent to him via Twitter.
I've been using Fastmail since 2014 and can't remember a time I couldn't log in and access my email. I'm sure the service has gone down in the past 7 years, but I can't tell any difference from Fastmail's uptime vs Microsoft's for Outlook 365 and Google's for GMail.
I just read some emails before opening HN and seeing this, in fact. Their caching works well enough that I didn't even know they were down. Now it's showing that it can't connect to the server. Well done, Team Fastmail--that's pretty slick.
Out of interest, when you say 'their caching', what do you mean?
One of my annoyances with their iOS app is it doesn't support offline mode which is fairly annoying...
One of my annoyances with their iOS app is it doesn't support offline mode which is fairly annoying...
Yeah, I haven't noticed any outages. Of course that doesn't mean they don't exist, but it has had zero impact on me.
Fastmail now reports they're "under attack":
> We're seeing an ongoing attack against our primary network provider. We're working with them to block the attack and restore access.
> We're seeing an ongoing attack against our primary network provider. We're working with them to block the attack and restore access.
"Services have been restored and we are investigating the underlying cause. No mail has been lost. Thanks for your patience."
This was 3 hours ago, but I can't connect to the server. I'm receiving emails though.
This was 3 hours ago, but I can't connect to the server. I'm receiving emails though.
Sending and receiving works from Sweden right now. Maybe there was a longer than usual delay, but hard to know.
Seems to work for me as well.(from US)
Works for me as well from the US. ‘Primary’ might be the key word? Fastmail must have redundant networks for situations like this.
It's back up for me, status page doesn't say anything new, though.
Still down for me, although some Android notifications came through.
Still down on this end.
Back for me now.
Interesting, Runbox is under attack too apparently
oh, i had that page bookmarked but didn't notice until now it has a feed too. subscribed.
Omg not a good look … a commercial e-mail provider…. IMAP and web are down. Where is the failover?
hugs best of luck to the ops team
hugs best of luck to the ops team
All of Facebook was down for hours a few weeks ago. Stuff happens.
I don’t pay money to use Facebook..
You'd be stunned how awful Office 365's downtime is. Fastmail probably has a tenth of their downtime.
I cannot imagine a single service you pay for that hasn’t had downtime in their past.
Advertisers do.
Everybody pays Facebook indirectly through advertisers.
> not a good look
nothing has 100% uptime, and you sure do not want to pay for something to try
nobody is so important that they can’t handle a little downtime now and then
nothing has 100% uptime, and you sure do not want to pay for something to try
nobody is so important that they can’t handle a little downtime now and then
Meh, In my whole time using it, I have never noticed it down. Even today it seems to be up as I see it now and I did not notice any outage. Email is not a real time system I rely on anyway. As long as they don't lose my data or have it leaked, I don't really care if an email arrives an hour late once a year.
Since Fastmail is the topic here:
Have any other Fastmail users been finding lately that their messages have been getting flagged by recipients' spam filters more than before?
I've been using Fastmail since January, but getting flagged as spam has only seemed to be a significant issue for me for the past couple of months.
Have any other Fastmail users been finding lately that their messages have been getting flagged by recipients' spam filters more than before?
I've been using Fastmail since January, but getting flagged as spam has only seemed to be a significant issue for me for the past couple of months.
I use FM for some of my domains and have recently seen Google/Gmail flagging some messages as spam. None of the other mail providers appear to be flagging FM messages that I have seen.
Well this explains a lot. Couldn’t access my email earlier and now I know why. Fuck the chicken shit little script kiddies causing a ddos right now, bunch’a fuckin children.
Fastmail was pretty good while I was a customer, but this sort of thing is partially why I switched to Google Workspace. Google might be terrible for privacy but at least I can trust them to not get hacked or knocked offline. If they ever decide to randomly lock my account or anything I can always just point my domain back at Fastmail. Good luck to them and hopefully they'll be back online soon!
I don't believe google is immune to outages.
https://en.wikipedia.org/wiki/Gmail#Outages
https://en.wikipedia.org/wiki/Gmail#Outages
Fair enough. A big part I didn't mention was the desire to deprecate my old @gmail.com addresses while still having access to Google services, and as much as some of the redesigns have bothered me, I still prefer Gmail's Web UI over other providers.
Sad but true. But what is it your privacy worth? You can always keep an emergency fallback Gmail account.
At this point I've sort of just accepted that in this day and age, surveillance is inevitable and past a certain point, trying to maintain my privacy comes at a huge cost in terms of convenience that I'm not willing to pay. I certainly value my privacy but at the end of the day it's a balance for me. I'd rather put the effort into making sure I'm not so tied to a specific platform that getting cut off would be devastating, which is why I've been working on moving stuff over from old Gmail accounts to my Google Workspace account with my custom domain.
Had this happened before while you were a customer? I believe it was just a year or two ago that Gmail suffered an extended outage and was even bouncing emails for a while.
I've always used Gmail for my main email accounts and can't recall ever experiencing an outage, although it's possible I just never noticed.
why is google immune to an attack?
They're a much larger company with massive infrastructure that dwarfs Fastmail's? The traffic Fastmail is accustomed to dealing with is likely nowhere near the amount of the traffic that Google deals with.
Yep, Google has the edge and frontend resources to absorb very large attacks, and DoSSec-SRE prowling around and adjusting the defenses around the clock. One of the reasons that cloud computing feels like feudalism is if you don't have the protection of an Amazon or a Google, you're pretty much screwed. Are your systems able to survive a billion packet-per-second botnet attack?
https://cloud.google.com/blog/products/identity-security/ide...
https://cloud.google.com/blog/products/identity-security/ide...
Curious. Did the larger feudal lords attack the smaller ones in the old days?
The monarch's job was basically to prevent this.
Is it really that consequential if your email isn’t available for a little while?
Depends on what you use it for?
And what could you be using it for where a rare bit of unavailability would be unacceptable?
You can't imagine any cases where someone relies on timely email delivery and availability?
I am giving a presentation in a few mins and Bob was supposed to email me the updated price sheet?
I am running a facilities desk and my team gets assignments via email from the ticketing system?
I forgot the name of my hotel in Prague, but it's OK I'll show my taxi driver the email confirmation.
Like I said, you literally can't imagine a case where someone relies on their email to either be notified of something urgent, look up something urgent, or simply as part of their "what do I do next" workflow?
I think all the examples above have workarounds (ie, dependency on email is avoidable) but that's a different question.
I am giving a presentation in a few mins and Bob was supposed to email me the updated price sheet?
I am running a facilities desk and my team gets assignments via email from the ticketing system?
I forgot the name of my hotel in Prague, but it's OK I'll show my taxi driver the email confirmation.
Like I said, you literally can't imagine a case where someone relies on their email to either be notified of something urgent, look up something urgent, or simply as part of their "what do I do next" workflow?
I think all the examples above have workarounds (ie, dependency on email is avoidable) but that's a different question.
I indeed can imagine being inconvenienced by email not working. I can’t imagine thinking an hour of downtime a year that might cause one of those inconveniences being something that is an unacceptable risk.
I don't know how to help you see it but I hope you don't do anything for a living that involves risk mitigation (I suspect you do not...)
I ran you through a bunch of cases where you need information from your email urgently and there are painful consequences when you don't get it. Do you see how each one of these cases involves a painful loss of money/productivity/time? Can you visualize how someone can blow a multi-million dollar sales presentation because a key piece of data doesn't reach them in time? Do you consider that acceptable risk?
Are you able to understand that the problem isn't "an hour of downtime a year" but the risk of that hour happening at a time that absolutely fucks you over?
It's like saying "who cares if there's an hour a year that my car doesn't work?" It doesn't sound important except if that hour happens to be when you are getting your wife to the hospital to give birth, or when you're on your way to the airport for a flight you need to make, or if you're driving through the desert and breaking down for an hour is terrifying.
I ran you through a bunch of cases where you need information from your email urgently and there are painful consequences when you don't get it. Do you see how each one of these cases involves a painful loss of money/productivity/time? Can you visualize how someone can blow a multi-million dollar sales presentation because a key piece of data doesn't reach them in time? Do you consider that acceptable risk?
Are you able to understand that the problem isn't "an hour of downtime a year" but the risk of that hour happening at a time that absolutely fucks you over?
It's like saying "who cares if there's an hour a year that my car doesn't work?" It doesn't sound important except if that hour happens to be when you are getting your wife to the hospital to give birth, or when you're on your way to the airport for a flight you need to make, or if you're driving through the desert and breaking down for an hour is terrifying.
>I hope you don't do anything for a living that involves risk mitigation (I suspect you do not...)
Quite the opposite, I have worked a long time very closely with reliability and have done no small amount of time studying failure. Everything fails sometimes, there are always unknowns that aren't accounted for.
>It's like saying "who cares if there's an hour a year that my car doesn't work?"
This is often the case? Oh no, you have a flat tire, a dead battery, or whatever. It's not world ending, get to your destination another way or reschedule.
>Can you visualize how someone can blow a multi-million dollar sales presentation because a key piece of data doesn't reach them in time? Do you consider that acceptable risk?
If millions of dollars are on the line and you can't figure out how to work around a simple communications issue, you deserve to lose the deal. Not being able to handle failure or insisting on 100% uptime for dependencies is a sign that you have no idea how to handle risk.
Quite the opposite, I have worked a long time very closely with reliability and have done no small amount of time studying failure. Everything fails sometimes, there are always unknowns that aren't accounted for.
>It's like saying "who cares if there's an hour a year that my car doesn't work?"
This is often the case? Oh no, you have a flat tire, a dead battery, or whatever. It's not world ending, get to your destination another way or reschedule.
>Can you visualize how someone can blow a multi-million dollar sales presentation because a key piece of data doesn't reach them in time? Do you consider that acceptable risk?
If millions of dollars are on the line and you can't figure out how to work around a simple communications issue, you deserve to lose the deal. Not being able to handle failure or insisting on 100% uptime for dependencies is a sign that you have no idea how to handle risk.
> Not being able to handle failure or insisting on 100% uptime for dependencies is a sign that you have no idea how to handle risk.
Like I said 8 messages up this thread, I agree that people should have more resilient methods for handling these needs, the point is that this is often not existent.
Like I said 8 messages up this thread, I agree that people should have more resilient methods for handling these needs, the point is that this is often not existent.
People don't have text messages, phone calls, messaging apps, in person conversations, USB thumbdrives, somebody else's email at a different provider... you're not a brain in a vat connected to the universe with email, you can figure out how to communicate without it for a while if it goes down.
I'm traveling from the SF Bay Area to Austin, TX for the US Grand Prix and Sky West (Alaska) is coincidentally unable to calculate their fuel vs load https://www.usatoday.com/story/travel/airline-news/2021/10/2...
My son was able to book the last 2 seats on a Southwest flight so we only lost a day of a $500/night AirBnB. Luckily, my son uses gmail so we got the confirmation w/ a check-in link OK. We lost our car rental reservation because Alaska's idea of a replacement reservation arrived 32 hours later and Budget only holds for 24 hours. Even the SWA was 26 hours later, but somehow Priceline + Thrifty found a car.
So a shitshow could have been even worse if the 2nd outage had started earlier & my kid had taken my advice to de-google. My sister & parents also use Fastmail due to my advice, but aren't as dependent on timely email.
One thing that did save me is that I brought a notebook which has all my old email. Almost left it behind, but it's 2.8 lb & 1.61 cm thick. Smartphones still have their limits....
My son was able to book the last 2 seats on a Southwest flight so we only lost a day of a $500/night AirBnB. Luckily, my son uses gmail so we got the confirmation w/ a check-in link OK. We lost our car rental reservation because Alaska's idea of a replacement reservation arrived 32 hours later and Budget only holds for 24 hours. Even the SWA was 26 hours later, but somehow Priceline + Thrifty found a car.
So a shitshow could have been even worse if the 2nd outage had started earlier & my kid had taken my advice to de-google. My sister & parents also use Fastmail due to my advice, but aren't as dependent on timely email.
One thing that did save me is that I brought a notebook which has all my old email. Almost left it behind, but it's 2.8 lb & 1.61 cm thick. Smartphones still have their limits....
One of their mail exchangers, web site, SMTP, POP, and IMAP servers are all on the same /24, that belongs to NYI (AS11403) and according to the BGP table is not multihomed.
Historical DNS records from last two years also show they weren't changed.
I guess, "primary" is still a technically correct qualifier for the "sole" provider.