Fly.io deleted all my apps without notifying me
I noticed my app was down and went to check it out. Lo and Behold my account had been flagged by their automatic fraud detection algorithm. This is despite my account having close to $500 of credit and being active for around a year now. I was not informed about this at all. I followed the prompt to allow them to charge my credit card in order to verify that I'm not a bot and now I have control of my account again. However none of my apps were restored. Has anyone else experienced something like this in the past?
53 comments
This account was not banned in error. Lukas, if you’d like me to explain publicly, I'm happy to do so.
Maybe you could start by replying to my email? I would not have complained publicly if there had been any form of communication from you but there was none.
I think here will do nicely. That OK with you?
Things got real quiet here, didn't they?
Did y'all respond to him via email like he asked? If so, then that'd explain his lack of replies here.
Y'all are doing some cool technical work, but this insistence on "either give us consent to air your dirty laundry in public or else we won't respond to you at all" raises a pretty big red flag. I'm sure there was a perfectly good reason for banning his account (I'd guess something to do with the VPN endpoint he said he was running on it), but this doesn't seem like a particularly reasonable way of handling it.
Y'all are doing some cool technical work, but this insistence on "either give us consent to air your dirty laundry in public or else we won't respond to you at all" raises a pretty big red flag. I'm sure there was a perfectly good reason for banning his account (I'd guess something to do with the VPN endpoint he said he was running on it), but this doesn't seem like a particularly reasonable way of handling it.
Anybody else getting the impression that I'm not about to let this go? Want to double down on their behalf? By all means, do.
Just to keep this perfectly clear: we do not have a system that automatically deletes flagged accounts, and what happened here was not an error on our part.
Just to keep this perfectly clear: we do not have a system that automatically deletes flagged accounts, and what happened here was not an error on our part.
Yeah, we all got that the first three times y'all said it. That doesn't make your responses any less weird. The only one "doubling down" here is you.
How is that an acceptable proposal? I use Fly, but the policy to communicate with customers only via public forum is a disaster.
That's not a decision we made. Anyways: I really don't think we have much more to say without this person's consent to proceed, beyond what was already said: we do not have a system that automatically deletes flagged accounts, and this was not an error.
(What you're describing as our policy is certainly not our policy; these are extraordinarily distinctive circumstances.)
(What you're describing as our policy is certainly not our policy; these are extraordinarily distinctive circumstances.)
[deleted]
Further: we do not have an automated system that deletes flagged apps.
> Has anyone else experienced something like this in the past
My account was marked as high risk and was prevented from deploying for a few hours: https://stackoverflow.com/a/76400473/5783745
But it didn't wipe out my apps. Took a couple hours to figure out what was going on and how to fix it (purchase some credit and run an esoteric and possibly unrelated command in the stack overflow answer). That (and a dozen other issues) was why I left fly. For now it's just not production-ready.
I do hope they build a successful product (as defined by happy customers, not by fund raising). The more players there are in this space, the more options we have as developers, which is good even if the path there is bumpy.
My account was marked as high risk and was prevented from deploying for a few hours: https://stackoverflow.com/a/76400473/5783745
But it didn't wipe out my apps. Took a couple hours to figure out what was going on and how to fix it (purchase some credit and run an esoteric and possibly unrelated command in the stack overflow answer). That (and a dozen other issues) was why I left fly. For now it's just not production-ready.
I do hope they build a successful product (as defined by happy customers, not by fund raising). The more players there are in this space, the more options we have as developers, which is good even if the path there is bumpy.
Thanks for this tip. I tried it and I keep getting the error:
"Error: Your account got flagged by our internal fraud protection service" even after verifying my credit card.
Well that just evaporated the interest I had in using Fly.io. Thanks for the heads-up.
That's what makes me so deeply suspicious of posts like this. It costs nothing but a few minutes to spring up a fake account and dump on the competition.
Who knows if Lukas is legit, but they don't seem to have asked the service provider where their data is yet. Or what happened. You're not going to get those answers here.
Going straight to social media before you've found anything out stinks.
Who knows if Lukas is legit, but they don't seem to have asked the service provider where their data is yet. Or what happened. You're not going to get those answers here.
Going straight to social media before you've found anything out stinks.
This is a reasonable criticism. I don't know how I can prove it as screenshots are not hard to fake, but here they are in any case. Also I emailed them and still haven't gotten a response, so I'm not inclined to be very understanding seeing as this appears to be happening quite frequently to people.
https://imgur.com/a/xLj3tqd
my company site is infoproc dot co. it is currently down. but if you google it you can see it has been indexed. And you can search for my company on linkedin if you want to know who I am.
https://imgur.com/a/xLj3tqd
my company site is infoproc dot co. it is currently down. but if you google it you can see it has been indexed. And you can search for my company on linkedin if you want to know who I am.
I really wasn't directly attacking you or demanding proof, just pointing out why I don't like the normalisation of complaining in public as an early step in a technical support.
you've made such a big change based on one post from a basically brand new account on HN? why?
It ain't really a big change; fly.io looked cool, but it ain't like I'm already a customer. And lukas_b doesn't seem to be all that brand-new of an account, either.
As for "why", it's because I've been burned by this sort of thing before.
As for "why", it's because I've been burned by this sort of thing before.
I haven’t had my apps deleted but the fraud detection system flags me almost every time I interact with the platform. It’s very disruptive.
This is where having Kamal config at hand is great.
Even if I would lose my VM and access, I can be up and running again pretty quickly on a different host.
I just change the IP in my config/deploy.yml
Even if I would lose my VM and access, I can be up and running again pretty quickly on a different host.
I just change the IP in my config/deploy.yml
And I guess offsite backups too?
Would love to set up everything on Kamal but just feels like slightly too much management cost in maintaining a database yourself for mission critical stuff. If I'm going to use a managed DB, might as well do the same for the container?
Maybe if I just ran with sqlite and a preconfigured litestream inside the main container that'd simplify things a fair bit, but then you're immediately locked onto a single machine.
Maybe if I just ran with sqlite and a preconfigured litestream inside the main container that'd simplify things a fair bit, but then you're immediately locked onto a single machine.
You mention Kamal as if it was some kind of a ubiquitous standard. It is the first time I hear about it. In what way is Kamal better than the hundreds of alternatives?
What were your apps?
a personal vpn and a company website
I'd guess that the personal VPN had something to do with it. Did you do anything on it that put Fly.io at risk of getting in trouble?
Dropbox deleted all my data because I didn't log in for a long time. Guess you're supposed to check in on your completely static data occasionally. Whatever, the lesson is not to let yourself seriously rely on any of these services.
No matter what service provider one uses, one should always have backups and a plan B/disaster recovery. The 3-2-1 rule is as relevant as ever.
well guess I’m keeping my current solution
I use the free tier of vercel or netlify and host all the static assets on IPFS, I keep everything pinned using filecoin for free. lol yup I use filecoin to pin IPFS CIDs
and because IPFS gateway CDNs get super unreliable, I created a js script that does a series of promises to every ipfs gateway and after one actually loads it cancels all the other promises, so that ensures availability to users and can handle spikes in traffic, without messing with the free tier
I use the free tier of vercel or netlify and host all the static assets on IPFS, I keep everything pinned using filecoin for free. lol yup I use filecoin to pin IPFS CIDs
and because IPFS gateway CDNs get super unreliable, I created a js script that does a series of promises to every ipfs gateway and after one actually loads it cancels all the other promises, so that ensures availability to users and can handle spikes in traffic, without messing with the free tier
this sounds terrifying. so their "algorithm" can just nuke your entire business without any signaling?
apparently I'm not the only one.
https://community.fly.io/t/account-flagged-apps-gone/14570
https://web.archive.org/web/20240710225525/https://community...
https://community.fly.io/t/account-flagged-apps-gone/14570
https://web.archive.org/web/20240710225525/https://community...
This is bad, really bad!
thanks for informing this, will avoid fly.io at all cost
[deleted]
The thing about fly.io / heroku / vercel is you are gaining convenience by giving up control over the underlying infrastructure.
When you give up control of the infrastructure you also give up the ability to decide what's right and wrong.
If you want to work with something that allows you retain that control but still have the convenience of a nice deployment workflow check out my profile.
When you give up control of the infrastructure you also give up the ability to decide what's right and wrong.
If you want to work with something that allows you retain that control but still have the convenience of a nice deployment workflow check out my profile.
Unless you build your own data center and run your own fiber, you’re at the mercy of someone else’s policies and could be shutdown at their whim.
Whilst this is true in theory, the experience of being a data centre customer, taking transit from one or two providers and peering with a handful of others is very different to being a user of a bulk cloud infrastructure provider.
I think the difference is automation, or rather lack of it. You build personal relationships with real people. Senior network engineers at the supplier don't consider it beneath them to ring up similarly competent people at the customer and chat directly.
In the past I've hosted kit on a larger scale for my businesses, but it seems you get the same personal experience as a tiny individual customer. Even with just a couple of racks of kit of for hobby projects and friends' stuff, I know the network engineers at the DC and transit providers and can email or call direct to discuss things when I need to.
More importantly, if (for example) one of my friends starts gobbling 10Gbps of $$$ transit because of a typo in a script, they ring my mobile to chuckle about it while we fix in realtime, rather than some automated process pulling the plug with the real engineers hiding behind support tickets and first-line support staff that can't string a sentence together.
I think the difference is automation, or rather lack of it. You build personal relationships with real people. Senior network engineers at the supplier don't consider it beneath them to ring up similarly competent people at the customer and chat directly.
In the past I've hosted kit on a larger scale for my businesses, but it seems you get the same personal experience as a tiny individual customer. Even with just a couple of racks of kit of for hobby projects and friends' stuff, I know the network engineers at the DC and transit providers and can email or call direct to discuss things when I need to.
More importantly, if (for example) one of my friends starts gobbling 10Gbps of $$$ transit because of a typo in a script, they ring my mobile to chuckle about it while we fix in realtime, rather than some automated process pulling the plug with the real engineers hiding behind support tickets and first-line support staff that can't string a sentence together.
I think it's important to realize, each level of abstraction has it's pros and cons. Concluding to have to go all the way down the stack and run your own data center and fiber is extreme.
In life it's always going to be a balance of what's feasible that you can do to keep control and giving up control because something is not feasible.
In this case obviously the author is having a problem high up the stack. The best solution would be to have more control in order to keep his business going.
In life it's always going to be a balance of what's feasible that you can do to keep control and giving up control because something is not feasible.
In this case obviously the author is having a problem high up the stack. The best solution would be to have more control in order to keep his business going.
Don't forget to bring a diesel generator if you forgot to build your own electric grid.
Even if you build your own datacenter and your own fiber, other ISPs might still either depeer from you or threaten to do so if you don't boot customer x off your service.
It's my understanding that Fly.io heavily uses Elixir and has several Elixir bigshots on staff. But I only see mention of Fly.io for people are moving off of it or having major issues. I'm wondering of that's really the case and how given the tech stack.
How is the programming language they use at all relevant?
I forgot functional programming in Elixir meant no bugs! It's built on Erlang which halves bugs tenfold too!
So 0 bugs because Elixir / 10 = 0
QED
So 0 bugs because Elixir / 10 = 0
QED
The issue is political and has nothing to do with Elixir (which is pretty great btw). Please read the post again.
Political? It was performed by their system. It doesn't sound like a human was in the loop at all, besides those who developed the system. And like I said, it isn't just this post alone. Read my comment again.
It was posted after your comment, but https://news.ycombinator.com/item?id=40938025 indicates that a human was in the loop.