Signal apps DDoS'ed their own server(twitter.com)
twitter.com
Signal apps DDoS'ed their own server
https://twitter.com/NovakDaniel/status/1350471722034745348
134 comments
First distrbuted computing fallacy is "Network is reliable".
Detecting if you can deliver a packet in advance is not exactly a solved problem - network detection APIs at best inform you if the PHY/MAC layer for a given connection is up, not if any given packet you generate can be routed out - and fail to detect a number of rather common edge cases (all traffic being routed over VPN on WiFi, without mobile fallback, for instance, or mobile connection appearing to be up, but not transferring data) - therefore ultimately the solution is to try, and try again, with proper backoff to prevent overwhelming your own infrastructure.
In this case the latter appears to have been missing.
Detecting if you can deliver a packet in advance is not exactly a solved problem - network detection APIs at best inform you if the PHY/MAC layer for a given connection is up, not if any given packet you generate can be routed out - and fail to detect a number of rather common edge cases (all traffic being routed over VPN on WiFi, without mobile fallback, for instance, or mobile connection appearing to be up, but not transferring data) - therefore ultimately the solution is to try, and try again, with proper backoff to prevent overwhelming your own infrastructure.
In this case the latter appears to have been missing.
I've seen "Network is unreliable" being used as an argument to hide laziness. I do agree you can have flaky network, but the biggest give of the laziness behind the argument is not inspecting error codes at all. In this particular case 508 for example is clear indicator of some error status. Why would you even keep blind retries in place? I hate the argument of devs arguing for "dumb clients" and server "taking care of everything". Even for network failures IDK how having a retry within millisecond will actually help. It will be more detrimental to your infra, bandwidth, battery than a once or twice snappy experience argument.
Smart server dumb client makes a lot of sense in the web-dev world, where the core client code is not under your control. As soon as you need to write your own app (Android/iOS/desktop), you should revisit this decision. The server should be able to serve a status code & header that requests the client to back off for a certain time period or with certain exponential parameters.
This is a very hard thing to deploy after the fact and can really save your bacon, especially if you have a number of very old clients still out in the field refusing auto-updates. Sure, you can refuse to serve them, but if they get stuck in a tight loop retrying until you do, you're pretty hosed.
This is a very hard thing to deploy after the fact and can really save your bacon, especially if you have a number of very old clients still out in the field refusing auto-updates. Sure, you can refuse to serve them, but if they get stuck in a tight loop retrying until you do, you're pretty hosed.
Sure but that just falls under proper backoff. If they decide to use different backoff strategies for different classes if errors, that'd be great. But it's an easy mistake to make, and a very hard one to catch in testing. Sure if they wrote the code better (or been less "lazy") this wouldn't have happened, but that type of feedback isn't constructive imo. This wasn't a conscious decision that was made, it was written this way by default because of some combination of their team/framework/api client/testing/documentation/etc. Talking about why those led to this is much more interesting I think.
> Circuit breaking, exponential backoff, bulkheading should be key pillars and no dev should be allowed to write a client without these key skills.
This is all the more critical for client applications that depend on app stores to review and approve updates and end users who may not immediately update to a newer version because they don’t have auto update turned on.
Edit: I can’t seem to find it, but there was a comment recently about servers using flags to tell clients to shut up for a specific duration and avoid exacerbating problems with retries. I think it was about some specific design of Dropbox.
This is all the more critical for client applications that depend on app stores to review and approve updates and end users who may not immediately update to a newer version because they don’t have auto update turned on.
Edit: I can’t seem to find it, but there was a comment recently about servers using flags to tell clients to shut up for a specific duration and avoid exacerbating problems with retries. I think it was about some specific design of Dropbox.
There’s a whole HTTP status for “you are connecting too fast” but that assumes the server is kinda working. https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/429
Most client code I have ever seen unfortunately only ever looks for status code != 200
Signal started to handle 404 7 months ago and the 508 code 13 hours ago. (https://github.com/signalapp/Signal-Android/blame/master/lib... strg+f 508)
Agreed server should have a rate-limiting on each account to prevent DDoSing in future.
In an emergency, one could rig up a server to drop TCP connections without sending RST. The result would be a client that thinks it’s connected without using server resources. Bonus points for some code on the server that responds to some fraction of incoming segments with ACKs to keep stringing clients along.
Computing isn't free. Energy is being produced/generated somewhere to prop up the demand created by the system you put in place, and your network congestion adversely effects everyone else. Don't let the "transparency" of the network lull you into Tragedy of the Commons behavioral patterns. Act with intention. If you find your thing is overwhelming your own capacity to handle traffic, either implement rate modulation capability, or restructure your program to mitigate whatever is creating the problem.
Just because there are selfish. a-holes who think the internet is an infinite resource intended solely for their own enjoyment, convenience and exploitation, does not mean it is okay for everyone to act that way. Not that anyone in this case is one of those types of person mind, but it's worth pointing out. This seems more like unexpected growth far outstripping provisioned resources.
Just because there are selfish. a-holes who think the internet is an infinite resource intended solely for their own enjoyment, convenience and exploitation, does not mean it is okay for everyone to act that way. Not that anyone in this case is one of those types of person mind, but it's worth pointing out. This seems more like unexpected growth far outstripping provisioned resources.
It's pretty effective to drop a percent of incoming SYNs.
But if you're behind a load balancer, and the load balancer doesn't know to do that, you can't do it on the origins, because the load balancer will fail the origins and that's not good.
But if you're behind a load balancer, and the load balancer doesn't know to do that, you can't do it on the origins, because the load balancer will fail the origins and that's not good.
Another scenario to keep in mind is the Thundering Herd Problem[0].
Exponential backoff without added jitter could still DDoS the servers.
[0]: https://en.wikipedia.org/wiki/Thundering_herd_problem#Mitiga...
[0]: https://en.wikipedia.org/wiki/Thundering_herd_problem#Mitiga...
Which form of jitter is better: adding a random wait time to a predetermined wait time that grows exponentially with each retry attempt, or following something like [0] where every retry attempt increases the possible wait time choices and the "jitter" is to randomly pick one of them?
To illustrate the latter option, suppose the smallest retry time unit is 1 second. The first attempt gives you a random choice in {0, 1}. The second attempt gives you a random choice in {0, 1, 2}. The third attempt gives you a random choice in {0, 1, 2, 4}. The fourth attempt gives you a random choice in {0, 1, 2, 4, 8}. This goes on until a ceiling in the number of attempts or a set wall clock time is reached.
[0]: https://en.wikipedia.org/wiki/Exponential_backoff#Example_ex...
To illustrate the latter option, suppose the smallest retry time unit is 1 second. The first attempt gives you a random choice in {0, 1}. The second attempt gives you a random choice in {0, 1, 2}. The third attempt gives you a random choice in {0, 1, 2, 4}. The fourth attempt gives you a random choice in {0, 1, 2, 4, 8}. This goes on until a ceiling in the number of attempts or a set wall clock time is reached.
[0]: https://en.wikipedia.org/wiki/Exponential_backoff#Example_ex...
It depends on what you want. The first option will tend towards longer waits, and the second towards shorter waits.
I would tend towards increasing the minimum wait at each iteration (until you get to some maximim wait), because it it failed 10 times in the last minute, it's likely going to fail many times in the next minute, so we don't need to try more than once or twice.
Also: in case the client random is broken, you don't want to accidentaly end up with everyone retrying after zero seconds forever.
I would tend towards increasing the minimum wait at each iteration (until you get to some maximim wait), because it it failed 10 times in the last minute, it's likely going to fail many times in the next minute, so we don't need to try more than once or twice.
Also: in case the client random is broken, you don't want to accidentaly end up with everyone retrying after zero seconds forever.
Ethernet uses the second one to avoid collision.
Is anyone still using shared medium Ethernet?
exponential backoff will still kill the servers because the first thing people do is kill/restart the app or reload the webpage and it will just restart the backoff again
So just set 'backoff until [timestamp]' rather than 'backoff for [time interval]'. Generally users restart because they don't know what's going an assume the client is stuck in loop or something. Think of a client that can say 'internet is up but my.server is having [local, regional, global] problems. I will try again at 11:37am EST.'
Downforeveryoneorjustme is exploring an API for service monitoring and it seems to me like every cloud client should have a standardized approach of trying to reach its own server, then checking internet access, then checking a service monitor, then checking social media status updates.
https://downforeveryoneorjustme.com/services/api
Also think apps and devices should have limited peer-to-peer information sharing instead of only talking to the operating system. In many ways our devices are like a roomful of people whispering status updates to the operating system and/or user but never talking to each other because 'security'.
Downforeveryoneorjustme is exploring an API for service monitoring and it seems to me like every cloud client should have a standardized approach of trying to reach its own server, then checking internet access, then checking a service monitor, then checking social media status updates.
https://downforeveryoneorjustme.com/services/api
Also think apps and devices should have limited peer-to-peer information sharing instead of only talking to the operating system. In many ways our devices are like a roomful of people whispering status updates to the operating system and/or user but never talking to each other because 'security'.
I make a mobile app and we do this. When the app wakes up it tries to GET a file from /.well-known and if that works, proceed. If it fails we have a notice with "will retry at T". Timing is random between 10 and 50 seconds
Yes, but even they much back pressure provides a lot of reprieve.
Restarting an app or even manually refreshing a web page takes a lot longer than an in-memory automatic retry() function call.
Restarting an app or even manually refreshing a web page takes a lot longer than an in-memory automatic retry() function call.
Dozen Years ago I built a backend Ruby server, (not rails), I needed max performance.
Our front end flex guy didn’t understand backing off on retries.
Some system had an issue, that caused the flex app for 1000 people to go insane. Each calling ruby app endlessly with no delay.
Ruby app remained healthy, server crashed in under 5 minutes from running out of disk space.
Our alarm interval on free space ran every 5 minutes. So no time for an alarm to sound.
A mix, of I was very proud of my little ruby app for scaling. And a a wtf, I needed to be more involved with front end people.
Our front end flex guy didn’t understand backing off on retries.
Some system had an issue, that caused the flex app for 1000 people to go insane. Each calling ruby app endlessly with no delay.
Ruby app remained healthy, server crashed in under 5 minutes from running out of disk space.
Our alarm interval on free space ran every 5 minutes. So no time for an alarm to sound.
A mix, of I was very proud of my little ruby app for scaling. And a a wtf, I needed to be more involved with front end people.
Take the flip side of it. If you are working on the backend, you should always assume/provision for the case where there will be hostile clients. If your backend can only work if the client plays nice, I'd be weary of pointing fingers too quickly.
Trouble is...you can't tell who is malicious in a DDoS scenario.
Yeah, that's my point actually. If your service is failing because of a DDOS, don't blame those trying to use your infra. Before pointing to issues of bad clients, Signal should be ready for this influx of users - especially if your leader believes that centralized services are the only ones able to create a viable alternative for the masses.
I'm unfamiliar with "circuit breaking" and "bulkheading" in this context. Could you provide a bit more description so i can research them and make sure i know what i should know? :)
Circuit breaking: also called “kill switch”. It’s having a way to shut down some feature if it becomes problematic.
“Bulkheading” is making sure that failures in one area don’t cascade into causing failures elsewhere analogous to how bulkheads on a ship prevent a single breach from sinking the whole ship.
“Bulkheading” is making sure that failures in one area don’t cascade into causing failures elsewhere analogous to how bulkheads on a ship prevent a single breach from sinking the whole ship.
As far as I know, the original source for these patterns is Michael Nygard's book Release It: https://www.amazon.com/gp/product/0978739213
A fantastic book! The first edition is a bit dated in places, but there's a second edition now: https://www.amazon.com/Release-Design-Deploy-Production-Read...
The context is somewhat important.
While 508s were being sent, there was also a significant number of 503s (IIRC). A 503 is an absolute hallmark of something reaching max utilization. Sometimes it's a bad code push that results in memory bloat and then swap or significantly increased request handling time on a poorly threaded server (think a code push that makes a blocking linear request in a loop), but the vast majority of time an upstream dependency (specifically a data store) has been overloaded.
So for whatever reason a data store gets slower. What happens upstream when this happens? The number of incoming requests is constant, but the time each individual thread spends attempting to talk to the data store is constant (or worsening). This means each request blocks longer (resulting in potential thread starvation) or there are more concurrent requests (load) to the data store.
This creates a feedback loop of doom: as a data store slows down, its load (the number of requests it's handling at once) increases until complete failure.
The only way to stop this behavior is by "failing fast." This is how a circuit breaker works. When the data store starts responding slowly, it’s important not to hammer it with even more load, so your client watches the number of load related failures (or response times) and automatically fails requests immediately without sending it to the data store (circuit breaker). This allows the data store to become unloaded and get out of the doomed feedback loop.
While 508s were being sent, there was also a significant number of 503s (IIRC). A 503 is an absolute hallmark of something reaching max utilization. Sometimes it's a bad code push that results in memory bloat and then swap or significantly increased request handling time on a poorly threaded server (think a code push that makes a blocking linear request in a loop), but the vast majority of time an upstream dependency (specifically a data store) has been overloaded.
So for whatever reason a data store gets slower. What happens upstream when this happens? The number of incoming requests is constant, but the time each individual thread spends attempting to talk to the data store is constant (or worsening). This means each request blocks longer (resulting in potential thread starvation) or there are more concurrent requests (load) to the data store.
This creates a feedback loop of doom: as a data store slows down, its load (the number of requests it's handling at once) increases until complete failure.
The only way to stop this behavior is by "failing fast." This is how a circuit breaker works. When the data store starts responding slowly, it’s important not to hammer it with even more load, so your client watches the number of load related failures (or response times) and automatically fails requests immediately without sending it to the data store (circuit breaker). This allows the data store to become unloaded and get out of the doomed feedback loop.
The difference between a circuit breaker and a kill switch is that a circuit breaker, like a real electrical one, automatically trips and stops requests going through for some time after a certain error threshold to the remote server has passes.
"Circuit breaking" is a threshold (examples: elapsed time, attempts, etc.) that triggers a discontinuation of the current action (examples: HTTP request, HTTP response, etc.).
"Bulkheading" is about one part not taking down a whole, which is more thoroughly described here: https://docs.microsoft.com/en-us/azure/architecture/patterns...
"Bulkheading" is about one part not taking down a whole, which is more thoroughly described here: https://docs.microsoft.com/en-us/azure/architecture/patterns...
Bulkhead pattern is one of my favorites.
Even with a monolith, often one can spin up multiple copies, and assign different routes in a front end http proxy to different copies of the monolith.
It makes it extremely clear which endpoints are consuming how much resources. And if one copy goes down, the others are ok.
Absolutely one of the best things you can do, not just for protecting against faults, buy also for understanding the cost of different endpoints.
Even with a monolith, often one can spin up multiple copies, and assign different routes in a front end http proxy to different copies of the monolith.
It makes it extremely clear which endpoints are consuming how much resources. And if one copy goes down, the others are ok.
Absolutely one of the best things you can do, not just for protecting against faults, buy also for understanding the cost of different endpoints.
Another (big company strategy) is deadlines:
https://cloud.google.com/appengine/docs/standard/python/how-...
The idea is that at the time a request is made, you have a pretty good idea of the maximum amount of time it can take, so you attach that as metadata with the request.
If any service anywhere in infrastructure sees that a request has taken more than its budgeted time, it has permission to immediately stop handling that request and return an error.
https://cloud.google.com/appengine/docs/standard/python/how-...
The idea is that at the time a request is made, you have a pretty good idea of the maximum amount of time it can take, so you attach that as metadata with the request.
If any service anywhere in infrastructure sees that a request has taken more than its budgeted time, it has permission to immediately stop handling that request and return an error.
I've seen it regularly happen where a user's company account is used for email that whenever the user changes their password, the email app on their phone causes their account to get locked out by doing repeated retries with the old password.
That kind of design is wrong because anyone can lock out any account (or all of them).
Mind sharing the app name?
Microsoft Outlook does this.
Native Mail client for Android still retries aggressively on an incorrect password, locking out accounts.
The native iOS mail client stopped the endless retry behavior a few releases ago (so sad it only took a decade to fix),
The native iOS mail client stopped the endless retry behavior a few releases ago (so sad it only took a decade to fix),
I remember some years ago iPhone Activesync client was retrying 5 time the same login password on authentication failure (http 401), and our AD was locking people out after 5 attempts ... We ended up writing a small proxy to fix it server side.
You'd think the semantics of "4xx is the client's fault, don't bother retrying without changing the request" is pretty easy to understand.
Has the "batteries included" aspect of things like exponential backoff improved? Last I was in the details, most of the popular client libraries offered little, and you had to implement it yourself.
Resilience4J has improved quite a lot. I have been using for over two years now, never missed out on anything.
Also token bucket: https://en.wikipedia.org/wiki/Token_bucket
it's also quite possible to design and implement this one the server side. I mean one can implement Circuit breaking, exponential backoff etc. at network level (e.g. istio) based on the "internal service" (could be VMs/nginx rever proxies) outage error.
On man. Your list of pillars is spot on.
>I think the @signalapp apps DDoS'ed the server.
and so the chickens come home to roost. Moxies vehement rejection of a distributed design seems less and less tenable each outage. Last time it was what...verification numbers that werent getting sent?
and FWIW the divinations from the community are exceedingly helpful in a time when not even the signal website seems to confirm or deny any sort of outage. Signups are still being taken and the outage page at signal.org is still as static as ever. Even the twitter hasnt seen an update in nearly a day. Is anyone at Signal foundation at the controls?
As a signal user myself I know this is going to sound rude but at this point other than the endorsements from musk and dorsey, why would any new user consider this service at all if its been down for nearly two days? theres never any postmortem, and communication is generally evangelical or solicitous in nature for either installs or donations.
and so the chickens come home to roost. Moxies vehement rejection of a distributed design seems less and less tenable each outage. Last time it was what...verification numbers that werent getting sent?
and FWIW the divinations from the community are exceedingly helpful in a time when not even the signal website seems to confirm or deny any sort of outage. Signups are still being taken and the outage page at signal.org is still as static as ever. Even the twitter hasnt seen an update in nearly a day. Is anyone at Signal foundation at the controls?
As a signal user myself I know this is going to sound rude but at this point other than the endorsements from musk and dorsey, why would any new user consider this service at all if its been down for nearly two days? theres never any postmortem, and communication is generally evangelical or solicitous in nature for either installs or donations.
You seem to make a lot of statements which imply you have more knowledge than I would expect. Yesterday you seemed to have insider details on how they were managing (or not) their capacity. Where are you getting your information on the inner workings of signal?
I am definitely getting strong signal that you are the type of person who has no perspective on what they are talking about, but I'm open to the idea I'm wrong and would love to be proven so. I think your expectations are out of line for the size and maturity of the company.
I am definitely getting strong signal that you are the type of person who has no perspective on what they are talking about, but I'm open to the idea I'm wrong and would love to be proven so. I think your expectations are out of line for the size and maturity of the company.
And you are right, on their profile about:
> Changing brake pads by day and learning python at night
FWIW, (pre-Facebook acquisition) WhatsApp's systems were serving tens if not hundreds of millions of people and they never failed so catastrophically.
There was a difference in approach, though. WhatsApp was using battle-tested technology (ejabberd, XMPP) and removing things that didn't make sense to them (federation). Moxie had one trick (end-to-end encryption) and wanted to build a whole messaging platform around that. Instead of standing in shoulders of giants, he chose to ignore all the lessons from the past and do everything from scratch out of the belief that he can do better.
At some point, these systems break. In Signal's case, this point has been reached.
There was a difference in approach, though. WhatsApp was using battle-tested technology (ejabberd, XMPP) and removing things that didn't make sense to them (federation). Moxie had one trick (end-to-end encryption) and wanted to build a whole messaging platform around that. Instead of standing in shoulders of giants, he chose to ignore all the lessons from the past and do everything from scratch out of the belief that he can do better.
At some point, these systems break. In Signal's case, this point has been reached.
While actual technology used and its effect on failure is an interesting debate, I think the growth curve involved has a vastly larger influence on stability. My potentially wrong understanding is they experienced a ~10x growth in a ~2 week period.
Technology does not exist absent its context. One might use MySQL, but is it sharded? are reads critical? Lets say you go from 1 database to 8 databases and then from 8 databases to 128? How much tooling do you think is involved in those changes (machine configuration, service discovery, shading behavior, machine replacement behavior, how machines are monitored). At what point do assumptions made under a small system completely fail in large systems? I can manually configure 8 machines by hand no problem. I can add 2 more no problem. But 100 machines? That territory starts to require automation. You can potentially look at 100 individual graphs for 100 different machines, but what about 1000 graphs? All of a sudden aggregation and drill down become necessary.
None of us (I think) have enough first hand information to understand why scaling up 10x was complicated for them. What I can almost promise is that if you have 10 servers and you go to 100, you are going to run into all types of problems you did not have at 10. Coordinated behavior across 100 machines is going to be a lot less forgiving than coordinated behavior across 10. Imagine a relationship where you have a every frontend talking to every data store. With 5 data stores, random failure isn't all that likely, but with 1000, it's much more likely. If every frontend talks to every data store, a single data store slowing down could result in ALL frontend stuck failing to make requests to it. Machine connection limits can be reached, interrupts from the NIC can bottleneck, service discovery can fail.
Does technology have an effect? Yes. Is lack of exponential backoff somewhat embarrassing? Yeah. Is failing spectacularly to keep up with exponential growth over a short period unexpected? No. I would expect any website or service already at scale to spectacularly fail to service 10x growth in a 2 week period.
Technology does not exist absent its context. One might use MySQL, but is it sharded? are reads critical? Lets say you go from 1 database to 8 databases and then from 8 databases to 128? How much tooling do you think is involved in those changes (machine configuration, service discovery, shading behavior, machine replacement behavior, how machines are monitored). At what point do assumptions made under a small system completely fail in large systems? I can manually configure 8 machines by hand no problem. I can add 2 more no problem. But 100 machines? That territory starts to require automation. You can potentially look at 100 individual graphs for 100 different machines, but what about 1000 graphs? All of a sudden aggregation and drill down become necessary.
None of us (I think) have enough first hand information to understand why scaling up 10x was complicated for them. What I can almost promise is that if you have 10 servers and you go to 100, you are going to run into all types of problems you did not have at 10. Coordinated behavior across 100 machines is going to be a lot less forgiving than coordinated behavior across 10. Imagine a relationship where you have a every frontend talking to every data store. With 5 data stores, random failure isn't all that likely, but with 1000, it's much more likely. If every frontend talks to every data store, a single data store slowing down could result in ALL frontend stuck failing to make requests to it. Machine connection limits can be reached, interrupts from the NIC can bottleneck, service discovery can fail.
Does technology have an effect? Yes. Is lack of exponential backoff somewhat embarrassing? Yeah. Is failing spectacularly to keep up with exponential growth over a short period unexpected? No. I would expect any website or service already at scale to spectacularly fail to service 10x growth in a 2 week period.
If you haven't tested or don't know how your systems are going to react when operating for a different scale, then don't let it grow in the first place.
By taking more that they could chew, they are burning each one of these 40M+ users. How many of these are going to keep trying other alternatives and how many are going to shrug and say "that shit doesn't work?"
By taking more that they could chew, they are burning each one of these 40M+ users. How many of these are going to keep trying other alternatives and how many are going to shrug and say "that shit doesn't work?"
> If you haven't tested or don't know how your systems are going to react when operating for a different scale,
In my opinion, this is a pretty unreasonable expectation. Not to mention any load testing is going to be done under a certain set of assumption, a set of assumptions that is almost guaranteed to be violated by actually scaling something.
What's the most naive load test you can think of? I am going to see the maximum amount of requests my dev instance can handle. Oh i can handle 10 requests per second. If I run 24 instances on a machine, and run 100 machines, then I must be able to service 24,000 RPS.
Ok... Now explain concretely, what kind of load testing would/should have been done that would have surfaced the lack of an exponential backoff without a priori knowledge of it?
The category of things you don't know you don't know at 10x scale is large. The justification for spending time "scale testing" is going to be a hard when your manager goes to social media and sees every single person wanting backups for their texts. "Hmm should we apply engineering resources to the feature we know everyone wants, or exploring the potential land of failures that may or may not have payoff, and definitely not immediate payoff." Most of these prioritization situations are directly and solely informed via experiencing outages, just like this one.
> then don't let it grow in the first place.
This is the type of insight you get from experience, it's also the type of call that I personally think would be very hard to make. As an engineer you don't always know if an outage is going to solve itself, take 10 minutes, 30 minutes, or take 10 hours. Then communicating that to someone with the power to create that policy...
How many (social) companies do you know of that haven't had absolutely catastrophic outages/security breeches?
This is the type of outage that justifies reliability work and gets resources applied to it. This is the type of thing that results in an organization creating a somewhat rigorous incident response policy that handles the business side (keeping users informed, giving them the best experience possible). This is a growing pain of rapid maturity.
I think you can talk the way you do because you have not personally faced this type of engineering situation (a guess). Easy to make comments from the peanut gallery, but try putting yourself in the engineer's or the CEO's shoes and walking through what information you have at the time and what options you have available at the time, without the benefit of hindsight. Imagine you have a pipeline of priorities you look at every day. Is spending time wooing a senior engineer to come work for you a lower priority than having a good story for what happens when you 10x your traffic in 2 weeks? Things can only be black and white in hindsight. Their response to this is infinitely more indicative of the type of organization they are than the actual outage.
In my opinion, this is a pretty unreasonable expectation. Not to mention any load testing is going to be done under a certain set of assumption, a set of assumptions that is almost guaranteed to be violated by actually scaling something.
What's the most naive load test you can think of? I am going to see the maximum amount of requests my dev instance can handle. Oh i can handle 10 requests per second. If I run 24 instances on a machine, and run 100 machines, then I must be able to service 24,000 RPS.
Ok... Now explain concretely, what kind of load testing would/should have been done that would have surfaced the lack of an exponential backoff without a priori knowledge of it?
The category of things you don't know you don't know at 10x scale is large. The justification for spending time "scale testing" is going to be a hard when your manager goes to social media and sees every single person wanting backups for their texts. "Hmm should we apply engineering resources to the feature we know everyone wants, or exploring the potential land of failures that may or may not have payoff, and definitely not immediate payoff." Most of these prioritization situations are directly and solely informed via experiencing outages, just like this one.
> then don't let it grow in the first place.
This is the type of insight you get from experience, it's also the type of call that I personally think would be very hard to make. As an engineer you don't always know if an outage is going to solve itself, take 10 minutes, 30 minutes, or take 10 hours. Then communicating that to someone with the power to create that policy...
How many (social) companies do you know of that haven't had absolutely catastrophic outages/security breeches?
This is the type of outage that justifies reliability work and gets resources applied to it. This is the type of thing that results in an organization creating a somewhat rigorous incident response policy that handles the business side (keeping users informed, giving them the best experience possible). This is a growing pain of rapid maturity.
I think you can talk the way you do because you have not personally faced this type of engineering situation (a guess). Easy to make comments from the peanut gallery, but try putting yourself in the engineer's or the CEO's shoes and walking through what information you have at the time and what options you have available at the time, without the benefit of hindsight. Imagine you have a pipeline of priorities you look at every day. Is spending time wooing a senior engineer to come work for you a lower priority than having a good story for what happens when you 10x your traffic in 2 weeks? Things can only be black and white in hindsight. Their response to this is infinitely more indicative of the type of organization they are than the actual outage.
You are right. I never been in those shoes, but for a different reason that you believe: I simply do not want to support/contribute for the development of any system that depends on centralizing and monopolizing the users.
I don't believe that it is the right way to do it. Moxie does. I rather have, work on and put skin on the game of decentralized/distributed systems. Moxie prefers to build centralized systems. He is the one putting skin on the game behind that. So I don't think we should be easy on him when the system he proposed fails so catastrophically precisely for the reasons that critics of centralized solutions said it was likely to fail.
Yeah, I would never get 40M+ users on a week. That would be by design. Instead of one company getting this many people (and potentially being killed in the process) I'd rather have 400 new providers, each working to get 10k customers. Easier to scale, more people figuring out issues, more people contributing to the system and making the infrastructure antifragile.
I don't believe that it is the right way to do it. Moxie does. I rather have, work on and put skin on the game of decentralized/distributed systems. Moxie prefers to build centralized systems. He is the one putting skin on the game behind that. So I don't think we should be easy on him when the system he proposed fails so catastrophically precisely for the reasons that critics of centralized solutions said it was likely to fail.
Yeah, I would never get 40M+ users on a week. That would be by design. Instead of one company getting this many people (and potentially being killed in the process) I'd rather have 400 new providers, each working to get 10k customers. Easier to scale, more people figuring out issues, more people contributing to the system and making the infrastructure antifragile.
In some ways, it seems like we are watching a warp-speed demonstration of how evolutionary processes work. Environmental pressure, lots of contenders coming with slightly mutations and all of them passing through some fitness filter.
It's just too bad that those arguing for decentralized systems fail to spread the meme more effectively. We federalists are like cockroaches.
It's just too bad that those arguing for decentralized systems fail to spread the meme more effectively. We federalists are like cockroaches.
Respectfully, how exactly would federation have helped with this outage?
Signal's own official clients failed to properly back off from spamming the server with incessant requests. I'm not entirely sure how more third party clients would have helped with this issue.
Signal's own official clients failed to properly back off from spamming the server with incessant requests. I'm not entirely sure how more third party clients would have helped with this issue.
My question exactly, if majority of the users have an account on the @matrix instance and if that fails, wont the same issue happen ?
The instance dies, not the whole system.
Synapse (Matrix reference server implementation) had for a long time severe performance issues when dealing with large rooms. Clients would try to enter a room and would take literally forever to be able to get messages for that room.
Also, the natural reaction from people upon seeing how some rooms from bigger instances were causing problem was to create their own instances and their own rooms.
Federated protocols are anti-fragile.
Synapse (Matrix reference server implementation) had for a long time severe performance issues when dealing with large rooms. Clients would try to enter a room and would take literally forever to be able to get messages for that room.
Also, the natural reaction from people upon seeing how some rooms from bigger instances were causing problem was to create their own instances and their own rooms.
Federated protocols are anti-fragile.
It would not have prevented the outage. But there would have been other servers where people could register and communicate during the outage. At least with Mastodon this seems to be working in practice. Over-populated instances usually switch to invite-only registration.
Federation would, technically, decentralize the network and ensure there is no single point of failure. Federation would mean Matrix is about as likely to have a network-wide outage as email (or, for a more apt comparison, Mastodon).
Realistically, I understand that the Matrix.org homeserver is an enormous part of the network and an outage there would be comparable to this one.
However I believe that to be an orthogonal problem that also should be solved.
I'm actually curious, why does Matrix promote the Matrix.org homeserver so strongly? Why isn't there an easy link to community-run homeservers?
If you look at Mastodon for example [0], their onboarding process directs you to a registry of community servers [1], which actively promotes diversity in the network and therefore network resilience and reliability.
Matrix meanwhile, just links you to element.io, which creates an account on the matrix.org homeserver.
[0]: https://joinmastodon.org/
[1]: https://joinmastodon.org/communities
Realistically, I understand that the Matrix.org homeserver is an enormous part of the network and an outage there would be comparable to this one.
However I believe that to be an orthogonal problem that also should be solved.
I'm actually curious, why does Matrix promote the Matrix.org homeserver so strongly? Why isn't there an easy link to community-run homeservers?
If you look at Mastodon for example [0], their onboarding process directs you to a registry of community servers [1], which actively promotes diversity in the network and therefore network resilience and reliability.
Matrix meanwhile, just links you to element.io, which creates an account on the matrix.org homeserver.
[0]: https://joinmastodon.org/
[1]: https://joinmastodon.org/communities
"Oh no! Ten million people set up their own web page yesterday!
The web broke. We just have wait a bit while the Elders of the Internet sort this one out."
Said no one ever.
The web broke. We just have wait a bit while the Elders of the Internet sort this one out."
Said no one ever.
They're asking people to share their debug logs to help diagnose this potential issue: https://community.signalusers.org/t/help-needed-please-send-...
Came here to post this. Extra important since they don't collect analytics!
FYI the debug logs are a very curious part of all this. Signal streams them usually to a domain they keep their ownership of somewhat hidden. The domain is: debuglogs.org and the endpoint is just api.debuglogs.org.
It appears to ultimately just front for AWS S3 backend so a very common architectural pattern.
It appears to ultimately just front for AWS S3 backend so a very common architectural pattern.
I'm not sure I understood what the curious part about the debug logs was.
Unfortunately, I believe this is correct. I received over 1,000 messages from one contact of mine that had the message of "secure session reset". It seems his phone tried to reset the encrypted connection with me over and over. Considering that I have 3 devices in total, that's thousands of messages just from one user alone.
I'm sure millions of devices doing the same thing probably bogged them down.
I'm sure millions of devices doing the same thing probably bogged them down.
I had this with one contact as well, the message on Android was Bad encrypted message and on desktop it was Error handling incoming message.
This is why you need to implement exponential backoff on retries for anything at scale https://en.wikipedia.org/wiki/Exponential_backoff
Looks like they had exponential backoff but they didn’t have any jitter (among other things)
Helpful comment from this thread: https://news.ycombinator.com/item?id=25803251
Helpful comment from this thread: https://news.ycombinator.com/item?id=25803251
Exponential backoff retry time slots can be purchased using bitcoin.
Is this what's commonly seen as "Retrying connection in:" with a number of seconds increasing like 5 -> 10 -> 20 -> 40 -> 80 etc?
Could be. If implementing exponential backoff it’s always a good idea to add some jitter (randomness) to the backoff, otherwise you can end up with multiple processes retrying all at the same time, all backing off again, and then all retrying at the same time etc etc
Called Thundering Herd problem.
This is why you need some form of network analytics to figure out if the new version you are rolling out is breaking anything or even seeing of something _is_ breaking. I get the "we don't want to compromise security even an inch", but how secure is a messaging app that cannot send messages?
Even more critical since anyone can write clients that function with the server and thus a malicious actor could attach them without them knowing how to identify it or stop it.
Even more critical since anyone can write clients that function with the server and thus a malicious actor could attach them without them knowing how to identify it or stop it.
I'm not sure it had anything to do with "a new version rolling out", the issue has always been there it just never had a trigger (massive enough load) up until this point.
I suppose the app that can't send messages is the most secure of all! All joking aside if their focus is security over anything else then this is acceptable. Unfortunately I don't think most of the users have the same order of priorities so it's bound to create some tension over time in multiple ways such as testing analytics, features that don't get added, or just general friction due to always requiring "most secure".
The last part I agree is probably the most concerning though, it seems like Signal's centralized services aren't ready to be battle tested from attacks bound to come due to it's popularity growth. I'm not sure matrix is the silver bullet to that problem either... email isn't resilient to DoS because it's distributed it's resilient because the key centralized players like Gmail can withstand constant attack without affecting service due to their scale. Neither Signal nor Matrix are ready for the attention that comes with serving billions from that perspective. It's something I think will come with growing pains though.
I suppose the app that can't send messages is the most secure of all! All joking aside if their focus is security over anything else then this is acceptable. Unfortunately I don't think most of the users have the same order of priorities so it's bound to create some tension over time in multiple ways such as testing analytics, features that don't get added, or just general friction due to always requiring "most secure".
The last part I agree is probably the most concerning though, it seems like Signal's centralized services aren't ready to be battle tested from attacks bound to come due to it's popularity growth. I'm not sure matrix is the silver bullet to that problem either... email isn't resilient to DoS because it's distributed it's resilient because the key centralized players like Gmail can withstand constant attack without affecting service due to their scale. Neither Signal nor Matrix are ready for the attention that comes with serving billions from that perspective. It's something I think will come with growing pains though.
Having gotten client-side retry logic wrong before, while also being the sysadmin responsible for keeping the server side up (back in the days of using a single dedicated server), I'm sympathetic to the Signal staff right now. We should go easy on them.
I have done the same mistake with my own app a few years ago. Fortunately I had a spend limit set on Google App Engine and it was set quite low, it could have bankrupted me :-). I believe that not many mobile developers actually think about this "retry hug" problem until it bites them. It's easy to just write few lines of automatic retries and think that the server will be fine.
[deleted]
It’s easy to judge, but distributed systems are hard to program correctly, and hard to test. One would need to run failure injection tests at scale to detect this.
This is made worse by mobile device virtualization being difficult to achieve as well. Is there a system I could use to spin up 10k emulated iPhones, to run a test in a CI job, for example?
This is made worse by mobile device virtualization being difficult to achieve as well. Is there a system I could use to spin up 10k emulated iPhones, to run a test in a CI job, for example?
It is easy to judge.
At an old job of mine we had devices we controlled that could be rebooted. The initialization sequence after a reboot is much more network heavy than normal operation. We tried to push a critical security patch to all devices at once, meaning they all rebooted at once. Whoops self-DDOS.
At an old job of mine we had devices we controlled that could be rebooted. The initialization sequence after a reboot is much more network heavy than normal operation. We tried to push a critical security patch to all devices at once, meaning they all rebooted at once. Whoops self-DDOS.
First rule of network programming: conservative sending, liberal receiving
That has turned out to be a really terrible rule.
It makes it very hard, sometimes impossible to upgrade a protocol, because some peer somewhere will try to interpret your new thing if it were a broken old thing; so security holes get enshrined. It is much better to fail fast, and make it easy to see why. The only people who benefit from enabling crap implementations, in the end, are makers of crap implementations. Their stuff should fail as soon as possible, so they have to fix it.
In some cases, it is acceptable have a "permit crap" mode that can be turned on in cases where the other end is not negotiable. But ideally the crap fails to work the first time it is plugged in, and returned to the manufacturer. Stuff that causes no end of trouble gets replaced eventually, but somebody has to expose the trouble, and it is best to have lots of company so the culprit is blamed.
It makes it very hard, sometimes impossible to upgrade a protocol, because some peer somewhere will try to interpret your new thing if it were a broken old thing; so security holes get enshrined. It is much better to fail fast, and make it easy to see why. The only people who benefit from enabling crap implementations, in the end, are makers of crap implementations. Their stuff should fail as soon as possible, so they have to fix it.
In some cases, it is acceptable have a "permit crap" mode that can be turned on in cases where the other end is not negotiable. But ideally the crap fails to work the first time it is plugged in, and returned to the manufacturer. Stuff that causes no end of trouble gets replaced eventually, but somebody has to expose the trouble, and it is best to have lots of company so the culprit is blamed.
> That has turned out to be a really terrible rule.
maybe in the ivory tower of web shit, but it is essential for internet/society. like you said
> in cases where the other end is not negotiable.
aiming for robustness is key if you want to enable a diverse ecosystem.
maybe in the ivory tower of web shit, but it is essential for internet/society. like you said
> in cases where the other end is not negotiable.
aiming for robustness is key if you want to enable a diverse ecosystem.
Explains why DNS was resolving to 127.0.0.1 during the outage. Just didn't think it could be caused by their own app.
To keep clients from DDoSing the server. Probably they set a % of requests to resolve to localhost and work with less load. It's a pretty good kill switch.
I remember a problem like this I ran into with a website where a short outage occured, then an avalanche of demand as end users reloaded or retried the page.
We settled on throttling demand based on the incoming IP at the firewall...dropping incoming http packets for 3/4 of the addr space, then later 1/2, then 1/4, etc.
That was long ago, though, long enough that it was an NetScape webserver.
We settled on throttling demand based on the incoming IP at the firewall...dropping incoming http packets for 3/4 of the addr space, then later 1/2, then 1/4, etc.
That was long ago, though, long enough that it was an NetScape webserver.
I've noticed that on Android, WhatsApp and Gmail sync their data on a periodic basis (probably using WorkManager or something like that).
What is an efficient architecture for something like this?
Are millions of devices hitting some gateway->load balancer everytime the Workmanager wakes up and polling for whether new messages have arrived?
For example, if my phone does a "sync" 3 minutes from now, as do millions of others around the world, do we all get routed to our respective box that's keeping a queue of our pending messages in memory? If we happen to receive these messages in between the poll intervals via persistent socket or push notification, then the system wipes the in-memory queue.
Am I imagining this correctly or is there a more optimal architecture for something like this?
What is an efficient architecture for something like this?
Are millions of devices hitting some gateway->load balancer everytime the Workmanager wakes up and polling for whether new messages have arrived?
For example, if my phone does a "sync" 3 minutes from now, as do millions of others around the world, do we all get routed to our respective box that's keeping a queue of our pending messages in memory? If we happen to receive these messages in between the poll intervals via persistent socket or push notification, then the system wipes the in-memory queue.
Am I imagining this correctly or is there a more optimal architecture for something like this?
So does this mean every message I sent till the update isn't rolled out to the client is still DDoSing them ?
Thermal runaway. Easily done and I suppose hard to see as a potential problem until you do it to yourself.
What's up with Elon's tweet about Signal... wtf?
He's just trying to be part of the conversation in a "how do you do, fellow kids" way.
that’s not a ddos. it’s just called not being able to handle a spike in traffic.
The application has a retry mechanism that will keep trying until a connection succeeds (with an exponential backoff), but it doesn't handle a case where the server is basically rejecting the messages on purpose (due to capacity issues). Suddenly all applications will start retrying the connection at once and there is no way to turn if off. So it will make the problem even much worse than it already is. They have added a server feature flag yesterday to change the maximum backoff time and they also added the handling of HTTP 508 error response.
Commits from yesterday:
Feature flag for maximum retry backoff time: https://github.com/signalapp/Signal-Android/commit/93e9dd642...
Add jitter to backoff time: https://github.com/signalapp/Signal-Android/commit/8f7fe5c3e...
Handle ServerRejectedException (HTTP 508): https://github.com/signalapp/Signal-Android/commit/c95f0fce6...
Feature flag for automatic session reset: https://github.com/signalapp/Signal-Android/commit/a3c7e7e55...
Feature flag for maximum retry backoff time: https://github.com/signalapp/Signal-Android/commit/93e9dd642...
Add jitter to backoff time: https://github.com/signalapp/Signal-Android/commit/8f7fe5c3e...
Handle ServerRejectedException (HTTP 508): https://github.com/signalapp/Signal-Android/commit/c95f0fce6...
Feature flag for automatic session reset: https://github.com/signalapp/Signal-Android/commit/a3c7e7e55...
Can somebody clarify if I'm correct in believing that feature flags are client features that can be activated server-side (as opposed to an app update)?
These can be set by the server, you don't need to update the app.
"A location for flags that can be set locally and remotely. These flags can guard features that are not yet ready to be activated."
Here is the file with javadoc on top:
https://github.com/signalapp/Signal-Android/blob/2c1c6fab356...
"A location for flags that can be set locally and remotely. These flags can guard features that are not yet ready to be activated."
Here is the file with javadoc on top:
https://github.com/signalapp/Signal-Android/blob/2c1c6fab356...
That is correct. Typically used to enable or modify a feature in the client on your own time table vs. App Store approval.
The technical term for this is “thundering herd”
I'm wondering if another way to handle that would be the server accepting the request, but not actually servicing it. Basically let it timeout so that the client doesn't retry (at least, does not retry before timeout)
USDOS - Unexpected Success Denial Of Service.
I woke up one Monday morning to crashed servers and angry phone calls once in that situation. Unfortunately it wasn't success on my part but what I had done the Friday before was so crap it couldn't handle a hundred users. But we blamed it loudly on unexpected success, quickly patched the mistake and everyone was happy. YMMV :)
I woke up one Monday morning to crashed servers and angry phone calls once in that situation. Unfortunately it wasn't success on my part but what I had done the Friday before was so crap it couldn't handle a hundred users. But we blamed it loudly on unexpected success, quickly patched the mistake and everyone was happy. YMMV :)
Lesson learned: never ship on Friday
Which is basically what a DDoS will do with its botnet. Nthis case it was an involuntary one without a C&C.
The general case to use Erlang for your reliability and scalability issues. Plenty of suggestions and time to switch. [0]
Now after the fact of its popularity from the source of it all [1].
My response: Use Erlang.
Downvoters: So for a chat app that is still down for several hours and needs to be reliable to handle the scale of new users, what should they (Signal) have used instead? Java?
[0] https://mobile.twitter.com/ejpcmac/status/977192665489035265
[1] https://mobile.twitter.com/anildigital/status/13502749007375...
Now after the fact of its popularity from the source of it all [1].
My response: Use Erlang.
Downvoters: So for a chat app that is still down for several hours and needs to be reliable to handle the scale of new users, what should they (Signal) have used instead? Java?
[0] https://mobile.twitter.com/ejpcmac/status/977192665489035265
[1] https://mobile.twitter.com/anildigital/status/13502749007375...
Ah, but they are using Java [0] - and obviously they need a solution that works with the stack they currently have - proposing Erland at this time is asinine.
Erlang would have been solution if they have predicted the brain-dead actions of Whatsapp/Facebook, and if they had ample time to prepare - and it would still have some hard limits on a number of messages it can handle.
[0] https://github.com/signalapp/Signal-Server
Erlang would have been solution if they have predicted the brain-dead actions of Whatsapp/Facebook, and if they had ample time to prepare - and it would still have some hard limits on a number of messages it can handle.
[0] https://github.com/signalapp/Signal-Server
Java is much capable of handling any sort of challenges, it's not so much about programming language from the looks of it, it's about architecting their backend and networking issues.
> Ah, but they are using Java
Exactly my point. It’s still down. Their Java servers doesn’t seem to handle this scale well and aren’t reliable for this.
The fact that they have to “Add more servers” to handle this traffic tells you exactly why Erlang / Elixir or BEAM VM languages are better suited for this (And also can save them money instead of provisioning more servers on AWS).
> obviously they need a solution that works with the stack they currently have
It was down before even when they had less users. One user made this suggestion 3 years ago, and Signal looked at this. Plenty of time to switch.
> proposing Erland at this time is asinine.
You mean 3 years ago with their $60M in funding? Plenty of time and money to hire an Erlang / Elixir consultant.
Exactly my point. It’s still down. Their Java servers doesn’t seem to handle this scale well and aren’t reliable for this.
The fact that they have to “Add more servers” to handle this traffic tells you exactly why Erlang / Elixir or BEAM VM languages are better suited for this (And also can save them money instead of provisioning more servers on AWS).
> obviously they need a solution that works with the stack they currently have
It was down before even when they had less users. One user made this suggestion 3 years ago, and Signal looked at this. Plenty of time to switch.
> proposing Erland at this time is asinine.
You mean 3 years ago with their $60M in funding? Plenty of time and money to hire an Erlang / Elixir consultant.
You realize erlang isn't a silver bullet right?
[deleted]
> The fact that they have to “Add more servers” to handle this traffic tells you exactly why Erlang / Elixir or BEAM VM languages are better suited for this (And also can save them money instead of provisioning more servers on AWS).
You do realize that the major reason BeamVM is popular for this use case is because it makes it easy to add more servers, right? Erlang doesn't magically make it so you never have to add more servers. Actually, it's the opposite. Erlang is pretty computationally inefficient compared to most statically typed languages like Java. If there is any VM out there preventing your from needing more servers, it's the JVM. Generally speaking you can do more work with fewer resources with the JVM than you can with the BeamVM...it's just that the BeamVM makes it easier to add more resources.
You do realize that the major reason BeamVM is popular for this use case is because it makes it easy to add more servers, right? Erlang doesn't magically make it so you never have to add more servers. Actually, it's the opposite. Erlang is pretty computationally inefficient compared to most statically typed languages like Java. If there is any VM out there preventing your from needing more servers, it's the JVM. Generally speaking you can do more work with fewer resources with the JVM than you can with the BeamVM...it's just that the BeamVM makes it easier to add more resources.
Problem: Client retries every n seconds when it receives a 5xx error. This causes outage amplification
Proposed fix: Rebuild the entire backend in a different language.
estimated time to outage mitigation: 6-12 months.
Now colour me "inexperienced" but I'd personally push a fix for exponential backoff to the client.
Proposed fix: Rebuild the entire backend in a different language.
estimated time to outage mitigation: 6-12 months.
Now colour me "inexperienced" but I'd personally push a fix for exponential backoff to the client.
How does erlang fix the problem of not having enough physical servers?
I think what GP is referring to is that WhatsApp's Erlang based system was able to handle 2M connections per server allowing you to do more with less [0] (although I think I read since the FB acquisition, WhatsApp has moved off from Erlang).
[0] http://highscalability.com/blog/2014/2/26/the-whatsapp-archi...
[0] http://highscalability.com/blog/2014/2/26/the-whatsapp-archi...
There are lots of languages that can do this, and it requires deliberate engineering effort in all of them, including erlang. I mean, you have to have all of your ports open, with multiple IP addresses allocated to a single server, and the process has to have efficient routing algorithms to do so. Erlang doesn't make this job more efficient than other languages. Erlang has two killer use cases: fault tolerance and horizontal scalability. Computational resource efficiency has never been a goal and it shows.
The grandparent comment is nothing more than a cargo cult. "If we use the same language as WhatsApp, we'll magically be just as scalable as WhatsApp". It's horrendously naive.
The grandparent comment is nothing more than a cargo cult. "If we use the same language as WhatsApp, we'll magically be just as scalable as WhatsApp". It's horrendously naive.
> I mean, you have to have all of your ports open, with multiple IP addresses allocated to a single server
Not sure what this section is about. IP connections are keyed on a 5-tuple of
{PeerAIP, PeerBIP, Protocol, PeerAPort, PeerBPort}
If you hold PeerAIP and PeerAPort and protocol fixed (single server IP, single port, TCP), you can accept 65535 connections from any other IP (so long as the system managing that IP can use all the ports; if it's CGNAT, it probably can; if it's a traditional OS, it will probably have soft limits, but easily thousands).
Not sure what this section is about. IP connections are keyed on a 5-tuple of
{PeerAIP, PeerBIP, Protocol, PeerAPort, PeerBPort}
If you hold PeerAIP and PeerAPort and protocol fixed (single server IP, single port, TCP), you can accept 65535 connections from any other IP (so long as the system managing that IP can use all the ports; if it's CGNAT, it probably can; if it's a traditional OS, it will probably have soft limits, but easily thousands).
65,535 is substantially less than 2,000,000. At a minimum, you'd need 31 IP addresses allocated to a single box in order to have 2,000,000 active connections on that box. Unless you're arguing that the chat clients would be opening dozens of connections...but that's ridiculous. Whatsapp was built for actual usage, not research projects, and more than 1-2 connections per client is overkill.
Let me be more concrete.
If your server is 192.0.2.23, and it listens on port 5222.
You can accept 65535 connections from 198.51.100.1.
198.51.100.1:X to 192.0.2.23:5222 for X in {1:65535} (port zero is weird)
You can also accept 65535 connections from 198.51.100.2
198.51.100.2:X to 192.0.2.23:5222 for X in {1:65535}
Pick any X, like say 12345; a connection between 198.51.100.1:12345 and 192.0.2.23:5222 does not preclude a connection between 198.51.100.2:12345 and 192.0.2.23:5222.
Thus, this only gets to be an issue if you have a large number of potential clients behind a single (public) IP address; which is generally not the case (except when you're testing from a load generator)
If your server is 192.0.2.23, and it listens on port 5222.
You can accept 65535 connections from 198.51.100.1.
198.51.100.1:X to 192.0.2.23:5222 for X in {1:65535} (port zero is weird)
You can also accept 65535 connections from 198.51.100.2
198.51.100.2:X to 192.0.2.23:5222 for X in {1:65535}
Pick any X, like say 12345; a connection between 198.51.100.1:12345 and 192.0.2.23:5222 does not preclude a connection between 198.51.100.2:12345 and 192.0.2.23:5222.
Thus, this only gets to be an issue if you have a large number of potential clients behind a single (public) IP address; which is generally not the case (except when you're testing from a load generator)
Absolutely, no doubt. Personally, I think Erlang is really interesting and is very well designed (kudos to Joe Armstrong!). But I find that Golang can do it just as well if not better -- it's also far easier to cross compile (e.g. for ARM or FreeBSD) and the ecosystem is much richer and vibrant than Erlang's.
So in my comment, it’s somewhat ‘naive’ to use Erlang for distributed systems like a messaging system and is ‘nothing more than a cargo cult’ all because I said WhatsApp is using it? Ha.
In fact it is the other way round and especially for Signal’s use case.
Signal is already seeking for Erlang expertise in their server developer roles so I’m sure they’re already looking into this.
Perhaps using languages that are not suitable or designed for distributed systems like Java sounds more like a cargo-cult than using Erlang.
They bought into the JVM hype until their server capsized on high traffic (again) and it is still down.
In fact it is the other way round and especially for Signal’s use case.
Signal is already seeking for Erlang expertise in their server developer roles so I’m sure they’re already looking into this.
Perhaps using languages that are not suitable or designed for distributed systems like Java sounds more like a cargo-cult than using Erlang.
They bought into the JVM hype until their server capsized on high traffic (again) and it is still down.
No, you're cargo culting because you think their choice of language caused this...as opposed to the code that they wrote. There is nothing wrong with using Erlang here, it's a perfectly capable language for systems like this. But there is nothing magical about erlang either. I have first hand experience seeing erlang systems choke and croak on 30k connections, and I've seen jvm systems and C++ systems and go systems capably handling north of 1M connections. The difference isn't the language, it is the architecture of the app.
If you don't believe me, I dare you to create an alternative. I'd easily take bets that Signal's current system could easily outperform your cargo cult alternative written in erlang.
If you don't believe me, I dare you to create an alternative. I'd easily take bets that Signal's current system could easily outperform your cargo cult alternative written in erlang.
> I have first hand experience seeing erlang systems choke and croak on 30k connections, and I've seen jvm systems and C++ systems and go systems capably handling north of 1M connections.
Maybe you should also apply that ’experience’ again by applying to their jobs posting here [0] as it seems you easily fit their requirements unless you doubt your own claims.
> If you don't believe me, I dare you to create an alternative. I'd easily take bets that Signal's current system could easily outperform your cargo cult alternative written in erlang.
So you’re expecting someone here to create an open-source alternative to Signal using Erlang for free? With that $60M in funding, Signal can easily hire someone who has ’first hand experience’ just like you to rearchitect their systems to be more reliable and fault tolerant.
The job [0] is still there. I can also bet you won’t help them out, even with your ’experience’.
[0] https://jobs.lever.co/signal/2a5fee8b-5875-46d4-a41d-773a28a...
Maybe you should also apply that ’experience’ again by applying to their jobs posting here [0] as it seems you easily fit their requirements unless you doubt your own claims.
> If you don't believe me, I dare you to create an alternative. I'd easily take bets that Signal's current system could easily outperform your cargo cult alternative written in erlang.
So you’re expecting someone here to create an open-source alternative to Signal using Erlang for free? With that $60M in funding, Signal can easily hire someone who has ’first hand experience’ just like you to rearchitect their systems to be more reliable and fault tolerant.
The job [0] is still there. I can also bet you won’t help them out, even with your ’experience’.
[0] https://jobs.lever.co/signal/2a5fee8b-5875-46d4-a41d-773a28a...
Nah, not in the mood for a voluntary paycut and demotion. But even if I was, I'm not the one running around telling extremely experienced devs that they should rewrite their apps in a different language because of a "web scale" cargo cult. I have nothing to prove to them, unlike you.
> Nah, not in the mood for a voluntary paycut and demotion.
No worries, It's understandable to not be in the mood to take a role where 10 to 20 million+ concurrent users already scares you to not bother applying anyway, despite your 'experience'.
> But even if I was, I'm not the one running around telling extremely experienced devs that they should rewrite their apps in a different language because of a "web scale" cargo cult.
So even when Signal is actually seeking those who have 'Deep knowledge of Erlang experience' in their server-side roles, In your view, they and I are 'cargo-culting'? Come on now.
> I have nothing to prove to them, unlike you.
Says the person who just yelled 'I have first hand experience' and then immediately bails out when challenged to apply said experience after 'daring' someone to create another competing open source messaging app using Erlang for free. Ha.
Sounds like it would have just been better to not even mention you have said 'experience' after all.
No worries, It's understandable to not be in the mood to take a role where 10 to 20 million+ concurrent users already scares you to not bother applying anyway, despite your 'experience'.
> But even if I was, I'm not the one running around telling extremely experienced devs that they should rewrite their apps in a different language because of a "web scale" cargo cult.
So even when Signal is actually seeking those who have 'Deep knowledge of Erlang experience' in their server-side roles, In your view, they and I are 'cargo-culting'? Come on now.
> I have nothing to prove to them, unlike you.
Says the person who just yelled 'I have first hand experience' and then immediately bails out when challenged to apply said experience after 'daring' someone to create another competing open source messaging app using Erlang for free. Ha.
Sounds like it would have just been better to not even mention you have said 'experience' after all.
A language is but just a brush in which to paint your logic.
Granted its a lot more difficult to have 2m open connections in python. It is possible though.
You're also assuming that Signal uses the same connection scheme that whatsapp does, which I highly doubt.
Granted its a lot more difficult to have 2m open connections in python. It is possible though.
You're also assuming that Signal uses the same connection scheme that whatsapp does, which I highly doubt.
First of all, they’re on AWS. They don’t ‘have’ physical servers.
Secondly, the point of Erlang is that you do not need that many physical/virtual servers to scale to these millions of users. Perhaps just a few or the same number of servers they have, but running Erlang instead. The BEAM VM and languages based on this is quite frankly designed for this scale.
Still working well for WhatsApp and Discord.
Secondly, the point of Erlang is that you do not need that many physical/virtual servers to scale to these millions of users. Perhaps just a few or the same number of servers they have, but running Erlang instead. The BEAM VM and languages based on this is quite frankly designed for this scale.
Still working well for WhatsApp and Discord.
This post is about the app not gracefully handling server errors. Server errors are inevitable no matter what your stack is.
Circuit breaking, exponential backoff, bulkheading should be key pillars and no dev should be allowed to write a client without these key skills.
Edit: Please add Jitter to the list of items. Any good library like https://resilience4j.readme.io/ will give you all of this for free out of box!