Building a high-scale chat server on Cloud Run(ahmet.im)
ahmet.im
Building a high-scale chat server on Cloud Run
https://ahmet.im/blog/cloud-run-chat-server/
116 comments
> And if it never takes off, that's fine too because you didn't spend a ton of time investing in making it work.
But if it “takes off” while you are caught with your pants down and you have no clue, you are stuffed with a friggin $60k bill and potentially have to remortgage your house.
But if it “takes off” while you are caught with your pants down and you have no clue, you are stuffed with a friggin $60k bill and potentially have to remortgage your house.
That's why businesses use expected cost to make business decisions. The odds of it taking off are very, very low.
And in the rare case it does, the infra is relatively portable (Docker container) and can be moved to a more-suitable VMs in less than the full month.
And in the rare case it does, the infra is relatively portable (Docker container) and can be moved to a more-suitable VMs in less than the full month.
Exactly, you can repurpose much of your build pipeline and simply boot the same docker image using a vm or scaling group of vms with a single command.
The cost of switching to something else if you actually need to is minimal. I improvised it a few weeks ago when I realized that doing background jobs wasn't something I should be doing on cloud run (it gets throttled). I added an endpoint that processed a data import in the background and that just wouldn't work in cloudrun. So, I wrote a script that runs the same container on a vm and then shuts it down. Took me about 10 minutes to write and it zipped through the import in a few minutes. For a proper setup, you'd want to use terraform or similar to provision the vms but that's not particularly challenging. Basically you point your provisioned infrastructure at your docker container and it will run it.
Very few businesses will ever see a quarter million concurrent users doing anything on their services. If you have that challenge, you can afford a few full time devops people to get you whatever infrastructure you need to handle this.
We've been running on cloud run since August. Standout features for me:
- it created a working CI/CD pipeline for me from my github repository with minimal effort and sane defaults. I got the service up and running within 20 minutes of me deciding to give it a try. I was so impressed with this that I decided on the spot to continue using it. That 20 minutes of devops time is technically the largest expense we've had so far on this setup as we mostly stay below the freemium threshold and it's otherwise a zero maintenance thing for us. Merge to master ... code deploys & runs. We've done this hundreds of times in the last months.
- comes with proper logging, monitoring, etc. tooling with absolutely no additional effort. I don't consider running blind an option so this stuff is usually not optional for me.
- we have no cloud run specifics in our github repository beyond the cloud build yml file. There is no vendor lock in. If you need a cheap way to run your docker container; this is it. Infrastructure is a blackbox that you get as a service. That's how infrastructure should be these days.
- once we got in to the test program; web sockets have been great as well. We just launched that feature a few days ago.
- low cost, low hassle, yet ticks all the boxes on the operational front. This is important to me as we don't have 24x7 people on staff and I hate having to deal with outages on a weekend or at night (as the CTO, I'm basically the person that gets to deal with this).
- I've been involved with devops on other projects. It can be a serious time sink. It certainly has sucked up months of my time. Many projects end up having several full time people dealing with that at typically premium consulting rates. Days of tinkering with terraform, chasing stackoverflow posts, reading piles and piles of documentation, etc. It just never ends. That stuff costs magnitudes more than you are likely to spend on hosting in your first year. Even the most expensive fancy AWS setup is a rounding error compared to that. Cloud run on the other hand is dead easy to setup and dirt cheap for small setups. It's great way to defer devops cost; potentially indefinitely.
The cost of switching to something else if you actually need to is minimal. I improvised it a few weeks ago when I realized that doing background jobs wasn't something I should be doing on cloud run (it gets throttled). I added an endpoint that processed a data import in the background and that just wouldn't work in cloudrun. So, I wrote a script that runs the same container on a vm and then shuts it down. Took me about 10 minutes to write and it zipped through the import in a few minutes. For a proper setup, you'd want to use terraform or similar to provision the vms but that's not particularly challenging. Basically you point your provisioned infrastructure at your docker container and it will run it.
Very few businesses will ever see a quarter million concurrent users doing anything on their services. If you have that challenge, you can afford a few full time devops people to get you whatever infrastructure you need to handle this.
We've been running on cloud run since August. Standout features for me:
- it created a working CI/CD pipeline for me from my github repository with minimal effort and sane defaults. I got the service up and running within 20 minutes of me deciding to give it a try. I was so impressed with this that I decided on the spot to continue using it. That 20 minutes of devops time is technically the largest expense we've had so far on this setup as we mostly stay below the freemium threshold and it's otherwise a zero maintenance thing for us. Merge to master ... code deploys & runs. We've done this hundreds of times in the last months.
- comes with proper logging, monitoring, etc. tooling with absolutely no additional effort. I don't consider running blind an option so this stuff is usually not optional for me.
- we have no cloud run specifics in our github repository beyond the cloud build yml file. There is no vendor lock in. If you need a cheap way to run your docker container; this is it. Infrastructure is a blackbox that you get as a service. That's how infrastructure should be these days.
- once we got in to the test program; web sockets have been great as well. We just launched that feature a few days ago.
- low cost, low hassle, yet ticks all the boxes on the operational front. This is important to me as we don't have 24x7 people on staff and I hate having to deal with outages on a weekend or at night (as the CTO, I'm basically the person that gets to deal with this).
- I've been involved with devops on other projects. It can be a serious time sink. It certainly has sucked up months of my time. Many projects end up having several full time people dealing with that at typically premium consulting rates. Days of tinkering with terraform, chasing stackoverflow posts, reading piles and piles of documentation, etc. It just never ends. That stuff costs magnitudes more than you are likely to spend on hosting in your first year. Even the most expensive fancy AWS setup is a rounding error compared to that. Cloud run on the other hand is dead easy to setup and dirt cheap for small setups. It's great way to defer devops cost; potentially indefinitely.
Thank you for an informative comment. One question:
> I realized that doing background jobs wasn't something I should be doing on cloud run (it gets throttled)
What do you mean by "it gets throttled". Do you mean that you exceed the CPU/memory limit in a container. Maybe you just have to plan the capacity for the background jobs: connections vs CPU vs memory. We use Cloud Run for background jobs with 0 issues.
> I realized that doing background jobs wasn't something I should be doing on cloud run (it gets throttled)
What do you mean by "it gets throttled". Do you mean that you exceed the CPU/memory limit in a container. Maybe you just have to plan the capacity for the background jobs: connections vs CPU vs memory. We use Cloud Run for background jobs with 0 issues.
We ran with 0 hosts so in between requests, we had a lot of things failing or getting killed. The pattern of firing up a background thread from a request, returning immediately, and processing in the background simply did not work for us. The same logic worked fine elsewhere so it really was the environment. You can't really use it like a stateful environment.
In some cases we saw no evidence of the job even starting because the whole thing died right after the request returned. In other cases it would start but never finish. The actual import jobs weren't that big even. It was just super flaky.
The whole problem went away as soon as we gave it a dedicated vm. All I did was run the same container and then trigger the same request via a curl command sshed into the vm. Ugly but it got the job done.
In some cases we saw no evidence of the job even starting because the whole thing died right after the request returned. In other cases it would start but never finish. The actual import jobs weren't that big even. It was just super flaky.
The whole problem went away as soon as we gave it a dedicated vm. All I did was run the same container and then trigger the same request via a curl command sshed into the vm. Ugly but it got the job done.
But if it doesn't really take off - why do you need it? The main benefit of easy scaling strikes me as the scaling, but if you don't want to scale too much for cost reasons...
In that case why not use digital ocean or similar cheap VM offering. Cheaper, scales in near term. And provides a path forward to a more enterprise stack if necessary.
I think people underestimate the serving capabilities of a few VMs coupled with caching/cdn.
I think people underestimate the serving capabilities of a few VMs coupled with caching/cdn.
If it takes off and you're supporting millions of customers, a $60k infra bill is an amazing problem to have.
"Oh no, we're too popular!". Who says this?
"Oh no, we're too popular!". Who says this?
We had a logging service that cost us 30k / mo.
$60/k for your breadwinner seems like a deal.
$60/k for your breadwinner seems like a deal.
I personally see the flip side, the original logging service at 30k/mo is totally wild. How much data was being logged/searched if you don't mind me asking?
About 300-600GB / hr. Data retention was based on a few different rules but could be stored for up to a month.
30k was ingress and ETL, that amount didn’t include storage. Not sure what storage costs were off-hand.
30k was ingress and ETL, that amount didn’t include storage. Not sure what storage costs were off-hand.
Thanks for sharing, that's a lot of logs but another good example of how linear scaling costs hurts. A lot cheaper than standard AWS CloudWatch which reckons it would be $200k+ a month.. but though 600GB an hr is not an easy problem to solve but that is a hell of a premium on both.
If you can constantly have 250k concurrent users. I don't think money would be an issue at that time.
[deleted]
1) you can cap the max instances in GCR. You can put a limit of like 2 instances and ramp it up gradually.
2) it's 60k for a full month running at 100% capacity. Chances are you will notice this.
This blog post is showing the upper limits of GCR, not the typical use case.
2) it's 60k for a full month running at 100% capacity. Chances are you will notice this.
This blog post is showing the upper limits of GCR, not the typical use case.
At this price point hiring a dedicated dev seems like a no brainer. Also using GCR name for this is a bit weird when there’s already another GCP container product with the same exact name.
Yeah but the price starts at $0 and scales linearly with traffic.
> Also using GCR name for this is a bit weird when there’s already another GCP container product with the same exact name.
Very true! Naming is hard
> Also using GCR name for this is a bit weird when there’s already another GCP container product with the same exact name.
Very true! Naming is hard
And Google seems especially bad at it.
The approach of branding things as Google Cloud XYZ makes for really homogeneous acronyms. It solves the problem that AWS has with terrible names (what the hell is a “Redshift”?) by giving everything a boring name, but in exchange you get GCS being the acronym for Google Cloud Storage, Google Cloud Spanner, Google Cloud Search, Google Cloud SQL...
> The name means to shift away from Oracle, red being an allusion to Oracle, whose corporate color is red and is informally referred to as "Big Red.
I learned this recently and thought I'd share! From https://en.m.wikipedia.org/wiki/Amazon_Redshift
I learned this recently and thought I'd share! From https://en.m.wikipedia.org/wiki/Amazon_Redshift
I actually don't mind the acronymed stuff so much. My primary complaint is that the names are so generic that you can't differentiate the products by name except by memorizing them. For example:
* BigQuery vs BigTable (though, there are at least some hints in these)
* Cloud Storage vs Filestore vs Datastore vs Firestore (vs Firebase? How are these related?)
I can of course make an effort to keep these straight, but then having to clarify in every design convo because my manager can't, or having a hard time googling (lol) things germane to the product I'm on, etc, is just a huge hassle.
* BigQuery vs BigTable (though, there are at least some hints in these)
* Cloud Storage vs Filestore vs Datastore vs Firestore (vs Firebase? How are these related?)
I can of course make an effort to keep these straight, but then having to clarify in every design convo because my manager can't, or having a hard time googling (lol) things germane to the product I'm on, etc, is just a huge hassle.
Agree about firestore vs filestore but if a manager doesn’t know what bigtable is I’d probably consider leaving
Clearly we should start using emoji triplets for service names... easy to read/remember, and easy to type out on modern systems
[deleted]
Spanner is also a bad name for a database. They should've called it Bigertable.
A. It’s not even correct b. It’s a terrible name bc it will get confused in code/configs all the time
In general I agree, but I think this is more an issue of the example, or rather the fact that there's a cap of 250 connections per Cloud Run instance. For something as barebone as chat clients, that really seems to be under utilizing even the single vCPU on the smallest instance.
One thing I haven't seen mentioned in the comments is how easy it is to quickly iterate on Cloud Run. I built a web app that uses Cloud Run for 90% of my server needs (Cloud Functions for the rest). One of my favorite features is that I can make some local changes, upload the container, and click a button to deploy the latest version or A/B test. Then if I notice anything weird I can just as easily revert the changes, which has saved me a few times. Admittedly, I haven't used VMs in years and it might be just as easy there these days with containerized VMs, but I don't know. Cloud Run was so trivial to set up that I didn't want to waste time on alternatives. Another really nice thing is that it integrates nicely with Firebase, which I use for hosting.
As my site matures, I might move some predictable parts to containerized VMs to save on costs. I have a crawler-type service that has extremely consistent traffic that probably shouldn't be on Cloud Run, but I did it anyway due to being able to iterate so quickly (see revisions here: https://imgur.com/fkFtIRM). Cloud Run also has a free tier which was enough when I was starting out and prototyping. It's also nice since my site gets very little traffic at night so at the moment the cost is not too bad. This is what my billing looks like: https://imgur.com/gIo3IGJ
If I was a big company or my site got super popular I might do things differently. But for a side project it has made me enjoy programming more than anything I've used before because I can focus 95% on code.
As my site matures, I might move some predictable parts to containerized VMs to save on costs. I have a crawler-type service that has extremely consistent traffic that probably shouldn't be on Cloud Run, but I did it anyway due to being able to iterate so quickly (see revisions here: https://imgur.com/fkFtIRM). Cloud Run also has a free tier which was enough when I was starting out and prototyping. It's also nice since my site gets very little traffic at night so at the moment the cost is not too bad. This is what my billing looks like: https://imgur.com/gIo3IGJ
If I was a big company or my site got super popular I might do things differently. But for a side project it has made me enjoy programming more than anything I've used before because I can focus 95% on code.
I wanted to edit my post to add more details, but since I can't do that here are some more screenshots:
This is the (very consistent) traffic for my scraper that should probably not be on Cloud Run but it's too convenient for me to switch at the moment: https://imgur.com/RYLdp2t
This is the traffic for my web server, but since it's 30 days it's hard to see that within each day I get very large (relative) spikes in the middle of the day. I also had some huge spikes earlier due to some Reddit posts I made that are not shown in the graph. Being able to automatically scale to any load has been a life saver as I've tried to do marketing on social media. Otherwise I'd need to provision to handle as much as the maximum load I expect: https://imgur.com/RL534de
If you're a startup or need to handle random spikes from social media or other unpredictable sources, I think Cloud Run is worth a look. Another great use if for random jobs you might need to run every now and then. Web servers also seem like a good fit due to the fluctuations depending on the time of day.
This is the (very consistent) traffic for my scraper that should probably not be on Cloud Run but it's too convenient for me to switch at the moment: https://imgur.com/RYLdp2t
This is the traffic for my web server, but since it's 30 days it's hard to see that within each day I get very large (relative) spikes in the middle of the day. I also had some huge spikes earlier due to some Reddit posts I made that are not shown in the graph. Being able to automatically scale to any load has been a life saver as I've tried to do marketing on social media. Otherwise I'd need to provision to handle as much as the maximum load I expect: https://imgur.com/RL534de
If you're a startup or need to handle random spikes from social media or other unpredictable sources, I think Cloud Run is worth a look. Another great use if for random jobs you might need to run every now and then. Web servers also seem like a good fit due to the fluctuations depending on the time of day.
Cloud Run has been fantastic.
I had a worker service running on Heroku. Very CPU intensive. The traffic pattern was extremely low throughout the day, but had completely unexpected surges.
On Heroku, my choices were: Paying $3k (basically paying for peak surge throughout the month) or having a lot of slow/failed responses during the surge.
Moved to Cloud Run very easily. Just a normal dockerized 12 factor app.
Now I pay ~$50/m and it automatically scales up when I need more workers.
If you want a managed app platform, it couldn't be more simple and cheap.
The scaling model of Cloud Run has been great. Now that they support websockets, I will be moving all my apps slowly to it.
My only gripe: I wish they had something for worker processes. I know that the rest of Google Cloud has solutions for it but being able to just spawn worker processes as part of the same deployment would be fantastic.
I had a worker service running on Heroku. Very CPU intensive. The traffic pattern was extremely low throughout the day, but had completely unexpected surges.
On Heroku, my choices were: Paying $3k (basically paying for peak surge throughout the month) or having a lot of slow/failed responses during the surge.
Moved to Cloud Run very easily. Just a normal dockerized 12 factor app.
Now I pay ~$50/m and it automatically scales up when I need more workers.
If you want a managed app platform, it couldn't be more simple and cheap.
The scaling model of Cloud Run has been great. Now that they support websockets, I will be moving all my apps slowly to it.
My only gripe: I wish they had something for worker processes. I know that the rest of Google Cloud has solutions for it but being able to just spawn worker processes as part of the same deployment would be fantastic.
How quick does it spin up?
In my case since its background workers, I don't mind the slow startup and I never measured it. Sorry couldn't be more helpful.
My Go-based containers start in much less than a second. I believe its dozens to maybe low hundreds of milliseconds
Depends on the size of the docker image too, we have one with libreoffice (for generating PDFs, invoices, contracts), it takes almost 3 seconds to start at 550MB
How do you differentiate background and worker processes? I’m currently running VMs for background processes processing pub sub topics. Is that not something that could go on cloud run?
When an instance isn't handling a request it either gets killed almost immediately, or if even if you have min idle instances configured, its CPU get's throttled to nothing. So you can't have goroutines or some such doing anything outside of the request context. It's my team's one gripe with Cloud Run. It'd be great to have an option for long-lived instances.
You can set minimum instances now ( well - currently Preview). https://cloud.google.com/run/docs/configuring/min-instances
Like I said, even if the instance is "alive", if it's not currently handling a request it's CPU is throttled to nothing so you can't trust any background process will finish its work.
You can use an HTTP push subscription to an IAM protected Cloud Run service.
Re: worker processes
Have a look at GCP EventArc. It's a new way to trigger cloud run instances. You could create an EventArc trigger that wakes up a worker image, or some such. Basically,EventArc enables serverless event driven design across all GCP. https://cloudblog.withgoogle.com/topics/developers-practitio...
Have a look at GCP EventArc. It's a new way to trigger cloud run instances. You could create an EventArc trigger that wakes up a worker image, or some such. Basically,EventArc enables serverless event driven design across all GCP. https://cloudblog.withgoogle.com/topics/developers-practitio...
> I wish they had something for worker processes
It may not fit your needs, but did you know that you can invoke other Cloud Run containers from a Cloud Run container?
Alternatively, you can enqueue Cloud Tasks from a Cloud Run container. Those Cloud Tasks can in turn invoke Cloud Run containers.
Not sure if they fit your usecase, but I use both of these approaches and they really expand the capabilities of Cloud Run for me.
It may not fit your needs, but did you know that you can invoke other Cloud Run containers from a Cloud Run container?
Alternatively, you can enqueue Cloud Tasks from a Cloud Run container. Those Cloud Tasks can in turn invoke Cloud Run containers.
Not sure if they fit your usecase, but I use both of these approaches and they really expand the capabilities of Cloud Run for me.
Oh dear. “$87/hour, which is $62.6K/month” for only 250k concurrent connections?
Around €1500 will get you a second hand Dell R720 with 190+ GB RAM and 48 cores.
It would be cheaper to handle such scaling with a vm instead of functions. Any kernel with a bit of tweaking will easily handle that number of connections. One just needs RAM and sufficient CPU. 16GB RAM and 8 vCPUs on aws will be enough for 500k.
Sleeping would be very uncomfortable if my service using such approach started being popular and was running at increasing capacity for a week because my team needs time to move back to vms.
Around €1500 will get you a second hand Dell R720 with 190+ GB RAM and 48 cores.
It would be cheaper to handle such scaling with a vm instead of functions. Any kernel with a bit of tweaking will easily handle that number of connections. One just needs RAM and sufficient CPU. 16GB RAM and 8 vCPUs on aws will be enough for 500k.
Sleeping would be very uncomfortable if my service using such approach started being popular and was running at increasing capacity for a week because my team needs time to move back to vms.
> It would be cheaper to handle such scaling with a vm instead of functions.
Ahmet makes that exact point:
> In the long term, as your load becomes more predictable, it makes more sense to move to VM-based compute (such as GCE or GKE) as several mid-size virtual machines can handle the same load, potentially 50x cheaper.
The shiny spot for something like Cloud Run -- or Knative -- is varying workloads. I described this as "workloads with unpredictable, latency-insensitive demand" in my book on the subject[0].
[0] https://livebook.manning.com/book/knative-in-action/chapter-...
Ahmet makes that exact point:
> In the long term, as your load becomes more predictable, it makes more sense to move to VM-based compute (such as GCE or GKE) as several mid-size virtual machines can handle the same load, potentially 50x cheaper.
The shiny spot for something like Cloud Run -- or Knative -- is varying workloads. I described this as "workloads with unpredictable, latency-insensitive demand" in my book on the subject[0].
[0] https://livebook.manning.com/book/knative-in-action/chapter-...
What I never get with these is: one has to really go out of their way to run the same code on a vm and serverless. So why even bother building the thing twice?
Do people really consider putting a server app in a function?
Do people really consider putting a server app in a function?
This is the beauty of cloud run. There is literally no difference in how you build your application. I can run my codebase locally using docker with no changes and no dependencies on the type of external libraries you'd need for AWS lambda or cloud functions.
Yes it's called a fat lambda
> So why even bother building the thing twice?
Time elapses. What makes sense today may not tomorrow. It's fine to adapt designs when the economics point that way. Also fine to leave things as they are when the economics point that way.
Time elapses. What makes sense today may not tomorrow. It's fine to adapt designs when the economics point that way. Also fine to leave things as they are when the economics point that way.
Yep we’re running our API at work in a lambda behind API gateway.
It’s “interesting”
It’s “interesting”
Sounds fun! Could you elaborate on the “interesting” part a bit more?
On one hand it’s very easy to run the code locally or in lambda because we use Go along with a library from AWS that translates the gateway proxy request event to a normal http request. So the request handling is all the same in code.
To prevent 404s hitting your function you have to define your routes in both your code and in api gateway. This starts to marry your code to your deployment.
Observability is now harder and you need yet another thing. We’re using X-ray with datadog.
To “safely” talk to Postgres you should use an rds proxy. It’s still unclear how much is different from pgbouncer or similar but in any case it’s pretty easy to knock over your db with runaway lambda invocations.
I could go on about the mistake we made of using SAM and how it ends up not supporting everything you need but I guess my main point is it’s easy to get sucked in by the idea of it being easy but you are trading that for service / operational complexity with a healthy dose of platform lock-in.
To prevent 404s hitting your function you have to define your routes in both your code and in api gateway. This starts to marry your code to your deployment.
Observability is now harder and you need yet another thing. We’re using X-ray with datadog.
To “safely” talk to Postgres you should use an rds proxy. It’s still unclear how much is different from pgbouncer or similar but in any case it’s pretty easy to knock over your db with runaway lambda invocations.
I could go on about the mistake we made of using SAM and how it ends up not supporting everything you need but I guess my main point is it’s easy to get sucked in by the idea of it being easy but you are trading that for service / operational complexity with a healthy dose of platform lock-in.
This is a pretty good example of how Cloud Run is better, as those connections to the database would be divided by 250 (lambda supports no concurrency). Lambda supports docker containers now but Cloud Run was also ahead of the curve there.
Yep. Running adhoc containers on demand is clearly the right pattern. Still plenty of sharp edges but gives you a very clear path to lift your system admin / structure skills in to "serverless".
I need to spend some more time playing with Cloud Run because I wasn't aware of the concurrency bit you are mentioning.
I need to spend some more time playing with Cloud Run because I wasn't aware of the concurrency bit you are mentioning.
Oh god, when's the funeral for your bank account?
Cloud Run uses docker containers, so the code can very easily be the same.
I'd put a server app up behind a cloud run endpoint, sure, as long as it can fit in a few GBs and start in a few 100 ms. The scale-to-0 aspect is nice, and the scale up aspect is nice too.
I'd put a server app up behind a cloud run endpoint, sure, as long as it can fit in a few GBs and start in a few 100 ms. The scale-to-0 aspect is nice, and the scale up aspect is nice too.
For me the selling point of cloud run is that is easy to develop and debug: make it run locally, test it, then ship it in a container.
For GCP cloud functions, AWS lambdas, Azure cloud functions you must fight somehow against the environment (that's the reason frameworks like Zappa exist).
If you want/need to use compute instances, you can use KNative in your K8s cluster and enjoy a "similar" scaling experience.
For GCP cloud functions, AWS lambdas, Azure cloud functions you must fight somehow against the environment (that's the reason frameworks like Zappa exist).
If you want/need to use compute instances, you can use KNative in your K8s cluster and enjoy a "similar" scaling experience.
I’ve seen ory hydra being fir in a lambda function and as awkward as it looks to be, it works reasonably well.
Thanks, good to know it’s a common technique.
Thanks, good to know it’s a common technique.
I think the specific issue here is the 250 connection limit, which seems to be really under utilizing the instance in something as basic as chat.
where can you find servers like that for that price?
servershop24.de; not affiliated, just a happy client.
And as hosting
Hetzner.de
Also happy client im easily handling hundred thousand connections for less than 80€/mo
Also happy client im easily handling hundred thousand connections for less than 80€/mo
Love to read more about these stories rather than the same crazy cloud costs that get normalized all the time. Just by looking at Hetzner and others, I can see this kind of thing would be extremely doable but concrete stories of those doing it would be something I think a lot of people would enjoy reading about trade offs etc.
Serverless is for uses cases with unpredictable load.
Bare metal is cheaper, but you're paying 100% of the time for your peak traffic. If you only have a couple of peaks with 250k concurrent connections, with serverless you'd pay much less than €1500 but still be able to satisfy that demand.
Bare metal is cheaper, but you're paying 100% of the time for your peak traffic. If you only have a couple of peaks with 250k concurrent connections, with serverless you'd pay much less than €1500 but still be able to satisfy that demand.
Taking steps a bit further, it would be interesting to know what variation saves what money. Eg if your load is 80% of the time between 80-100k connections but the 20% at 250k, you save $500 a month using serverless or vice versa.
Is it as simple as $60k/number of connections? If so the cross over point would be just 6250 connections as your base load. IMO, that $1500 server starts to look very appealing very quickly but I imagine it isn't as simple as that.
Is it as simple as $60k/number of connections? If so the cross over point would be just 6250 connections as your base load. IMO, that $1500 server starts to look very appealing very quickly but I imagine it isn't as simple as that.
This is useful for services that have short surges but very little traffic around the month. If you only need a surge for 1 hour per month, then it should cost around $100/month.
> Any kernel with a bit of tweaking will easily handle that number of connections.
I don't think tweaking a kernel and "easily" fit together. Unless you are kernel developer.
> Any kernel with a bit of tweaking will easily handle that number of connections.
I don't think tweaking a kernel and "easily" fit together. Unless you are kernel developer.
>> Any kernel with a bit of tweaking will easily handle that number of connections.
> I don't think tweaking a kernel and "easily" fit together. Unless you are kernel developer.
I think he means twiddling the kernel knobs in /proc to raise the limit of file descriptors. Last I checked, most distros default to 100k maxmimum file descriptors, with a different (10k) max per process.
Changing the max by 'tweaking' involved echoing a new number to a file in /proc.
> I don't think tweaking a kernel and "easily" fit together. Unless you are kernel developer.
I think he means twiddling the kernel knobs in /proc to raise the limit of file descriptors. Last I checked, most distros default to 100k maxmimum file descriptors, with a different (10k) max per process.
Changing the max by 'tweaking' involved echoing a new number to a file in /proc.
It's called a C1M problem. From 2013: https://www.metabrew.com/article/a-million-user-comet-applic.... It's pretty easy. Been working on a product doing this for 500k connections per machine (MQTT) back in 2014. EC2, m3.xlarge instances back then.
Adding to this, how many watts is this consuming?
epoll allows high concurrent client count, but how many requests per second and more importantly how many requests per watt!?
Also what is the response payload size?
I can do 1000 HTTP req/res (4K) / watt on my new atom 8-core server with my own HTTP app server passively cooled and tiny mini-itx size!
epoll allows high concurrent client count, but how many requests per second and more importantly how many requests per watt!?
Also what is the response payload size?
I can do 1000 HTTP req/res (4K) / watt on my new atom 8-core server with my own HTTP app server passively cooled and tiny mini-itx size!
I use Cloud Run for all my server rendered web apps and I _love_ it. That’s where the pricing model makes sense since as soon as the request is served I stop paying. The actual cash price is less than what I was paying to run on GKE and the total cost of ownership is far lower.
Web sockets are a handy thing to have access to but the app described is a pretty pathological case for the Cloud Run pricing model. I think people knocking the article on pricing alone are missing the point.
Web sockets are a handy thing to have access to but the app described is a pretty pathological case for the Cloud Run pricing model. I think people knocking the article on pricing alone are missing the point.
Granted, the pricing assumptions assume 100% utilization at all times and not taking advantage of scaling-to-zero, which is where Cloud Run excels.
Given that, it seems like WebSockets/persistent connections is a weird use case.
Given that, it seems like WebSockets/persistent connections is a weird use case.
It demonstrates the combination of concurrency and scale to 0 which I havent seen in other public clouds.
WhatsApp had 2 Million simultaneous chat TCP connections per server back in 2014...
Why do we now need thousands of instances to achieve the same?
Why do we now need thousands of instances to achieve the same?
Probably cause erlang scales extremely well as gateway and it doesn't take much hardware to handle 2 million live connections with it
(https://www.phoenixframework.org/blog/the-road-to-2-million-...)
From memory, they also had really big servers and did a lot of work tuning the OS. Pretty sure it was BSD initially. The suggested approach by the author is for people who don't have (1) the money to buy big servers or (2) the expertise to tune the OS appropriately. Not saying I agree with the approach, but I can see where it would be helpful.
This isn't a valid comparison. You don't know how many (and the hardware of) instances WhatsApp was running to support those 2M connections. And you're not comparing apples-to-apples: WhatsApp is not the same software that the OP is running.
Moreover, the 250K limit is an artificial limit imposed by Google for a generic cloud service. If you controlled _that_ software and tuned it for your specific application, you probably could squeeze an order of magnitude (or two or three) more connections out of it. Whether that's a worthwhile tradeoff is a function of whether the time required to rewrite the functionality of Cloud Run yourself (and run it, and maintain it) is lower than the cost of simply spinning up more boxes. If you're building a product and have the cash, it really doesn't matter whether you can run 250k connections or 2M connections if one could exist this week and suit your needs and the other could exist in 6-8 months when you build/scale/deploy it.
Moreover, the 250K limit is an artificial limit imposed by Google for a generic cloud service. If you controlled _that_ software and tuned it for your specific application, you probably could squeeze an order of magnitude (or two or three) more connections out of it. Whether that's a worthwhile tradeoff is a function of whether the time required to rewrite the functionality of Cloud Run yourself (and run it, and maintain it) is lower than the cost of simply spinning up more boxes. If you're building a product and have the cash, it really doesn't matter whether you can run 250k connections or 2M connections if one could exist this week and suit your needs and the other could exist in 6-8 months when you build/scale/deploy it.
To me it seems like it's directly comparable and the efficiency demonstrated here seems truly abysmal.
Being able to handle 250K connections versus sustaining 250K connections are two different things. If you have a chat application used by mostly north american users, your usage is going to vary wildly between day and night. Cloud Run allows you to scale this up and down with ~zero ops work.
Engineering for highly variable capacity and engineering for extreme scales are two entirely different problem spaces. If a managed platform like Cloud Run makes your business more efficient and you can save on compute costs, the "efficiency" that you're talking about is completely moot. Being able to handle bursty loads when you need to and being able to consistently handle large volumes of traffic are two separate and unrelated problem spaces.
Engineering for highly variable capacity and engineering for extreme scales are two entirely different problem spaces. If a managed platform like Cloud Run makes your business more efficient and you can save on compute costs, the "efficiency" that you're talking about is completely moot. Being able to handle bursty loads when you need to and being able to consistently handle large volumes of traffic are two separate and unrelated problem spaces.
OP here. You don’t, and that’s not what I wrote about here. You’re comparing bare metal
mainframe-like architecture to the cloud serverless paradigm. The “Pricing” section in the article might give an idea.
compare that to 400k connections on a single server using erlang early this Millenium. Wirth's law in action.
https://en.wikipedia.org/wiki/Wirth%27s_law
https://en.wikipedia.org/wiki/Wirth%27s_law
You wouldn't get 400k connections on Cloud Run using erlang, because it limits concurrent requests per container to 250.
but.. why does it do that?
Most software becomes unhappy if you send unlimited concurrent requests. It's better to have a hard limit that can be raised than to find out under fire what your software tolerates.
but can it be raised?
No idea. Probably, but it depends on whether that limit is hard coded or is a config flag that an account team can get changed.
$$$
This is a really neat demo, but there's a few limitations that are glaring issues for a chat server or most scenarios I can think of that I would want to use websockets. As many have mentioned, the 250 connection limit per instance is crazy. You can get 200x that many easily on a single 1 VCPU box.
The second problem is that really the demo has just kicked your scaling problem over to redis. The demo isn't doing anything actually interesting, redis is doing all the work. The reality is that one of the best parts of Cloud Run over something like serverless functions is that you can have state in your server. You don't need redis at all if you are doing the same demo in Kubernetes without Cloud Run, since it's pretty easy to use channels in golang to do the same thing. However, in order for you to really make that work, you need some way to at the very least route the traffic consistently so that people watching the same resource get routed to the same server, or to be able to have cross node communication.
So this is a great start, but a little work to go before many people would be able to switch over to something like this due.
The second problem is that really the demo has just kicked your scaling problem over to redis. The demo isn't doing anything actually interesting, redis is doing all the work. The reality is that one of the best parts of Cloud Run over something like serverless functions is that you can have state in your server. You don't need redis at all if you are doing the same demo in Kubernetes without Cloud Run, since it's pretty easy to use channels in golang to do the same thing. However, in order for you to really make that work, you need some way to at the very least route the traffic consistently so that people watching the same resource get routed to the same server, or to be able to have cross node communication.
So this is a great start, but a little work to go before many people would be able to switch over to something like this due.
It's a fun example in how easy it is to setup, and I don't want to poke too hard on a hypotheticals, but I don't like two things:
1) Egress is hand-waved, when in reality the free GBs would basically exhaust in a month on just pongs and reconnection requests, so doing nothing at all. That amount of people just sending single emojis as messages for just one day (to play along "short marketing event") is already matching monthly napkin estimate stated.
2) Isn't there's a good chance Memorystore wouldn't be able to handle this at fairly charitable load interpretations under that many CCUs? 1‰ of those 250k CCUs sending a message every second means Redis needs to publish 250k messages as well.
Again though, I'm aware it's only a hypothetical scenario demonstration, and it really does look cute in how it's simple to deploy. Fun stuff.
1) Egress is hand-waved, when in reality the free GBs would basically exhaust in a month on just pongs and reconnection requests, so doing nothing at all. That amount of people just sending single emojis as messages for just one day (to play along "short marketing event") is already matching monthly napkin estimate stated.
2) Isn't there's a good chance Memorystore wouldn't be able to handle this at fairly charitable load interpretations under that many CCUs? 1‰ of those 250k CCUs sending a message every second means Redis needs to publish 250k messages as well.
Again though, I'm aware it's only a hypothetical scenario demonstration, and it really does look cute in how it's simple to deploy. Fun stuff.
OP here.
1) You’re right, 1 GiB free tier will exhaust fast. Beyond that standard GCP-wide egress rates apply. So if you were using VMs to serve this chatroom, you'd be paying the same https://cloud.google.com/network-tiers/pricing#premium-prici...
2) I think you’re missing out the part that in this setup only 1000 instances connect to Redis (using 1000 connections). We don’t initiate a per-user connection from the backend to Redis. Once a message is pushed to an app instance, the sample app pushes it to all its connected users.
1) You’re right, 1 GiB free tier will exhaust fast. Beyond that standard GCP-wide egress rates apply. So if you were using VMs to serve this chatroom, you'd be paying the same https://cloud.google.com/network-tiers/pricing#premium-prici...
2) I think you’re missing out the part that in this setup only 1000 instances connect to Redis (using 1000 connections). We don’t initiate a per-user connection from the backend to Redis. Once a message is pushed to an app instance, the sample app pushes it to all its connected users.
> If you deploy this app with 128MB RAM and 1 vCPU today, it will cost (0.00002400 + (0.00000250/8)) * 60 * 60 = $0.0875 per hour per instance. 1 This means if you have 1,000 instances actively running and serving 250K clients, it will cost $87/hour, which is $62.6K/month.
> Any Cloud Run service, by default, can scale up to 1,000 instances. (However, by opening a support ticket, you can get this number elevated.) This means we can support 250,000 clients simultaneously without having to worry about infrastructure and scaling!
Aren't these statements at odds with one another?
> Any Cloud Run service, by default, can scale up to 1,000 instances. (However, by opening a support ticket, you can get this number elevated.) This means we can support 250,000 clients simultaneously without having to worry about infrastructure and scaling!
Aren't these statements at odds with one another?
I don't see why, maybe you can elaborate?
Scaling limits put in place by cloud providers are there in part to make sure their customers don’t accidentally incur runaway costs due to a configuration mistake. It’s not uncommon that a cloud provider would cover the costs if a customer make such an error (first time, ask nicely, etc). They are covering their own bases from having to do that on a massive scale.
Because that's a lot of money.
Is it the "without having to worry about infrastructure and scaling" that you disagree?
I don't personally find the two statements at odds. If you have 250k connections 24/24 7/7 for a whole month, that should mean your business is generating more than enough money to cover the cost.
Or, at the very least, that the losses by not being able to scale are higher than the cost of the infrastructure.
If this is false and you'll be bankrupt by those 250k connections, you should obviously not let your instances to scale that much.
I don't personally find the two statements at odds. If you have 250k connections 24/24 7/7 for a whole month, that should mean your business is generating more than enough money to cover the cost.
Or, at the very least, that the losses by not being able to scale are higher than the cost of the infrastructure.
If this is false and you'll be bankrupt by those 250k connections, you should obviously not let your instances to scale that much.
It's still a lot of money even if it won't bankrupt the business.
or you can pay $80/month at a co-lo facility and host your own big box and do millions of simultaneous connections...
This estimate misses the entire benefit of Cloud Run which is metered compute billing. Nobody in their right mind would run a serverless stack if they had 250K connections 24/7. It’s specifically documented that it isn’t for long-running processes, just for event-based apps. More realistically you’d get 250k requests, serve them with a total of ~5 container instances, and then the billing stops 5 seconds later. It’s cheap. It’s not for websocket apps. It’s perfect for MVPs and spiky traffic. It’s an easy path to move the hosting to K8s, App Engine, or normal VMs (GCP VMs can just run containers with minimal config), which are all far better choices if traffic is predictable.
I think this is a far more valuable model than Function-as-a-service, where the runtime environment is strictly defined. Since “it’s just containers” you could write a web server in brainfvck if you felt like it, write a tiny dockerfile, then ‘git push origin/main’ and Cloud Run gives you a running endpoint. Okay, there is some devops work to set up that pipeline, but that is mostly boilerplate.
It’s Heroku but it scales down to zero (ie free) and up to 1000+ instances of your server in ms.
I think this is a far more valuable model than Function-as-a-service, where the runtime environment is strictly defined. Since “it’s just containers” you could write a web server in brainfvck if you felt like it, write a tiny dockerfile, then ‘git push origin/main’ and Cloud Run gives you a running endpoint. Okay, there is some devops work to set up that pipeline, but that is mostly boilerplate.
It’s Heroku but it scales down to zero (ie free) and up to 1000+ instances of your server in ms.
1000 instances with 250 concurrent connections each is 250k, but I fear I may be missing your point
I think the point is that one sentence says you don't have to worry about scaling and the next sentence shows scaling is very costly implying you do have to worry about it.
I don't really get what's so special about this. But I am always surprised by how extremely scalable Redis is, scaling the stateless chat servers is trivial, scaling state is hard.
I feel exactly same, other than ease of scaling to 1000 instances I don't see anything special, everything is offloaded to Redis. Maybe example use case should have been different to showcase scaling ease
[deleted]
I might be missing the point of this article? Is it very difficult when using Cloud Run to figure out how to achieve a high number of concurrent connections? Is that what the article is about? Or like what is otherwise noteworthy here?
Use a dedicated server and IRC. 250k users isn't that scary anymore.
I remember that in 2012, the Rizon IRC network maxed out at 80,000 concurrent users on an AMD Bulldozer CPU.
I remember that in 2012, the Rizon IRC network maxed out at 80,000 concurrent users on an AMD Bulldozer CPU.
Exactly. Numbers like these are actually quite pathetic when we look back 10 and even 20 years to IRC.
I'm trying to figure out a good architecture for a chat back end so I found this interesting. Anyone have some good pointers to other such articles?
Discord's blog has some interesting articles.
https://blog.discord.com/engineering-posts/home
https://blog.discord.com/engineering-posts/home
Why you wouldn't use AWS http events for this. Especially because connections are decoupled from processing
A little off-topic, but AWS has a service (actually 2) specifically for managing websocket communications.
API Gateway supports an unlimited (you need to ask for an increase from the 500 new connections per second rate) number of connections and is only around $0.25 per million minutes + $1 per million messages.
So just having 250,000 connections open would cost only around $270 per month (and scales up and down as you please).
API Gateway supports an unlimited (you need to ask for an increase from the 500 new connections per second rate) number of connections and is only around $0.25 per million minutes + $1 per million messages.
So just having 250,000 connections open would cost only around $270 per month (and scales up and down as you please).
That’s only part of the API gateway pricing, but it is still much much cheaper.
I think Ahmet's arithmetic is incorrect. The limits that he says lead to the 250k figure are:
1. 250 connections per container, and
2. 1,000 containers.
However the 250 concurrency limit does not refer to connections, it refers to requests. 250 concurrent requests can actually represent thousands of clients, depending on their think time.
1. 250 connections per container, and
2. 1,000 containers.
However the 250 concurrency limit does not refer to connections, it refers to requests. 250 concurrent requests can actually represent thousands of clients, depending on their think time.
I think it's correct...
https://cloud.google.com/run/docs/triggering/websockets says:
"Since Cloud Run supports concurrent connections (up to 250 per container)..." "WebSockets requests are treated as long-running HTTP requests in Cloud Run."
https://cloud.google.com/run/docs/triggering/websockets says:
"Since Cloud Run supports concurrent connections (up to 250 per container)..." "WebSockets requests are treated as long-running HTTP requests in Cloud Run."
I stand corrected.
If you are a GCP or AWS or bare metal expert that can set this thing up in their sleep, that's great but the majority of people can really benefit from a PaaS like GCR.
Because Cloud Run uses vanilla Docker containers, once you have validated the idea you can move to GKE or VMs or a server under your desk or whatever. And if it never takes off, that's fine too because you didn't spend a ton of time investing in making it work.
Ahmet if you are reading this (hi) it would be REALLY cool to see something like using GKE for the base load and dynamically bursting to GCR to fill in the gaps. Not sure if this possible with GCLB today, but would be super cool.
(Disclaimer: I used to work on this team at Google.)