Ask HN: How do you scale your service worldwide in 2021?
5 comments
Hi,
I faced this situation after a Show HN (https://news.ycombinator.com/item?id=20105567), all I had to do is simply to add a CDN (Cloudflare or AWS Cloudfront) in front of the servers in order to serve the static assets and pages from the CDN, and the API requests by the servers.
If you are not familiar with CDNs yet, learn how works the Cache-Control HTTP header (https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Ca...) so you can send a different header for API responses than for static assets responses. Then learn how to purge the CDN after each release.
I would recommend to start with Cloudflare as it's cheaper, and easier to use.
A CDN with correct Cache-Control headers and Heroku Autoscaling should bring you very far in term of traffic you can handle.
If you are not familiar with CDNs yet, learn how works the Cache-Control HTTP header (https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Ca...) so you can send a different header for API responses than for static assets responses. Then learn how to purge the CDN after each release.
I would recommend to start with Cloudflare as it's cheaper, and easier to use.
A CDN with correct Cache-Control headers and Heroku Autoscaling should bring you very far in term of traffic you can handle.
This does help a decent bit but not for dynamic content. I'm facing the same issue with my job board [1]. I'm trying to do this without using anything like AWS; mostly because I haven't used it and seems it's a bit complicated for pricing.
I would cache most of the common database queries, then monitor which is pulling down your system. A load balancer usually does the trick.
[1] - https://arbeitnow.com
I would cache most of the common database queries, then monitor which is pulling down your system. A load balancer usually does the trick.
[1] - https://arbeitnow.com
If you like GraphQL and don't mind managed services:
- Fauna (http://fauna.com) or Hasura (https://hasura.io) for the backend
- Vercel (http://vercel.com) or Netlify (https://www.netlify.com) for the frontend and functions
- Fauna (http://fauna.com) or Hasura (https://hasura.io) for the backend
- Vercel (http://vercel.com) or Netlify (https://www.netlify.com) for the frontend and functions
Fly.io would be my goto stack right now (especially for read heavy workloads). Edge compute in conjunction with static assets being served by a traditional cdn will go a long way.
I haven’t used Heroku, but I’ll assume you’re running virtual hosts in the cloud as opposed to something like Lambda.
I don’t have enough information from your post to give a concrete answer, but here are some things I might consider in your shoes.
How is your system struggling? Do you have metrics that measure your cpu, memory, or disk usage on your application servers? Can you see a pattern of how they’ve regressed over time, in a time series graph? Did you make a change that caused this? Are you certain whatever scaling issue you’re trying to solve is because of real customer request growth and not some regression?
How many requests does your service accept right now, per second?
What’s the latency to process each request? Can we figure out what all contributes to this? What are the cheap things and expensive things, or is it all uniform? Have you profiled?
You could measure this at the application level as the end time (before you send a response down) minus the start time (when you received the request).
Is your Redis instance on the same host? Is it a distributed cache? And how is load on whichever host it lives on? Redis is pretty dang fast. Is it receiving too many connections? Is it actually not doing much and the issue is elsewhere?
What’s the load on Postgres? Is your data/table schemas optimized based on your query patterns? Do you have the right indices set up?
Where do your customers live? Do you track latency from the webpage or mobile client? Is it slower for customers in some parts of the world than others? If so, have you thought about scaling out to other regions? If the component contributing latency is on a specific part of your webpage or front end, does it even see many eyeballs? Should you do away with that functionality entirely or is it worth the ROI?
Are your Postgres servers writing in a distributed manner (sharding) or is one just a replica? Is one server getting more traffic than the other and so load isn’t even balanced?
Is there content on your page that is queried from Postgres but could be moved to a cache like Redis or read from a static file?
What response patterns does your application have? When a customer signs up, can you just show them a thank you screen and put any expensive processing into some queue? Once it succeeds or fails, send them an email.
We obviously want to avoid over complicating things. Is there a specific business function in your app that is really expensive to do on the server at your current scale? Can we move that business function to a separate server or make it it’s own web service?
How does load balancing work on your customer facing page or service? Is it optimally distributing requests? How much of your traffic is real customers and not someone spamming your API? Are you measuring this?
I don’t have enough information from your post to give a concrete answer, but here are some things I might consider in your shoes.
How is your system struggling? Do you have metrics that measure your cpu, memory, or disk usage on your application servers? Can you see a pattern of how they’ve regressed over time, in a time series graph? Did you make a change that caused this? Are you certain whatever scaling issue you’re trying to solve is because of real customer request growth and not some regression?
How many requests does your service accept right now, per second?
What’s the latency to process each request? Can we figure out what all contributes to this? What are the cheap things and expensive things, or is it all uniform? Have you profiled?
You could measure this at the application level as the end time (before you send a response down) minus the start time (when you received the request).
Is your Redis instance on the same host? Is it a distributed cache? And how is load on whichever host it lives on? Redis is pretty dang fast. Is it receiving too many connections? Is it actually not doing much and the issue is elsewhere?
What’s the load on Postgres? Is your data/table schemas optimized based on your query patterns? Do you have the right indices set up?
Where do your customers live? Do you track latency from the webpage or mobile client? Is it slower for customers in some parts of the world than others? If so, have you thought about scaling out to other regions? If the component contributing latency is on a specific part of your webpage or front end, does it even see many eyeballs? Should you do away with that functionality entirely or is it worth the ROI?
Are your Postgres servers writing in a distributed manner (sharding) or is one just a replica? Is one server getting more traffic than the other and so load isn’t even balanced?
Is there content on your page that is queried from Postgres but could be moved to a cache like Redis or read from a static file?
What response patterns does your application have? When a customer signs up, can you just show them a thank you screen and put any expensive processing into some queue? Once it succeeds or fails, send them an email.
We obviously want to avoid over complicating things. Is there a specific business function in your app that is really expensive to do on the server at your current scale? Can we move that business function to a separate server or make it it’s own web service?
How does load balancing work on your customer facing page or service? Is it optimally distributing requests? How much of your traffic is real customers and not someone spamming your API? Are you measuring this?
Built a service in the form of an app during the course of the pandemic for people to watch videos and receive news.
However since the pandemic, I had a huge surge of people signing up and straining my backend, leaving me frantically trying to keep the servers up and respond to customers at the same time.
It must have been hard 10 or 20 years ago to scale your backend/service to reach as many people as possible worldwide than it is today, but I still struggled with this.
I'm curious how would one do this when faced with a TON of customers as I had to face.
If you're curious of my stack: it's all on Heroku, Redis, Node & 2 Postgres servers.
Would also like to know what you would have done better as well.