Ask HN: Do you use Nginx in production? or have you switched
22 comments
We use HAProxy at the load balancers, and HAProxy/Varnish/Apache on the individual app server VMs.
This separates concerns completely:
- HAProxy knows about and manages the TLS, balancing, client routing etc;
- Varnish knows about and manages response caching and ESI processing (and often a combination of both);
- Apache knows about and runs the various backend services (a php web app, a couple of ruby third party tools, etc)
Nginx has some significant downsides to what we currently use, unless we opt for the paid version which best I can tell is ~$1K/instance/month. These aren't hypothetical differences these are features we actually use:
- no sync for load balancing data (sticky peer data, rate limit data, etc): HAProxy supports this out of the box;
- no active health checks: HAProxy supports this out of the box;
- no API for purging cache: Varnish supports this out of the box.
- no ESI support: Varnish supports this out of the box. Best I can tell even the paid version of nginx doesn't support this.
This separates concerns completely:
- HAProxy knows about and manages the TLS, balancing, client routing etc;
- Varnish knows about and manages response caching and ESI processing (and often a combination of both);
- Apache knows about and runs the various backend services (a php web app, a couple of ruby third party tools, etc)
Nginx has some significant downsides to what we currently use, unless we opt for the paid version which best I can tell is ~$1K/instance/month. These aren't hypothetical differences these are features we actually use:
- no sync for load balancing data (sticky peer data, rate limit data, etc): HAProxy supports this out of the box;
- no active health checks: HAProxy supports this out of the box;
- no API for purging cache: Varnish supports this out of the box.
- no ESI support: Varnish supports this out of the box. Best I can tell even the paid version of nginx doesn't support this.
Sounds like you're pushing the limits of each product. Looks cool to see. Do you think it is possible to simplify the stack or would you loose too much?
And another question? Do you use varnish on edge? In case, how do you do this? I normally just use bunnycdn for caching.
And another question? Do you use varnish on edge? In case, how do you do this? I normally just use bunnycdn for caching.
I'm not an expert, but I don't think realistically we're anywhere close to the limits of what they can do.
Our "edge" is CF right now, but that's likely to change (coincidentally to bunny) because our traffic model doesn't really fit their offering very well.
This actually goes to answer your first question: we've progressively moved more things into our own origin stack because what CF offers us doesn't work out (and in some cases probably wouldn't be easily solved with nginx either).
As an example: CF and Nginx both have basic rate limiting, but AFAIK, neither can track two separate rates out of the box, if at all. We use a relatively high req/period limit, and then a much lower threshold error/period limit which counts any requests that result in a client-caused error (i.e. 4xx errors).
I'm pretty certain we would lose flexibility if we "simplified" by either offloading some aspects to a service, or even if we tried to consolidate several parts into a kitchen-sink tool like nginx.
We mostly use edge (i.e. CDN) caching the same way we expect a browser cache to work (but across users): stuff that we absolutely know isn't changing (i.e. we reference asset (img, js, css) content using URLs based on a content hash). We use our internal Varnish cache for application content, combined with a job in our queue that can purge that content (selectively) when required. Could we use a commercial CDN for that part? Sure in theory, but it's just another thing we won't have control over, and have to develop specific to the CDN we're using at the time, subject to the rules of their system - and it still wouldn't necessarily to ESI.
Our "edge" is CF right now, but that's likely to change (coincidentally to bunny) because our traffic model doesn't really fit their offering very well.
This actually goes to answer your first question: we've progressively moved more things into our own origin stack because what CF offers us doesn't work out (and in some cases probably wouldn't be easily solved with nginx either).
As an example: CF and Nginx both have basic rate limiting, but AFAIK, neither can track two separate rates out of the box, if at all. We use a relatively high req/period limit, and then a much lower threshold error/period limit which counts any requests that result in a client-caused error (i.e. 4xx errors).
I'm pretty certain we would lose flexibility if we "simplified" by either offloading some aspects to a service, or even if we tried to consolidate several parts into a kitchen-sink tool like nginx.
We mostly use edge (i.e. CDN) caching the same way we expect a browser cache to work (but across users): stuff that we absolutely know isn't changing (i.e. we reference asset (img, js, css) content using URLs based on a content hash). We use our internal Varnish cache for application content, combined with a job in our queue that can purge that content (selectively) when required. Could we use a commercial CDN for that part? Sure in theory, but it's just another thing we won't have control over, and have to develop specific to the CDN we're using at the time, subject to the rules of their system - and it still wouldn't necessarily to ESI.
That's so cool! I actually made a YouTube video on why I don't use CF anymore. Shameless plug here :D
https://youtu.be/TWtGwW_e56w
I really love your idea of doing content hash for (permanent?) cache. I'll definitely do that. Right now we use a semi-permanent cache, but sometimes we screw up and it really bites us. If it is hashed you know for sure it is permanent. It reminds of IPFS, but for your little corner of the internet :D
https://youtu.be/TWtGwW_e56w
I really love your idea of doing content hash for (permanent?) cache. I'll definitely do that. Right now we use a semi-permanent cache, but sometimes we screw up and it really bites us. If it is hashed you know for sure it is permanent. It reminds of IPFS, but for your little corner of the internet :D
We started using content hash addressing with our assets build system (i.e. combining CSS & JS into a single URL), so that we'd get maximum cache life without needing to artificially manage cache busting query strings.
The more substantial use now though, is for the "content" of the site (purchasable ebooks/etc and associated artwork) which is constantly growing and occasionally old content gets updated with newer editions/revisions. We use content hashing to actually store the files (free de-duplication) and obviously it then gives us the same caching benefits.
The more substantial use now though, is for the "content" of the site (purchasable ebooks/etc and associated artwork) which is constantly growing and occasionally old content gets updated with newer editions/revisions. We use content hashing to actually store the files (free de-duplication) and obviously it then gives us the same caching benefits.
This is so cool Stephen. I'd love to hear more. This is definitely something I could use in one of my clients projects. You wanna schedule maybe a meeting next week?
https://martinbaun.com/book/
https://martinbaun.com/book/
I'd be happy to talk about it more, but I'll be out of the office all next week on a short break.
Can you drop me an email (it's in my profile) with a rough idea of what you want to discuss, and we could follow up the following week ( with a call/chat?
Can you drop me an email (it's in my profile) with a rough idea of what you want to discuss, and we could follow up the following week ( with a call/chat?
Awesome Steven, I'll msg you once you're back. Happy break
CDNs, ingress, etc... haven't had to use a web proxy directly for years. At the end of the day funnily enough it's still nginx or similar behind the scenes.
Caddy for local development. Less config and setup.
Caddy for local development. Less config and setup.
You use Caddy for local development? How come, I have never used it for local dev
To make it as close to production without all the hassle.
- Autoprovison SSL
- Serve static files without another web server
- Rewrites, redirects, map HMR, websocket ports etc
Sure nginx works too but Caddy is definitely less work and less hassle
- Autoprovison SSL
- Serve static files without another web server
- Rewrites, redirects, map HMR, websocket ports etc
Sure nginx works too but Caddy is definitely less work and less hassle
Cool, but how do you provision SSL locally if you're behind a NAT?
For the serving static files/rewrites I usually try to do it in my own code so I don't have to deal with extra configs. Of course, this is a bit slow but locally it's fine :)
For the serving static files/rewrites I usually try to do it in my own code so I don't have to deal with extra configs. Of course, this is a bit slow but locally it's fine :)
> Cool, but how do you provision SSL locally if you're behind a NAT?
Remote / work from home without crazy VPN requirements. Some workplaces / offices I've been to also don't have them.
> I usually try to do it in my own code
I gets a lot harder when you have multiple services and/or if you mean you've just written your own reverse proxy then I'd prefer learning the configs.
Remote / work from home without crazy VPN requirements. Some workplaces / offices I've been to also don't have them.
> I usually try to do it in my own code
I gets a lot harder when you have multiple services and/or if you mean you've just written your own reverse proxy then I'd prefer learning the configs.
Writing my own, ah no :D just setting up my local env to serve static files. Flask, Node, Go, DJango supports it out of the box. Though, they're extremely slow in most cases. Then in production I just override the url with Caddyserver. Pretty neat :)
It works, it's easy to configure, it's fast, and it's been solid for the 10 years I've used it.
True, but I did get annoyed trying to get SSL certificates to autorenew, especially when you have multiple server srunning
I use Nginx:
I mean, the autorenew bots need more priviledges, such as:
2. https://datatracker.ietf.org/doc/html/rfc8555#section-8.3
3. https://letsencrypt.org/docs/faq/#what-ip-addresses-does-let...
- as a layer on top the app servers for not having to expose Node.js, and loadbalancing app servers,
- brotli_static,
- serving avif conditionally[1]
- anonymizing IPs in logs
- injecting the caching headers
- injecting the CSP header
- SSL Offloading
I mean, the autorenew bots need more priviledges, such as:
- HTTP challenges need to be via HTTP (not HTTPS) [2],
- HTTP challenges need write permissions on a servable directory,
- DNS or HTTP challenges would need a program on a live server,
- need ‘pass out’ firewall exceptions without IP scope. "We don’t publish a list of IP addresses we use to validate… Let’s Encrypt" [3]
2. https://datatracker.ietf.org/doc/html/rfc8555#section-8.3
3. https://letsencrypt.org/docs/faq/#what-ip-addresses-does-let...
Interesting, what do you for SSL instead?
I use acme.sh with DNS challenges on an external machine. Then I push up (rsync) the certs and reload Nginx.
Here’s a blog post https://blog.uxtly.com/isolated-tls-certificate-creation
Here’s a blog post https://blog.uxtly.com/isolated-tls-certificate-creation
Thanks eFortis? I guess this mainly for security and separation of concerns?
Yes, the certificate renewal and the server are more hardened this way.
Cool! Do you do other things to hardend the servers like Knockerd?
I used NGINX for many years, but around 1.5 year ago I switched to Caddyserver as the SSL certificates are just so nice getting them autohandled. I noticed a slight performance decrease, but for my kinda services it is not so important.
But I am curious, what do you use? and more importantly why?