Fair queuing is an interesting and complementary approach that I have not considered, thanks for the suggestion.
I usually work with "internal" systems that don't always have an obvious "user" identifier, but it would be fairly easy to add one of some sort (e.g. an application id or similar). However, for that to be useful, there still has to be some sort of limit. In your case, the limit seems to be 1 concurrent request per user, up to a maximum of 100. In the case of some internal system, like say a metrics aggregator, what is the correct limit? I'm not sure, and I think it would require tuning.
That said, given a reasonable limit, this would definitely ensure that the "misbehaving" application (e.g. the one flooding the metrics aggregator with packets) is the one that gets punished, rather than everyone else. I'll have to think about this more. Thanks!
I agree that a universal solution sounds difficult. However, it seems like it should be possible to have something that works "well enough" for some (broad) class of servers. My example is TCP, which does a "good enough" job of controlling the flow of packets over a wide range of networks.
This is a good point, although I don't think I said "connections", I said "requests". The definition of requests is going to vary significantly. Yes, very slow clients are yet another problem that extremely robust systems don't have to handle.
In my personal case, nearly all of my work has always been on "internal" services that are being used inside a single organization inside a data center, where this is rarely a problem. This is much more of an issue for services that are accessed over the public Internet, where you need to handle malicious attackers as well as users that have horrible connections.
In any case, I believe what I wrote still applies, but you do need to set appropriate timeouts (yet another parameter to tune; yuck!)
I use App Engine's static file serving to serve my site. It has a 1 GB/day bandwidth limit. It turns out Hacker News still moves a lot of traffic, so this is the first time I've crossed it! Oops. Billing is now enabled on this project.
Thanks for the link, the abstract seems relevant. I agree: a control algorithm seems like it should work. The challenge is figuring out the metrics and parameters to make something that works "well enough" for most applications, like TCP does for networks. I would really love for someone to figure that out, so we don't run into this very often.
You make an excellent point: probability says a 4-byte CRC32C provides much weaker guarantees as the length of the message gets longer. These CRCs are typically optimized for pretty short messages, and that is what I had in mind when I wrote that article (e.g. the kind of messages that might get exchanged as part of an online serving system).
This is possible, but I find these restrictions to be hard to follow. As soon as you need to call a function, you now need to audit that function to determine that it only calls other async signal safe functions. When you come back to the code to fix a bug six months later, you need to remember these restrictions.
Do this when you must, but it is easy to screw up.
Thanks for pointing this out! Interesting that the code states that part of the concern is increasing memory usage. I don't quite understand why it would "duplicate the memory usage" since fork uses copy-on-write?
Anyway, I didn't explain it well, but this was what I was trying to get at with "fork a worker at the beginning of your program, before there can be other threads."
Fair point. My opinion is that in today's age of multi-core CPUs, shared memory concurrency is extremely useful for making efficient use of computing resources. As a result, I find threads to be unavoidable in most large systems I've dealt with recently.
You are completely correct! The thing that surprised me is how difficult that can be. For example, on Mac OS X if you want to read the system proxy settings, you magically get threads added to your program that you can't control.
Part of my motivation for writing these things is because after I've wasted so much time, I'd love to help others not do the same. Hopefully the article shows up when you search for the right error message. Its also so I remember what the heck was happening.
Re: multiprocessing in 3.4: I think it is unfortunate that due to backwards compatibility, the "forkserver" mode can't become the default on Unix, since it would help avoid this particular issue.
You are completely correct: it is in fact possible to use fork if you can carefully control the state of threads in your program. My point is that can be difficult in large software projects, particularly when random APIs like "get the system proxy settings" use threads without you having any ability to control them.
I've been trying to get Docker to include this workaround when it creates containers to ensure people don't run into it, but this has not gotten any attention: https://github.com/docker/docker/issues/18776
Yes, with some caveats: Lots of configurations that use veths use NAT to share the IP address of the host. For example, this is Docker's default configuration. In this case, the host kernel checks the TCP checksum no matter what, so this issue doesn't apply.
This problem only happens when the packets are routed from the host to the container. This happens in Kubernetes, which assigns each container its own IP address, Docker's IPv6 configuration (same), and Mesos, which using Linux Traffic Control rules to share a range of ports with each container.
Thanks for the details! This is basically what Vijay and I were guessing by the fact that the MTU is something less than 1500 on Google Compute Engine.
The demonstration that Vijay added to the post was done on Google Container Engine, using Kubernetes. The packets were sent corrupted using netem. We tested a few configurations and were unable to get corrupt packets to be delivered to a Google Container Engine instance, so I agree with your assessment. Most importantly: it appears that the Container Engine TCP load balancer drops corrupt packets from the public Internet.
However: If someone is using some weird routing or VPN configuration, it might be possible (but this seems unlikely). Notably: I seem to recall that if you send corrupt packets to a Compute Engine instance, they are received corrupted (through the Compute Engine NAT). So if you used your own custom routing to get packets to a Google Container Engine application, this might apply. But again, you would have to really try to have this happen :)
I usually work with "internal" systems that don't always have an obvious "user" identifier, but it would be fairly easy to add one of some sort (e.g. an application id or similar). However, for that to be useful, there still has to be some sort of limit. In your case, the limit seems to be 1 concurrent request per user, up to a maximum of 100. In the case of some internal system, like say a metrics aggregator, what is the correct limit? I'm not sure, and I think it would require tuning.
That said, given a reasonable limit, this would definitely ensure that the "misbehaving" application (e.g. the one flooding the metrics aggregator with packets) is the one that gets punished, rather than everyone else. I'll have to think about this more. Thanks!