talawahtech

no profile record

Submissions

Extreme HTTP Performance Tuning

976 pointsby talawahtechhá 5 anos145 comments

comments

há 9 mesesdiscuss

Thanks for posting this. I just made a donation to the PSF.

ano passadodiscuss

Amazon Benchmarking | L5 SDEs | Seattle, WA | Full-time | https://www.amazon.jobs/en/jobs/2933471/software-development...

Help make AWS faster, more efficient, and cheaper for our customers!

Our team is hiring multiple SDEs with experience in distributed system performance and strong problem-solving/detective skills.

Feel free to reach out to the hiring manager on LinkedIn with any questions: https://www.linkedin.com/feed/update/urn:li:activity:7324200...

há 5 anosdiscuss

1.2M req/s means 2.4M (send/recv) syscalls per second. I definitely think io_uring will make a difference. Just not sure if it will be 5% or 25%.

há 5 anosdiscuss

The answer to that question is not quite as straight-forward as you might think. In many ways, this experiment/post is about figuring out the answer to the question of "what is the best the hardware can do".

I originally started running these tests using the c5.xlarge (not c5n.xlarge) instance type, which is capable of a maximum 1M packets per second. That is an artificial limit set by AWS at the network hardware level. Now mind you, it is not an arbitrary limit, I am sure they used several factors to decide what limits make the most sense based on the instance size, customer use cases, and overall network health. If I had to hazard a guess I would say that 99% of AWS customers don't even begin to approach that limit, and those that do are probably doing high speed routing and/or using UDP.

Virtually no-one would have been hitting 1M req/s with 4 vCPUs doing synchronous HTTP request/response over TCP. Those that did would have been using a kernel bypass solution like DPDK. So this blog post is actually about trying to find "the limit", which is in quotes because it is qualified with multiple conditions: (1) TCP (2) request/response (3) Standard kernel TCP/IP stack.

While working on the post, I actively tried to find a network performance testing tool that would let me determine the upper limit for this TCP request/response use case. I looked at netperf, sockperf and uperf (iPerf doesn't do req/resp). For the TCP request/response case they were *all slower* than wrk+libreactor. So it was up to me to find the limit.

When I realized that I might hit the 1M req/s limit I switched to the c5n.xlarge whose hardware limit is 1.8M pps. Again, this is just a limit set by AWS.

Future tests using a Graviton2 instance + io_uring + recompiling the kernel using profile-guided optimizations might allow us to push past the 1.8M pps limit. Future instances from AWS may just raise the pps limit again...

Either way, it should be fun to find out.

há 5 anosdiscuss

Hard to say exactly. I have been working on this in my spare time, but pretty consistently since covid-19 started. A lot of this was new to me, so it wasn't all as straight-forward as it seems in the blog.

As a ballpark I would say I invested hundreds of hours in this experiment. Lots of sidetracks and dead ends along the way, but also an amazing learning experience.

há 5 anosdiscuss

The MTU is 9001 within the VPC, but the packets are less than 250 bytes so the MTU doesn't really come into play.

This test is more about packets/s than bytes/s.

há 5 anosdiscuss

I didn't make any coordinated omission changes (I really didn't make many changes in general), so twrk does what wrk does. It attempts to correct it after the fact by looking for requests that took twice as long as average and doing some backfilling[1].

I am no expert where coordinated omission is concerned, but my understanding is that it is most problematic in scenarios where your p90+ latency is high. Looking at the results for the 1.2M req/s test you have the following latencies:

  p50     203.00us
  p90     236.00us
  p99     265.00us
  p99.99  317.00us
  pMAX    626.00us

If you were to apply wrk's coordinated omission hack to these result, the backfilling only starts for requests that took longer than p50 x 2 (roughly) = 406us, which is probably somewhere between p99.999 and pMAX; a very, very small percentage.

I am not claiming that wrk's hack is "correct", just that I don't think coordinated omission is a major concern for *this specific workload/environment*

1. https://github.com/wg/wrk/blob/a211dd5a7050b1f9e8a9870b95513...

há 5 anosdiscuss

That is an area I am curious about as well, especially if you throw io_uring into the mix. I think most kernel bypass solutions get some of their gains by just forcing you to use the same strategies covered in the perfect locality section. It doesn't all just come from the "bypass" part.

Even if isn't quite as fast as DPDK and co, it might be close enough for some people to start opting to stick with the tried and true kernel stack instead of the more exotic alternatives.

há 5 anosdiscuss

Thank you for sharing all your amazing tools and resources brendangregg! I wouldn't have been able to do most of these optimizations without FlameGraph and bpftrace.

I actually did the same thing and hacked up the perl code to generate the my custom palette.map

Thanks for the tip re: the disconnected kernel stacks. They actually kinda started to grow on me for this experiment, especially since most of the work was on the kernel side.

há 5 anosdiscuss

Recompiling the kernel using profile guided optimizations[1] is yet another thing on the (never-ending) to-do list.

1. https://lwn.net/Articles/830300/

há 5 anosdiscuss

I would also love to see that dhclient issue resolved upstream, or at least a cleaner way to work around it. But we should also be mindful that for most workloads the impact is probably way, way less.

Some of these things really only show up when you push things to their extremes, so it probably just wasn't on the developer's radar before.

há 5 anosdiscuss

Thanks, that actually means a lot. It took a lot of work, not just on the server/code, but also the writing. I asked a lot of people to review it (some multiple times) and made a ton of changes/edits over the last couple months.

Thanks again to my reviewers!

há 5 anosdiscuss

It is a statically generated site created with vitepress[1] and hosted on Cloudflare Pages[2]. The only dynamic functionality is the contact form which sends a JSON request to a Cloudflare Worker[3], which in turn dispatches the message to me via SNS[4].

It is modeled off of the code used to generate Vue blog[5], but I made a ton of little modifications, including some changes directly to vitepress.

Keep in mind that vitepress is very much an early work in progress and the blog functionality is just kinda tacked on, the default use case is documentation. It also definitely has bugs and is under heavy development so wouldn't recommend it quite yet unless you are actually interested in getting your handa dirty with Vue 3. I am glad I used it because it gave me an excuse to start learning Vue, but unless you are just using the default theme to create a documentation site, it will require some work.

1. https://vitepress.vuejs.org/

2. https://pages.cloudflare.com/

3. https://workers.cloudflare.com/

4. https://aws.amazon.com/sns/

3. https://github.com/vuejs/blog

há 5 anosdiscuss

This comprehensive level of extreme tuning is not going to be directly useful to most people; but there are a few things in there like SO_ATTACH_REUSEPORT_CBPF that I hope to see more servers and frameworks adopt. Similarly I think it is good to be aware of the adaptive interrupt capabilities of AWS instances, and the impacts of speculative execution mitigations, even if you stick to the defaults.

More importantly it is about the idea of using tools like Flamegraphs (or other profiling tools) to identify and eliminate your bottlenecks. It is also just fun to experiment and share the results (and the CloudFormation template). Plus it establishes a high water mark for what is possible, which also makes it useful for future experiments. At some point I would like to do a modified version of this that includes DB queries.

há 5 anosdiscuss

There is no published limit, but based on my tests the network device for the c5n.xlarge has a hard limit of 1.8M pps (which translates directly to req/s for small requests without pipelining).

There is also a quota system in place, so even though that is the hard limit, you can only operate at those speeds for a short time before you start getting rate-limited.

há 5 anosdiscuss

Yea, I looked it wrk2 but it was a no-go right out the gate. From what I recall the changes to handle coordinated omission use a timer that has a 1ms resolution. So basically things broke immediately because all requests were under 1ms.

há 5 anosdiscuss

Thanks!

> I always disable C-states deeper than C1E

AWS doesn't let you mess with c-states for instances smaller than a c5.9xlarge[1]. I did actually test it out on a 9xlarge just for kicks, but it didn't make a difference. Once this test starts, all CPUs are 99+% Busy for the duration of the test. I think it would factor in more if there were lots of CPUs, and some were idle during the test.

> Try receive flow steering for a possible boost

I think the stuff I do in the "perfect locality" section[2] (particularly SO_ATTACH_REUSEPORT_CBPF) achieves what receive flow steering would be trying to do, but more efficiently.

> Would also be interesting to discuss the impacts of turning off the xmit queue discipline

Yea, noqueue would definitely be a no-go on a constrained network, but when running the (t)wrk benchmark in the cluster placement group I didn't see any evidence of packet drops or retransmits. Drop only happened with the iperf test.

1. https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/processo...

2. https://talawah.io/blog/extreme-http-performance-tuning-one-...

há 5 anosdiscuss

Thanks!

The c5 instance type is about 10-15% faster than the m5, but the m5 has twice as much memory. So if memory is not a concern then switching to c5 is both a little cheaper and a little faster.

You shouldn't need the c5n, the regular c5 should be fine for most use cases, and it is cheaper.

Nginx in front of nodejs sounds like a solid starting point, but I can't claim to have a ton of experience with that combo.

talawahtech · HackerTrans