Ask HN: What are you using for http load testing?
I'm looking for something more flexible than apache ab, but not as heavy weight as JMeter. Ideally I'd like to be able to specify some random values in urls or post requests. My environment is linux/ubuntu. What are you using these days?
34 comments
I like to use wrk[1] for quick/spot testing:
* It allows me to focus on latency versus throughput independently
* It has tools for testing performance on pipelined requests (very important in my space)
* It supports scripting complex authentication processes (like magic headers and oauth2)
* It supports scripting fuzzing (random values) for URLs and POST requests
[1]: https://github.com/wg/wrk
For wide-area load testing, I simply buy some advertising since that gives me cheap access to millions of simultaneous users.
* It allows me to focus on latency versus throughput independently
* It has tools for testing performance on pipelined requests (very important in my space)
* It supports scripting complex authentication processes (like magic headers and oauth2)
* It supports scripting fuzzing (random values) for URLs and POST requests
[1]: https://github.com/wg/wrk
For wide-area load testing, I simply buy some advertising since that gives me cheap access to millions of simultaneous users.
We use wrk since we are often load testing fairly simple services, occasionally with some Lua for cache busting. It's a great tool. Would love if it supported http2 since I work on CDN type stuff where that matters.
I love your 2nd approach (advertising). Any pointers to how to get started?
I love your 2nd approach (advertising). Any pointers to how to get started?
go-wrk[1] promises HTTP/2 support and is similar to wrk. I don't have a lot of experience with it though.
[1]: https://github.com/tsliwowicz/go-wrk
If you want to purchase advertising for this purpose, I'm happy to help with this. My contact details are linked through my profile.
- Your HTTP endpoints will need to be whitelisted with traffic quality groups and creative scanners (This can take a week or so). I recommend using a separate dns domain name for this since adblockers tend to subscribe to these lists.
- Your JavaScript should not be abusive. There are subjective tests for this. Generally doing an image request or some POST data of a few kb isn't going to be an issue. Some ad servers (like sizmek) download 2mb "test images" to exercise their bandwidth from some locations (like the US).
- Costs vary based on device, region, the web page the ads are on, and silly things like demographic. Some of these factors are interesting to benchmarkers but others are not. Thinking about what you actually need to test can help keep your costs efficient.
[1]: https://github.com/tsliwowicz/go-wrk
If you want to purchase advertising for this purpose, I'm happy to help with this. My contact details are linked through my profile.
- Your HTTP endpoints will need to be whitelisted with traffic quality groups and creative scanners (This can take a week or so). I recommend using a separate dns domain name for this since adblockers tend to subscribe to these lists.
- Your JavaScript should not be abusive. There are subjective tests for this. Generally doing an image request or some POST data of a few kb isn't going to be an issue. Some ad servers (like sizmek) download 2mb "test images" to exercise their bandwidth from some locations (like the US).
- Costs vary based on device, region, the web page the ads are on, and silly things like demographic. Some of these factors are interesting to benchmarkers but others are not. Thinking about what you actually need to test can help keep your costs efficient.
I agree. I also love wrk and it is pretty useful and allows scripting
Here is a good list — https://github.com/denji/awesome-http-benchmark
It includes the good ones like Vegeta, Siege, Goad, Hey, Httpref, Gatling, among others.
It includes the good ones like Vegeta, Siege, Goad, Hey, Httpref, Gatling, among others.
Locust. https://locust.io
+1. Our team also uses Locust.
Plus 1 for this!
From the Microsoft world:
WCAT: https://www.iis.net/downloads/community/2007/05/wcat-63-x86 (pretty old and quite manual, but found it really effective, can control many clients distributing the load)
West Wind's WebSurge: https://websurge.west-wind.com/ Bloody good bit of kit and very easy to user
Bombadier: https://github.com/codesenberg/bombardier Very fast Go HTTP(S) benchmarking.
And Netling: https://github.com/hallatore/Netling I just really like it.
WCAT: https://www.iis.net/downloads/community/2007/05/wcat-63-x86 (pretty old and quite manual, but found it really effective, can control many clients distributing the load)
West Wind's WebSurge: https://websurge.west-wind.com/ Bloody good bit of kit and very easy to user
Bombadier: https://github.com/codesenberg/bombardier Very fast Go HTTP(S) benchmarking.
And Netling: https://github.com/hallatore/Netling I just really like it.
> not as heavy weight as JMeter
JMeter is indeed a tad heavier, but I find it's much more user friendly as in, you spend a couple of hours to build basic tests, or afternoon and have learned most of its functionality. It also has a variety of ways implementing this randomized user behavior [1].
https://tpaschalis.github.io/jmeter-random-user-actions/
JMeter is indeed a tad heavier, but I find it's much more user friendly as in, you spend a couple of hours to build basic tests, or afternoon and have learned most of its functionality. It also has a variety of ways implementing this randomized user behavior [1].
https://tpaschalis.github.io/jmeter-random-user-actions/
You can definitely do all those things with JMeter, although the user experience is pretty naf, Gatling[3] is definitely much better at that front if you're comfortable writing Scala.
When it comes to Protocol Level Testing, those two tools are still pretty much the best bang for buck, and by far the most popular open source load generators.
If the target you're hitting is a web app and you want to create a more realistic load scenario without crafting individual requests, check out a tool called Flood Chrome, which uses thousands of instances of Google Chrome[1] to generate load. It uses the same scripting language as SeleniumJS, so it's quite easy to learn.
Also on the topic of Flood [2], they're a very good platform for abstracting away the server orchestration and results collection you'll face when you try to scale up from 1 to N servers running either Gatling or JMeter.
[1]: https://chrome.flood.io/
[2]: https://flood.io/
[3]: https://gatling.io/
When it comes to Protocol Level Testing, those two tools are still pretty much the best bang for buck, and by far the most popular open source load generators.
If the target you're hitting is a web app and you want to create a more realistic load scenario without crafting individual requests, check out a tool called Flood Chrome, which uses thousands of instances of Google Chrome[1] to generate load. It uses the same scripting language as SeleniumJS, so it's quite easy to learn.
Also on the topic of Flood [2], they're a very good platform for abstracting away the server orchestration and results collection you'll face when you try to scale up from 1 to N servers running either Gatling or JMeter.
[1]: https://chrome.flood.io/
[2]: https://flood.io/
[3]: https://gatling.io/
Vegeta. https://github.com/tsenart/vegeta. The following post covers an approach for generating some randomness. https://thisdata.com/blog/load-testing-api-interfaces-with-g....
Load testing can be hard. Wrk works for me for the most part but it's not ideal for complex flows (say e-commerce trying to simulate a real browse and checkout complete with typeahead search).
However the smarter apps can have problems too. I've pinned all available cores on fairly beefy machines because they are gunzipping bodies (behaving more like a real browser).
If you are at a scale where you need really good testing I recommend that you think of wrk results as comparable to a unit test and something like jmeter as comparable to an integration test. It's not worth the effort for most companies but where I work we serve a decent amount of traffic and downtime costs 6 figures per minute so good performance engineering and good business overlap.
However the smarter apps can have problems too. I've pinned all available cores on fairly beefy machines because they are gunzipping bodies (behaving more like a real browser).
If you are at a scale where you need really good testing I recommend that you think of wrk results as comparable to a unit test and something like jmeter as comparable to an integration test. It's not worth the effort for most companies but where I work we serve a decent amount of traffic and downtime costs 6 figures per minute so good performance engineering and good business overlap.
Gatling (https://gatling.io)
Artillery via AWS lambda https://github.com/Nordstrom/serverless-artillery
Bees with machineguns
https://github.com/newsapps/beeswithmachineguns
I use hey[1].
Shameless plug:
I have developed a small tool called vex [2] ( using PHP ) for this purpose but it's not feature complete.
[1] https://github.com/rakyll/hey [2] https://github.com/vamsiikrishna/vex
Shameless plug:
I have developed a small tool called vex [2] ( using PHP ) for this purpose but it's not feature complete.
[1] https://github.com/rakyll/hey [2] https://github.com/vamsiikrishna/vex
Jmeter is de facto in telecom , it's far from perfect but it let you define complex flows quick , the pain is that you need to ménage the test plans on the server to run all the jmx files
I wish there was full web interface on the server already
We use JMeter and run them on AWS load generators using redline13.com
One ot the best tools for load testing on ubuntu is yandex tank.
https://gettaurus.org/ + BlazeMeter (for reports)
[deleted]
k6 by Load Impact: https://k6.io
a mix of ab and vegeta ( https://github.com/tsenart/vegeta ). More and more vegeta lately, thou.
Siege
apache benchmark (ab) is a good alternative for simple test scenarios
HN users.