Locust – A modern load testing framework(locust.io)
locust.io
Locust – A modern load testing framework
http://locust.io/
22 comments
That isn't a very good comparison, because Locust's use-case is very different.
Locust is a tool/framework for writing code that simulates real user behaviour in a fairly realistic way. For example, it's very common to store state for each simulated user. Once you have written your "user behaviour code", you can then simulate a lot of simultaneous users by running it distributed across multiple machines, and hopefully get realistic load sent to you system.
If I wanted to just send a lot of requests/s to one or very few URL endpoints, I would also use something like ApacheBench, and I'm author of Locust.
Locust is a tool/framework for writing code that simulates real user behaviour in a fairly realistic way. For example, it's very common to store state for each simulated user. Once you have written your "user behaviour code", you can then simulate a lot of simultaneous users by running it distributed across multiple machines, and hopefully get realistic load sent to you system.
If I wanted to just send a lot of requests/s to one or very few URL endpoints, I would also use something like ApacheBench, and I'm author of Locust.
If you're using ab or httpperf, you should also take a look at wrk (https://github.com/wg/wrk). It can generate significant amount of load and has support for Lua scripting for request generation, response processing and reporting.
There is another fork of it (https://github.com/giltene/wrk2) which can be used if you want test at a consistent RPS.
There is another fork of it (https://github.com/giltene/wrk2) which can be used if you want test at a consistent RPS.
Used it for some substantial loadtesting, loved it.
It's real advantage compared to AB, siege, etc, is how easy it makes it to simulate real user interactions, not just mash on a few endpoints.
Only flaw in my books is the crazy high ulimit you need for simulating high numbers of unique actors. The easy clustering mitigates this though.
Shameless self promotion - I just wrote up a little doc [1] on using Locust and Kubernetes to run and scale distributed load tests along with some sample code [2].
I looked at JMeter and Gatling as alternatives but they were either a bit clunky (JMeter) or not distributed out of the box (Gatling). Late in the game I did discover Tsung (Erlang) but didn't get time to try it out - looks pretty powerful though.
[1] https://cloud.google.com/solutions/distributed-load-testing-...
[2] https://github.com/GoogleCloudPlatform/distributed-load-test...
I looked at JMeter and Gatling as alternatives but they were either a bit clunky (JMeter) or not distributed out of the box (Gatling). Late in the game I did discover Tsung (Erlang) but didn't get time to try it out - looks pretty powerful though.
[1] https://cloud.google.com/solutions/distributed-load-testing-...
[2] https://github.com/GoogleCloudPlatform/distributed-load-test...
Great writeup.
It is so close to what we built at tradewave.net (except for Locust). I am glad I am not the only one that saw load testing using scalable containers. At some point, we will be going towards HFT and having a true load testing suite is handy.
I used locust to supplement testing that third party was doing (they used some ibm product that I didn't have a license too).
I liked locust because it was easy to start using, but you write a decent amount of code if you have a complex application. I suggest using pyquery (https://pypi.python.org/pypi/pyquery) with it if you need to inspect any html, it was quite useful
I liked locust because it was easy to start using, but you write a decent amount of code if you have a complex application. I suggest using pyquery (https://pypi.python.org/pypi/pyquery) with it if you need to inspect any html, it was quite useful
Locust is a great tool. We use it to loadtest our websites and even a single Mac can make huge traffic. If somebody is interested in a highly optimized Vagrant VM for loadtesting and Locust (it's possible through a VM, but not exorbitant high loadtesting) check out this url https://github.com/PyBossa/loadtesting
My fave locust setup is spinning up aws instances to tests from time to time. I've created ansible automation to help on that (https://github.com/gleicon/locust-swarm) but it can be done manually. I still look on how to make it run casper.js tests but I think it would interfere with locust's performance.
Does it support testing a websocket app ?
Not out of the box. However, as long as it's "request/response based" it's quite simple to write and use a custom client. See: http://docs.locust.io/en/latest/testing-other-systems.html
Does anyone know how it compares to Gatling (http://gatling.io) ?
I used Locust for a good long time around a year back, and here are some observations.
Locust is awesome to start with, and has master/slave for spawning multiple servers, great for testing real concurrency with large loads. Gatling doesn't have master/slave mode yet.
Main problems with Locust are:
* Lack of good presentation of results (compare Gatling and Locust results, and Locust is very primitive)
* Lack of Pipelines/Scenarios. This IMO was a big deal breaker for me. You can't say: "100% users hit the home screen, out of which 70% add a product, out of which 25% actually pay" etc. This is easily modeled with Gatling, and the same request session with same data falls through each scenario. I'm still following https://github.com/locustio/locust/issues/171 though, but I don't think they have any idea of implementing it.
Locust is awesome to start with, and has master/slave for spawning multiple servers, great for testing real concurrency with large loads. Gatling doesn't have master/slave mode yet.
Main problems with Locust are:
* Lack of good presentation of results (compare Gatling and Locust results, and Locust is very primitive)
* Lack of Pipelines/Scenarios. This IMO was a big deal breaker for me. You can't say: "100% users hit the home screen, out of which 70% add a product, out of which 25% actually pay" etc. This is easily modeled with Gatling, and the same request session with same data falls through each scenario. I'm still following https://github.com/locustio/locust/issues/171 though, but I don't think they have any idea of implementing it.
Gatling is far more advanced but both tools share similar architectural designs regarding how they handle lots of traffic. Locust is more leightweight (and I would say also more easy, but depends on your needs) and is missing good graphical presentations of results and you will not get a browser recorder with it.
So from what I understand this stands somewhere between as complex as Apache Bench and something as simple as Seige? I've had very little success with the former since its scripts are massive XML files and the clunky UI also doesn't help much.
As a python developer, I'm glad to see this and will surely give this a try shortly. Thanks!
As a python developer, I'm glad to see this and will surely give this a try shortly. Thanks!
I've been using siege for load testing. I'll give this a go at some point.
I don't use python as much as I'd like and this seems like a good work related use-case for me.
I don't use python as much as I'd like and this seems like a good work related use-case for me.
I used it to test a Django app once. It's great because you can test the entire "flow" of an app, clicking on various forms.
Does it provide graphical data representation or any kind of dashboard/intergation with graphite or something else?
Do we have an equivalent for WS yet?
A bit tongue in cheek, but is a testimonial from someone at EA/Dice something you'd want to highlight? I know Battlefield 4 is almost two years old at this point, but man, I still have the scars from that game...
In all seriousness, definitely going to check this out, not enough load testing is done at the company I work for, plus I want to learn more python.
In all seriousness, definitely going to check this out, not enough load testing is done at the company I work for, plus I want to learn more python.
I have exactly the same point:) More python, balancing needs to be done, Battlefield 4.. ouuuuch.
Furthermore there were a lot of timing inaccuracies between the same test runs. It certainly could have been a fault in our code, but ab and Httperf seemed to give a lot more reliable and useful timing information. And they are just simpler to use. For one, not being browserbased can be nice when you are trying to iterate quickly.
Not trying to sell anyone against Locust, but at the least, I recommend you take a look at ab and Httperf to make sure you don't need something a whole lot simpler.