Taurus: Writing JMeter Load Tests as Code(sheshbabu.com)
sheshbabu.com
Taurus: Writing JMeter Load Tests as Code
https://www.sheshbabu.com/posts/taurus-writing-jmeter-load-tests-as-code/
4 comments
It is XML based with an optional
GUI. I’ve programmatically generated and executed JMeter test plans in the past without touching the GUI.
Not bad. I still prefer wrk/wrk2 and a simple lua script but it doesn't have the bells and whistles of jmeter.
why yaml plus real code? why not just real code? does yaml ever actually provide benefits in testing like this?
On some level, they're just providing an interface by which you can configure your tests, and a tool where you can send that configuration to run those tests.
So...Why YAML (plus a weak escape-hatch) and not something obviously Turing-complete for that interface?
1. Some fraction of the target audience aren't programmers. I know a lot of people who can look at a few YAML examples, play with a few requests in Postman, and produce valid-enough tests. I know a lot fewer who can write that in any "real" language.
2. There's an obvious "right" way to use a tool like this. Each scenario has certain properties spatially colocated, the scenarios are colocated, .... Turing-complete languages make it hard to expose an interface enforcing good, nice, safe patterns just via the type signature. That's a blessing and a curse (I'd much rather have an escape hatch, personally), but it's definitely a benefit for somebody.
3. If you don't "need" (it's, admittedly, hard to determine this up-front, and I think Turing-complete YAML comments are far worse than just using a Turing-complete configuration language) a Turing-complete language, reaching for one is asking for trouble. I can expand if you're curious, but going into the rabbit-hole of "principle of least power" should at least give you a few ideas to poke at.
4. Even amongst programmers, many more people know JSON, CSV, YAML, NDJSON, XML, or _some_ quasi-human-readable configuration language than those who know your favorite Turing-complete option. Look at neovim and their init.lua file as an example. How many active users actually know how to do anything meaningful with it? How many struggle even to add a few keybindings and a plugin whose README doesn't explicitly spell out how it interacts with their favorite plugin manager? Lua isn't a hard language, but diving into a totally foreign ecosystem takes time.
5. Suppose you have something Turing-complete in the middle. Now your hands are tied to that ecosystem. You have to bundle a language or re-write an interpreter/compiler. Alternative implementations are similarly constrained. Tree-sitter grammers are a pain-in-the-ass to write if you want to make your config files easy to edit in people's favorite editors. User bug-reports might be real bugs or might just be programmers making dumb programmer mistakes again. This point is kind of an elaboration of point (3), but there are real costs to full-blown languages.
Mind you, with YAML having a 1000+ page spec, never once implemented properly, never once fully understood by any one person, and countless Fortune-500 companies having major issues due to unexpected YAML heuristics, I don't think that would be my first choice of configuration language. It has the benefit of popularity because of Kubernetes and CloudFormation and that whole ecosystem though, so I can see the point.
So...Why YAML (plus a weak escape-hatch) and not something obviously Turing-complete for that interface?
1. Some fraction of the target audience aren't programmers. I know a lot of people who can look at a few YAML examples, play with a few requests in Postman, and produce valid-enough tests. I know a lot fewer who can write that in any "real" language.
2. There's an obvious "right" way to use a tool like this. Each scenario has certain properties spatially colocated, the scenarios are colocated, .... Turing-complete languages make it hard to expose an interface enforcing good, nice, safe patterns just via the type signature. That's a blessing and a curse (I'd much rather have an escape hatch, personally), but it's definitely a benefit for somebody.
3. If you don't "need" (it's, admittedly, hard to determine this up-front, and I think Turing-complete YAML comments are far worse than just using a Turing-complete configuration language) a Turing-complete language, reaching for one is asking for trouble. I can expand if you're curious, but going into the rabbit-hole of "principle of least power" should at least give you a few ideas to poke at.
4. Even amongst programmers, many more people know JSON, CSV, YAML, NDJSON, XML, or _some_ quasi-human-readable configuration language than those who know your favorite Turing-complete option. Look at neovim and their init.lua file as an example. How many active users actually know how to do anything meaningful with it? How many struggle even to add a few keybindings and a plugin whose README doesn't explicitly spell out how it interacts with their favorite plugin manager? Lua isn't a hard language, but diving into a totally foreign ecosystem takes time.
5. Suppose you have something Turing-complete in the middle. Now your hands are tied to that ecosystem. You have to bundle a language or re-write an interpreter/compiler. Alternative implementations are similarly constrained. Tree-sitter grammers are a pain-in-the-ass to write if you want to make your config files easy to edit in people's favorite editors. User bug-reports might be real bugs or might just be programmers making dumb programmer mistakes again. This point is kind of an elaboration of point (3), but there are real costs to full-blown languages.
Mind you, with YAML having a 1000+ page spec, never once implemented properly, never once fully understood by any one person, and countless Fortune-500 companies having major issues due to unexpected YAML heuristics, I don't think that would be my first choice of configuration language. It has the benefit of popularity because of Kubernetes and CloudFormation and that whole ecosystem though, so I can see the point.