Continuous Load Testing(slack.engineering)
slack.engineering
Continuous Load Testing
https://slack.engineering/continuous-load-testing/
6 comments
Hi James, I'm working on a load testing tool product. Any chance I could talk to you about your headaches, and how you do things? I'm trying to learn what people are doing now, and where the pain is. Maybe you could tell me if my ideas are crazy--I am taking a different approach than what's out there. :-) I'm at ebeland on gmail.
I like to run over night. It's not superfast feedback but still a reasonable timeframe
Maybe they should be doing continuous UI latency testing
I don't know how Slack operates as a business, but if they work like most companies I've worked for, every employee gets a similarly spec-ed out machine. Maybe it's a MacBook, maybe not.
But at least some people should get machines that are a few years old - because that's what the users will have.
This used to be a classic problem in PC game development - developers get kitted out machines (because they need to run more heavy duty tasks during development, or run GPU debug code, or need to test the limits of the engine, or maybe just because they are gamers themselves so they buy newer hardware more often), so they tend to miss the mark on how powerful the average consumer machine is and only optimize code until it runs smoothly on their machine.
But at least some people should get machines that are a few years old - because that's what the users will have.
This used to be a classic problem in PC game development - developers get kitted out machines (because they need to run more heavy duty tasks during development, or run GPU debug code, or need to test the limits of the engine, or maybe just because they are gamers themselves so they buy newer hardware more often), so they tend to miss the mark on how powerful the average consumer machine is and only optimize code until it runs smoothly on their machine.
Shush, be calm. Just keep writing more javascript...
Pipelines take a lot longer to run (because the load test environments are expensive we have one or two and serialize test runs on them to stop test runs stepping on each other, but the teardowns are slow, especially on failed runs)
To keep the test run fast we limit them to about 5 minutes but this creates unrealistic expectations of how fast our code would actually need to scale.
But it is nice knowing which changes have broken the load test. Hunting them down is so painful.
What I’d to have is proper metrics showing how close we are to our Non Functional Requirements over time to make it easier to sell “Performance stories” ahead of breaking the NFR rather than just fixing it as we break.