Very cool! I implemented some shared-memory atomic snapshots in Rust [0] and also did my best to take automated testing very seriously. I started out using loom [1], the library mentioned in the article, but latter switched to shuttle [2].
Shuttle's approach is to be randomized, instead of exhaustive like loom. However, the scheduler does still give probabilistic guarantees about finding bugs. I found that shuttle was faster and scaled to more complicated test scenarios.
Similar to the article, shuttle also lets you save the random seed if a particular schedule causes the test suite to fail. Being able to quickly reproduce failing tests is really important and enables you write explicit test cases for bugs that were previously caught and fixed [3].
I see this as typical of organizations that value creating a more inclusive environment for their developers.
Also, for what it's worth, Jenkins changed their terminology to use "agent" instead of slave in 2020 [0]. These efforts might seem futile in isolation, but they add up over time.
I've been working on implementations of classic algorithms in distributed computing, and used Turmoil for testing correctness in message-passing / HTTP systems [0].
Overall, my experience has been positive. When it works, it's great. A pattern I've been following is to have a single fixture that returns a simulation of the system under a standard configuration, for example N replicas of an atomic register, so that each test looks like:
1. Modify the simulation with something like`turmoil::hold("client", "replica-1")`.
2. Submit a request to the server.
3. Make an assertion about either the response, or the state of the simulation once the request has been made. For example, if only some replicas are faulty, the request should succeed, but if too many replicas are faulty, the request / simulation should time-out.
One of the things I have found difficut is that when a test fails, it can be hard to tell if my code is wrong, or if I am using Turmoil incorrectly. I've had to do some deep-dives into the source in order to fully understand what happens, as the behavior sometimes doesn't line-up with my understanding of the documentation.
Disjoint-Sets have a very cool implementation whose amortized time complexity is extremely slow growing. It is not quite constant, but even for a disjoint-set with as many elements as there are particles in the universe, the amortized cost of an operation will be less than or equal to 4.
Shuttle's approach is to be randomized, instead of exhaustive like loom. However, the scheduler does still give probabilistic guarantees about finding bugs. I found that shuttle was faster and scaled to more complicated test scenarios.
Similar to the article, shuttle also lets you save the random seed if a particular schedule causes the test suite to fail. Being able to quickly reproduce failing tests is really important and enables you write explicit test cases for bugs that were previously caught and fixed [3].
[0] https://github.com/kaymanb/todc/tree/main/todc-mem [1] https://github.com/tokio-rs/loom [2] https://github.com/awslabs/shuttle [3] https://github.com/kaymanb/todc/blob/0e2874a70ec8beed8fae773...