Author here :) We did have high-level metrics and expectations for how this change would behave, but a couple of factors made it much harder to reason about in practice that were happening in parallel.
Data in these systems moves slowly and with a lot of inertia, so the effects show up gradually and can lag behind the change itself. On top of that, the impact wasn’t uniform. Most of the overhead came from a small subset of volumes, so it took time to isolate what was actually driving the increase. These systems are hard to test at scale!
Author here. With SMR, you do have large zones that are essentially immutable. However, in this case our extents and volumes are immutable because we do volume level striping for erasure coding. This mean that if any extent changes, the parities have to be rewritten as well. Others, do block level striping, so they can just move data around within disk. There are lots of trade-offs with both approaches. Also, keeping volumes/extents immutable makes reasoning through correctness much simpler.
Data in these systems moves slowly and with a lot of inertia, so the effects show up gradually and can lag behind the change itself. On top of that, the impact wasn’t uniform. Most of the overhead came from a small subset of volumes, so it took time to isolate what was actually driving the increase. These systems are hard to test at scale!