I have interviewed CTOs & EMs of > 50 new-age companies about their testing cultures And learned something completely opposite to what I believed while working at Morgan Stanley.
We have a anonymiser which identifies common sensitive /Personally identifiable data like credit card, zip code and replaces them with anonymised data.
We also provide configuration option to specify additional fields are needed to be anonymised
Are you somehow automatically hooking into those function calls, recording their return values, and then mocking the functions in the replays? - this is correct.
Yes, we rely a lot on openTelemetry for this. They have really good support for most libraries in Java, node and are progressing quickly in others. We are also contributing to it by extending support for other languages, which we'll be open sourcing soon.
I can relate to this perspective, however, some complexities we have come across in building this so far:
- Support for high number of languages, downstream dependencies
- Intelligent sampling to choose requests with high coverage and auto update them over time
- Performance, safety and data compliance guarantees
Good observation, it's challenging to solve these problems, here's how we are going about it -
To reduce false positives - we run the same request twice to eliminate flaky fields in response like current timestamp, mock the downstream dependencies as they behaved in prod env and are providing options to ignore / modify the sampled requests
To make integration easier - we are building on top of opentelemetry which has seen remarkable increase in support across languages / frameworks, which makes it easier for us to support different tech stacks.
Regression testing - our primary goal is to provide regression tests. We have come across two type of teams where this makes sense - companies with low test coverage and companies which high number of micro-services as they find it hard to cover every production scenario in tests
it's similar in the sense that both rely on production traffic and user sessions to generate tests. However, we are focusing on API testing and I think Meticulous is building for UI testing.