Ask HN: What parts of software testing can realistically be autonomous today?
Modern systems have good observability and better tooling than ever, yet testing often remains manual and slow.
In your experience, what aspects of testing can actually run autonomously today, and what still seems fundamentally hard or unsolved?
5 comments
Historically speaking, if an autonomous system found a bug [monkeys and fuzzers have done that for years ;)], you'd just get a "test failed" log. You'd still need to manually dig through the stack traces to pinpoint the "why" of it. Well, an Ai app can tell you that the system crashed or a button moved 5px to the left. But it still can't tell you if that movement broke the UX without a human explicitely defining the heuristic first.
IMO, the hard parts that actually getting solved right now is the autonomous serialization of state; context capture and issue repro steps to be precise (and definitely not the "finding of bugs" part).
IMO, the hard parts that actually getting solved right now is the autonomous serialization of state; context capture and issue repro steps to be precise (and definitely not the "finding of bugs" part).
It's an interesting one. Every time I speak with engineering teams about reliability and correctness, they all want more of it, yet when it comes to investing in it, it's never really a priority.
More often than not, people test the wrong things; they struggle to even identify the right properties to test.
I question my worldview on this because I don't think it's a particularly difficult problem. There are companies like Antithesis that have done incredible work in this space.
I am building in automated property-based testing, and it's not an easy sell.
More often than not, people test the wrong things; they struggle to even identify the right properties to test.
I question my worldview on this because I don't think it's a particularly difficult problem. There are companies like Antithesis that have done incredible work in this space.
I am building in automated property-based testing, and it's not an easy sell.
We did not fail to automate testing. We automated the easy parts and kept calling it progress. The hard part was never running tests. It was deciding what actually matters. We cannot automatically detect risk. Risk lives in context, trade-offs, and user expectations...none of those are deterministic.
Testing is answering "does it do, what it is supposed to do?" and autonomous means "according to it's own law(s)". Sounds like a contradiction to me. I'd answer with "none".
One can define properties the software is supposed to have, then autonomously test for those properties (as in, initiate a process that spends arbitrary amounts of time running new tests to try to show the software fails to have those properties.)
Is this not autonomous because the properties weren't created without humans being involved? How could that even be possible?
Is this not autonomous because the properties weren't created without humans being involved? How could that even be possible?