I have to admit I used to think any level of formal verification was worthwhile.
But I recently tried having agents try starting with formal models, and then building code using that as basis. In the end to models were never detailed enough to catch the most common issues that were shaken out with getting a high level of coverage and mutation testing.
At which point, the predictive power of the formal models became somewhat vacuous.
I’m trying to be more optimistic that maybe using dependently typed languages will yield better results.
I’ve been loving it for personal projects. But for work, some of our homebrew tooling relies heavily on hashes. So any operation that causes snapshotting to occur tends to lead to breakage that utterly confounds Claude. I tried giving it rules and instructions to make sure to sync git up to the correct commit for the current snapshot, but as soon as the signal for that is too week in the context it blows a gasket trying to figure out what has gone wrong.
It might have more to do with the first release of Z3 being in 2012, with the first stable Rust release being in 2015. Rather than the authors of Z3 passing some kind of judgment on Rust…
Not sure what you are using, but that’s easier said than done. I just set up an agent to ensure that my other agent would follow my coding guidelines by using hooks. The coding agent responded by switching to editing with `sed`, etc. to circumvent the hooks.
Claude Opus is going to give zero fucks about your attempts to manage it.
A bit back I got a https://kryoflux.com/ to try to get some data off old Apple II disks. I think I successfully captured the images, but then I couldn’t really find any tools that would left me examine the filesystems.
But I recently tried having agents try starting with formal models, and then building code using that as basis. In the end to models were never detailed enough to catch the most common issues that were shaken out with getting a high level of coverage and mutation testing.
At which point, the predictive power of the formal models became somewhat vacuous.
I’m trying to be more optimistic that maybe using dependently typed languages will yield better results.