OK. Yeah so I made the decision early to go one level below Playwright and target CDP directly, because I believe I need the tight connection with Chrome/Chromium to make it as fast as possible. Also I really value the distribution aspects of writing this in Rust and compiling to a single executable with everything in it.
Thanks! I don't know much about it, but from what I can tell from their site, it does differential testing based on screenshots and behavior, and mocks out network IO, focusing on the client side. I'm not sure what the AI part is, if it's for extracting information from the page, for generating actions, for smarter diffing, or some combination of those.
Bombadil doesn't mock the network (although I'm working with a contributor to add something like Playwright's Route system for manipulating responses). It runs on your full stack (if you want) and if you need determinism for perfect bug reproducibility, you should run it all in Antithesis.
On the AI front, my view is that Bombadil should be optimized for using LLMs to generate and refine specs over time (properties and action generators), but that it should be an offline process. It's not like a "browser testing agent". It's analogous to having the LLM generate a program, rather than having the LLM itself perform actions. For some problems, the latter makes more sense, but for many problems the former is better, and I believe that PBT (and things like proof systems, but that's not my domain) are suited to the former strategy.
I have some sketches on how the refinement-with-human-in-the-loop thing should work with Bombadil and LLMs, but it's only a roadmap item for now.
One smartass way to answer that for the general case is: you don't. And I mean that in the same sense as "you don't write example based tests using a PBT framework". You supply independent actions through generators and have it apply them "chaotically", not strictly in sensible sequences.
That said, for things like auth, you need to do some fixed steps to get anywhere at all. I haven't added support for this yet, but as I'm envisioning it, you'd either:
a) authenticate the browser using headers or cookies that you inject into the Bombadil browser, or
b) use a "custom action" (essentially a stringifed JS function) that'd perform your particular steps
Or what you could do now: add preconditions to your actions so that they'd necessarily fill out the form before anything else could be done.
Another thing we might need in Bombadil is secrets, but for now I think it's fine to inject test credentials that are plain visible text.
Please let me know if you try it out and if you need any help!
> I think his jolly chaotic good energy, along with how he's uncorruptible and sort of outside the normal world, fits a fuzzer in a spiritual sense. But it might all just be a good excuse to use one of my favorite names from my favorite book.
It's also my favorite part of the book, and definitely not something I'd skip. I know some people would object, but I think of it as an homage rather than stealing.
Go for it! I've been meaning to do an "architecture of Bombadil" blog post that'd likely answer this question. It's not super advanced by any means, but it's a mindset shift to how you might think about browser testing coming from the mainstream frameworks like Playwright.
Yes, and in fact, capturing "prior" values with bindings and closing over them in temporal operator thunks is how you talk about some relation between s and s' in a Bombadil formula (not having that particular syntax though). It's a deliberate way of embedding this LTL flavor in JS/TS in the most natural and ergonomic way I could think of. I didn't want a deep EDSL or even a new bespoke spec language that people (and LLMs) would have to learn, and to have to write tools for. Now you can write Bombadil specs with a good LSP and be able to import packages off of npm or whathaveyou. Most web devs will probably be comfy with JS or TS, so that's why I chose that style.
Hey, yeah, so actions are indeed independent and you can't express fixed sequences of them (e.g. fill out thus form by doing X, then Y, then Z). If you want to enforce certain sequentiality you'd need to precondition your generators. This is admittedly limited right now. I do want to add QoL things like `filter` on the generators to make that more ergonomic.
Also, as you note, you can't implement custom actions. And that's just something that I haven't gotten to yet. It'd be quite straightforward to add a plain function (toStringed) as one of the variants of the Action type and send that to the browser for eval. (Btw, we're taking contributions!)
Weighting is also important right now. There's no smart exploration in Bombadil yet, only blind random, so manual weighting becomes pretty crucial to have more effective tests (i.e. unlikely to run in circles). I'd like to both make the Bombadil "fuzzer" better at this, but eventually you might want to run Bombadil inside Antithesis instead to get a much better exploration, even in a single campaign.
The Until operator is also one of those things that I haven't gotten to yet. I actually didn't expect someone to hit this as a major limitation of the tool so quickly, which is why I've focused on other things. Surprising!
To add some more context, Bombadil is an OSS project with one developer (but we're hiring!) and it's 4 months old and marked experimental. I'm sorry you were disappointed by its current state, but it's very early days, so expect a lot of these things to improve. And your feedback will be taken into account. Thanks!
Not yet but I definitely will record one soon. I've been working at the debugging UI for Bombadil (some hints at https://x.com/owickstrom/status/2034925537706020978) that will make for a nice cohesive demo.
Cool. I think that is a very neat way of expressing properties of UIs (and stateful systems more generally) that works out nicely in testing. There are some gotchas related to the finiteness of testing, but it's manageable.
That's true, clean app state gets you far. And that's something I'm going to add to Bombadil once it gets an ability to run many tests (broad exploration, reruns, shrinking), i.e. something in the spec where you can supply reset hooks, maybe just bash commands.
Regarding state machines: yeah, it can often become an as-complex mirror of the system your testing, if the system has a large complicated surface. If on the other hand the API is simple and encapsulates a lot of complexity (like Ousterhout's "Deep Modules") state machine specs and model-based testing make more sense. Testing a key-value store is a great example of this.
I'd say it does! Bombadil is very new but its predecessor has found complicated and very real bugs in my work projects, and in the paper we wrote about it, we found bugs in more than half of the TodoMVC example apps: https://arxiv.org/abs/2203.11532
This is a great point. Bombadil _is_ also tied to the DOM much like those tools, but as you focus on providing just a set of generators (which can be largely the defaults already supplied by Bombadil), you get a lot of testing from a small spec. You might need to specify some parts in terms of DOM selectors and such, and that has coupling, but I think the power-to-weight ratio is a lot better because of PBT.
Good feedback! Short answer: a lot of stuff is remaining. It's a very new projects and I've been trying to cover the basics. There's a ton to do around better state space exploration, reporting/debugging (working on this now!), integration with other tools and platforms like CI, etc. But a living section in the README or the Manual for "planned but not yet built" probably makes sense.