Thanks for the suggestions. I thought tilted was less rigid and matched the aesthetic of the dungeon maps at https://watabou.itch.io/one-page-dungeon which I like. But I hear you. You can press space to see the full puzzle, but it can be hard to read the words on a large level. That was one UI concern I didn't quite figure out. I'll consider your suggestions!
OP here, cool to see all the similar yet varied testing approaches! Your situation sounds tricky with a real time physics based game. Converting it to step based sounds like it has promise, but as you mentioned, every dilution to the full e2e harness also dilutes the validation veracity. When you were describing your game I kept thinking of Bret Victor's "Inventing on Principles" talk where he "collapses time" in a physics game to render trajectories of objects in all positions at once to visually intuitively tell if it works right. Perhaps that could apply?
Thank you. You have a great suggestion. I didn't do that, but I did consider it and I think it can be very powerful. I had 2 example use cases that having an actual AI felt good for, first validating a new feature based on the spec, and second, finding unexpected bugs (like trying to enter a locked room through the back wall). It didn't do so well on the latter, but did great on the former. Having a million simulated games could probably catch those, but how would you track the reports after? Perhaps using an LLM to read the logs/reports could be a good use. Your set up sounds awesome, nice work.
OP here: Thank you and I appreciate the thoughtful questions. To answer:
1) I used a text representation because it made sense for my game and let me "render" certain details in a more AI-friendly way, like the compact map. You could use something like agent-browser and it would probably work just fine, but I figured it added an extra layer of indirection that I didn't need, plus it would be a lot of screenshots! Being able to have a turn based loop really helped make this work.
2) I had a skill on just how to use the playtest server. I also gave it context on what the game is and how to play it. From there, it probably depends on your use case. I wasn't that impressed with its natural ability to playtest for bug discovery, so I would consider making a skill describing what a playtester would normally do. Focused playtester instances is a good idea. Ultimately what I found to be most helpful was to point it at a feature or bug that I was aware of and have it validate it. Not only was it fairly successful, that was the part that saved the most time for me.
3) I think I only burned about 300K tokens on my longest play-test session, and that includes a bunch of code tweaks too. Running it after every feature as a validation step is pretty cheap. Running it overnight in "open" playtesting could add up.
Good luck, please let me know how it goes if you get somewhere helpful!
OP here. Thanks for the feedback. I agree that frame to frame consistency is quite bad currently. I did address that in the post, hinting at some of the techniques others have mentioned here, like in/out-painting and masking previous frames.
For me, the exciting parts of this experiment was finding the opportunities and limits of realtime generation, and exploring ways of grounding generated content in a solid yet player controlled world layer.
OP here, I remember both of the draw fast and lens demos! I'm pretty sure those were in the back of my subconscious, inspiring me to explore my take on real time rendering. Thanks for sharing your similar experience. I agree, this was a lot of fun to work on, and like one of the other commenters pointed out, experiencing it viscerally is a whole new kind of feeling, even with the consistency issues. I'd also like to see more experiments on what new kinds of UX could be possible with this tech.