Let me share a couple demos, with the caveat that this project is very much still a beta. There's a bug report/feature request button in-app, so if you notice anything egregious feel free to reach out there.
[0] Building a hubless bucky ball dome. It's a bit deceptively-simple, because the agent only has to call the procedure with the right values. That's an intended flywheel: agents and users can save procedures to a community library, and future agents can discover and invoke the pre-existing procedures, instead of re-deriving them. They can copy and edit the procedure as needed, so it basically becomes a RAG library for carpentry procedures written in the project's DSL.
[1] A sawhorse, re-implemented by Opus 4.8 from plans published by Home Depot. Now saved as a procedure so you can drop a sawhorse anywhere you need one.
[2] A small shed with stick frame walls and a lean-to roof.
[3] The Picture frame prompt mentioned in my first comment.
Quick note: I recently broke the smart de-duplication for step-by-step playback, so it will show every single cut, one by one. Hit "stop" and then click the final frame in the timeline to see just the finished state.
It's a woodworking simulator. There are tools (tape measure, pencil, miter saw, table saw, jigsaw, drill, router) and wood stocks (2x4s, plywood).
There is a chat window and human UI similar to SketchUp. You can task the agent with using the tools and assembling the project, or do it yourself. It outputs real cad files, plans, and a build guide video.
Qwen 3.6 has been great because it's multimodal and good at tool calling. As it builds it can get screenshots and all that. The basic output is an operation list with real measurements and bevel/angle settings and the like.
I've been using Qwen3.6-Plus in my carpentry-simulator agent harness. That model performs pretty well, but not at Opus 4.7 levels. I will try 3.7 Plus when the pricing is announced and report back.
I use Qwen3.6-A3B locally on my Strix halo for testing simple prompts like "make the largest polygonal picture frames you can, each from a single 8 foot 2x4. Start with 3 sided and go up from there, lay them all out in a 9 by 9 grid."
It's a cheap (off-grid solar power, so ~free) way to sharpen the harness: if less-than-SOTA models can succeed then the tool surface is clear. Bumping up to 3.6 Pro has been a nice middle step closer to the frontier models without spending an arm and a leg on credits.
I had a similar experience recently while helping my 5 year old daughter vibe code a sandcastle-themed tower defense game (https://sandcastles.finley.lol).
I ended up thinking it might be easier to generate rigged models, animate them, and capture from an iso perspective, then do some kind of pixel art style transfer on the masked sprite sheet. Eventually I realized my kid didn't really care too much about the visuals so I didn't get too far with it.
Surely they are testing their optimizations against common benchmarks internally? I bet the "real world task" degradation is larger by some multiple than it appears when measured through a benchmark that is part of the target.
I've noticed this and thought about it as well, I have a few suspicions:
Theory 1: Some increasingly-large split of inference compute is moving over to serving the new model for internal users (or partners that are trialing the next models). This results in less compute but the same increasing demand for the previous model. Providers may respond by using quantizations or distillations, compressing k/v store, tweaking parameters, and/or changing system prompts to try to use fewer tokens.
Theory 2: Internal evals are obviously done using full strength models with internally-optimized system prompts. When models are shipped into production the system prompt will inherently need changes. Each time a problematic issue rises to the attention of the team, there is a solid chance it results in a new sentence or two added to the system prompt. These grow over time as bad shit happens with the model in the real world. But it doesn't even need to be a harmful case or bad bugged behavior of the model, even newer models with enhanced capabilities (e.g. mythos) may get protected against in prompts used in agent harnesses (CC) or as system prompts, resulting in a more and more complex system prompt. This has something like "cognitive burden" for the model, which diverges further and further from the eval.
I can see a market for virtual copies of incredibly unpopular CEOs, but I don't think Mark would like how people would likely choose to use these digital effigies.
For both of these scenarios, it seems to happen when the context limit is getting full and the context is summarized. I've found it usually works to respond with the right file, i.e. "great, let's apply those changes in @path/to/file", but it may also be a good time to return to an earlier conversation point by editing one of your previous messages. You might edit the message that got you the response with changes not linked to a specific file, including the file path in that prompt will usually get you back on track.
[0] Building a hubless bucky ball dome. It's a bit deceptively-simple, because the agent only has to call the procedure with the right values. That's an intended flywheel: agents and users can save procedures to a community library, and future agents can discover and invoke the pre-existing procedures, instead of re-deriving them. They can copy and edit the procedure as needed, so it basically becomes a RAG library for carpentry procedures written in the project's DSL.
[1] A sawhorse, re-implemented by Opus 4.8 from plans published by Home Depot. Now saved as a procedure so you can drop a sawhorse anywhere you need one.
[2] A small shed with stick frame walls and a lean-to roof.
[3] The Picture frame prompt mentioned in my first comment.
Quick note: I recently broke the smart de-duplication for step-by-step playback, so it will show every single cut, one by one. Hit "stop" and then click the final frame in the timeline to see just the finished state.
[0] https://sawdust.diy/share/b0de719c-0e9f-4f4f-9282-085c521163... (bucky ball dome)
[1] https://sawdust.diy/share/45557307-0f78-4b5b-bf0e-eae77a9853... (sawhorse)
[2] https://sawdust.diy/share/cbb591fc-5511-40d9-8ccd-0d9f8c10be... (shed)
[3] https://sawdust.diy/share/ba109216-8849-49e3-ae9c-5a15982d24... (polygonal frames)