ponyous

Interesting way to show off a model last on every benchmark. Not sure any other lab is doing this

15 hari yang laludiscuss

[flagged]

23 hari yang laludiscuss

As someone who just had success with equivalent system in EU my recommendation is to get someone who's done it before to do it for you. I hired an agency. They took 8% fee, which is pretty low, usually it's between 10% and 15%.

ponyous

24 hari yang laludiscuss

Absolutely. Running it now, will update this comment in about 30 mins.

Edit: Surprisingly very good results with 3.0 flash with high thinking.

Cost: $0.06

Duration: 3.22 min

Code Errors: 1.3 per attempts (meaning on average it had to retry 1.3 times)

Adherence was on par with 3.5 flash Low thinking

ponyous

24 hari yang laludiscuss

I don't have the eval results live yet, so I cannot share them yet.

I was benchmarking using a soon to be released new version of my AI CAD modeling software[0]. It's basically an agent that has access to tools that can execute build123d scripts, get sculpted models, blender to combine sculpts + parametric models, tools to inspect the model (visually and with code), search datasheets, ...

I tried what you recommend a while ago (asking an AI to evaluate using different angles) and the AI evaluations were extremely bad - barely any correlation to what I scored. Things have gotten better, but I don't trust it enough yet.

Here is how I score adherence (and how AI did as well, but I tried methods where it would just give back a boolean "pass" or not):

    <0.2 → Poor – Misses core intent; largely irrelevant or incorrect.
    <0.4 → Weak – Partially relevant; significant omissions or errors.
    <0.6 → Fair – Covers main points but lacks completeness or precision.
    <0.8 → Good – Mostly accurate; minor gaps or deviations.
    <=1.0 → Excellent – Fully aligned; precise, comprehensive, and faithful to intent.

Here is the scenario list (prompts are much more detailed):

    dragon-bottle-stopper
    editing-param-mid-conv
    editing-parametric-enclosure
    editing-swap-material-param
    editing-text-edit-cube
    multi-turn-bird-house
    multi-turn-dice-tower
    multi-turn-modular-planter
    multi-turn-phone-stand
    multi-turn-shelf
    one-shot-bookend
    one-shot-cable-clip
    one-shot-chess-queen
    one-shot-coaster
    one-shot-coffee-cup
    one-shot-dog-tag
    one-shot-dragon-figurine
    one-shot-hex-bracket
    one-shot-keychain-fob
    one-shot-low-poly-tree
    one-shot-pegboard-hook
    one-shot-pi4-case
    one-shot-threaded-jar

[0]: https://grandpacad.com

ponyous

24 hari yang laludiscuss

Just ran and scored 63 3d model generations (via code) across high and no reasoning. 3D Modeling benchmark quickly shows spatial, logic and code performance of the model so I think it's a very good indicator of the quality.

Here are the results compared to Gemini 3.5 Flash:

    Model + config          CodeErr/gen   Cost/gen   Median time   Quality
    gemini-3.5-flash, low      0.71        $0.18        68s       baseline
    GLM 5.2, reasoning high    0.61        $0.18       289s         -6.0%
    GLM 5.2, reasoning off     1.52        $0.10       126s        -13.6%

Although it is cheaper, it is significantly slower, and results are worse overall. Surprisingly - high reasoning produces less code errors than gemini 3.5 flash, but when I actually look at the models they are worse.

Edit: I recently ran evals with Kimi 2.7 and MiniMax-M3 and this is clearly open source SOTA model, by far.

ponyous

bulan laludiscuss

Basically double from Opus 4.8 IIRC

ponyous

bulan laludiscuss

> Surprisingly written by a human :)

Article ends with this

ponyous

2 bulan yang laludiscuss

Bun is(was?) a lot about performance. How does it compare to zig?

ponyous

2 bulan yang laludiscuss

I've run a tons of benchmarks for OpenSCAD for all kinds of models and setups, and what I realised is:

- Models are very jagged (might excel in one type of 3d model, but not another)

- Gemini models are the least jagged in my experience and have the best image understanding

- Gemini models are also the most creative (which may be undesirable if you want precise CAD part)

- Overall this benchmark doesn't prove much because one 3d model (and one attempt) is just not enough. I am usually testing on at least a dozen models each generated 3 times, but should really do much more, but it's too pricey for a solo dev.

Still, thanks for publishing this. Will be definitely run flash 3.5 soon to see how it performs.

ponyous

2 bulan yang laludiscuss

I've seen this and other attempts like this[0] while exploring improvements for my CAD AI[1]. And I think these are potentially powerful solutions, but none of the current projects/weights have enough training (data or time) to make it work on arbitrary models. MeshCoder pretty much works only on models based of their training data. I haven't tried GenCAD but other commenters have confirmed my suspicion.

[0]: https://daibingquan.github.io/MeshCoder/

[1]: https://grandpacad.com

ponyous

2 bulan yang laludiscuss

Seems like Chinese labs are the only ones that are trustworthy (at least when it gets to this specific issue). This feels so ironic haha

ponyous

2 bulan yang laludiscuss

The site is currently unavailable 503 so I can't read it. But I wonder, what should you consent to? Every dependency? Every dependency above 1GB?

ponyous

2 bulan yang laludiscuss

Kimi is nowhere near GPT or Opus unfortunately. I really wish it was. I’m running evals where models have to generate code that produces 3D models and it’s obvious that it lacks spatial understanding and makes many more code errors before it succeeds.

Maybe it’s better in one particular case here and there and I think this blog post is example of that.

ponyous

2 bulan yang laludiscuss

Gregor! Not sure I want to say more on here.

ponyous

2 bulan yang laludiscuss

Been following you guys a while, seems like you've been gaining some traction recently, lets goo and congrats!

I have been working on GrandpaCAD[0] for a while, a very similar product. I thought of you as my biggest competitors but noticed recently you are focusing more and more on professionals while I am focusing on total noobs in modeling who just want to whip out a quick model. So I guess we are not competitors anymore?

My evals[1] show that Opus 4.7 and GPT 5.5 are very comparable in terms of generation quality, but GPT 5.5 is slower and costs sooo much more in my harness. And the original breakthrough model was Gemini 3.1. I'm curious do you have more written about your benchmarks setup?

If you want to chat email is in my profile. Btw, just met "your"(?) neighbour on a plane a couple of days ago. World is small.

[0]: https://grandpacad.com

[1]: https://grandpacad.com/en/blog/public-benchmarks-misled-me-o...

ponyous

3 bulan yang laludiscuss

Another library I have to integrate and benchmark against OpenSCAD for my AI SaaS[0]. I am really curious how constructive solid geometry compares to sketching and extruding that CadQuery is build on.

Anyone curious in the writeup? I have a pretty good harness for evaluating 3d generation performance.

[0]: https://grandpacad.com

ponyous

3 bulan yang laludiscuss

Why does M1 Max project significantly higher revenue than M3 Max with double the ram?

ponyous

3 bulan yang laludiscuss

http://grandpacad.com - 3D Modeling tool intended for creating 3D prints. AI based. Allows for dimensionally accurate parts as well as organic shapes.

I massively improve it every month. Pretty proud of it.

ponyous

3 bulan yang laludiscuss

As far as I know reddit does the same thing. If you don’t follow normal human patterns you will quickly get banned. I read some guy’s report where he got banned 8 times detailing each behaviour and indeed it was nothing crazy.

Submissions

Comparing State of the Art LLMs for 3D Generation

comments