I'm using the pi-mono coding agent (open source, free) without any extensions and very simple prompts. The 3.6 27B model (BF16, 250k context) uses 67GB VRAM on an RTX PRO 9000.
It's very capable on almost any coding task I've thrown at it, and very good for easy-to-medium hard scripts, new code bases.
It struggles on some complex tasks in larger code bases, e.g. using to debug and fix bugs in llama.cpp it gets close to working code but often introduces errors. For such tasks its still very useful as a search/explore tool and drafting fixes.
Something I've had good progress with using local models and simple open-source harnesses is to repeat, in a new context, simple verification prompts.
I'd run the following 5-10 times with one model, then again with a 2nd model.
"Verify the correctness and completeness of all security configs/rules in SETUP.md. Consider if anything is missing, and if anything is not needed. Do not modify any files; only write potential findings to report.txt"
"Verify all findings and claims in report.txt."
Replace "SETUP.md" with whatever you're working on.
It's both terrifying and incredible watching what the models get correct and what they get completely wrong.
However, after enough runs they tend to settle on a state they claim does not need any more edits. And that result is generally useful with much fewer errors/hallucinations compared to a single run.
Many crypto wallets use a key derivation function (KDF) to add an amount of computation (and memory usage) per password tried - to mitigate brute force of weak passwords.
The increase in compute (decrease in brute-force cost) combined with price increases in many crypto tokens means brute-forcing old wallets can become worth it years after passwords were forgotten.
And of course even smaller, local AI models can now easily write optimized scripts to brute-force any given KDF function.
One set of models run on 8GB VRAM / 16GB RAM and another set runs on 24GB VRAM / 64GB RAM. Both are very useful for easy and easy-to-moderate complex code, respectively.
The latest open, small models are incredibly useful even at smaller sizes when configured properly (quant size, sampling params, careful use of context etc).
Bit of a hype madhouse whenever a new model is released, but it's pretty easy to filter out simple hype from people showing reproducible experiments, specific configs for llama.cpp, github links etc.
This. And when possible, first asking the AI to add more granular logging around the code where the problem is - then re-run the code and feed the new log in a new context.
I've used this to debug some moderately complex bugs in golang and godot code and it works really well - the combo of having a new context with the (sometimes overly) granular debug logging and only the required, specific source code.
Keep it simple and run a fresh, new context for each prompt.
I use the pi-mono coding agent with several different new open models running locally.
The simpler and more precise the prompt the better it works. Some examples:
"Review all golang code files in this folder. Look for refactor opportunities that make the code simpler, shorter, easier to understand and easier to maintain, while not changing the logic, correctness or functionality of the code. Do not modify any code; only describe potential refactor changes."
After it lists a bunch of potential changes, it's then enough to write "Implement finding 4. XYZ" and sometimes add "Do not make any other changes" to keep the resulting agent actions focused.
Both are possible; increasing demand and bubble collapse.
The way this could happen is if model commoditization increases - e.g. some AI labs keep publishing large open models that increasingly close the gap to the closed frontier models.
Also, if consumer hardware keep getting better and models get so good that most people can get most of their usage satisfied by smaller models running on their laptop, they won't pay a ton for large frontier models.
That's a very good point! And an opportunity for game worlds; extrapolate those blind spots by assuming small planets and planets further out from their stars are more common than what's been confirmed so far.
Thanks for the link! Looks like pretty useful tools.
I'm playing with a space game idea of physics simulation somewhere between the fidelity of KSP and Eve Online. More robust ships and easier gameplay than KSP, but much more in-depth physics than Eve.
A bit too early (and too much AI slop code!) to share but can push to github if useful - wrote some scripts to parse the gaia DR3 release: https://gea.esac.esa.int/archive/
Parses all rows of the gdr3/Astrophysical_parameters/ files and filters out all objects within X ly of our solar system.
Same idea there as with the exoplanets; build a statistical distribution from real-world data and use it to generate fictional solar systems.
Tried a few new, open, local AI models by giving them the CSV file and asking them to write a simple python script:
1. Parse all rows and build statistical distribution of mass, radius etc.
2. Use those distributions to generate fictional exoplanets.
Playing with this for a space game idea where star systems are populated with fictional exoplanets, but all their params are from the real statistical distributions of all known exoplanets.
A way to get some harder sci-fi using real world data :)
It's very capable on almost any coding task I've thrown at it, and very good for easy-to-medium hard scripts, new code bases.
It struggles on some complex tasks in larger code bases, e.g. using to debug and fix bugs in llama.cpp it gets close to working code but often introduces errors. For such tasks its still very useful as a search/explore tool and drafting fixes.