c0rruptbytes

kemarin duludiscuss

i think to fall in love with Pi, bundled skills are a bit antithetical - you realistically only need a couple of skills that you maybe design yourself

c0rruptbytes

3 hari yang laludiscuss

inference is profitable, these companies are in the red because they're paying a premium to get the compute now versus later (because compute is the only moat when open models are catching up)

we're literally looking at insane margins over compute, as energy gets cheaper, margins get wider - china focusing on cheap solar is probably going to be a key reason why their AI is so much cheaper

c0rruptbytes

10 hari yang laludiscuss

OpenAI prioritized compute much much earlier on, so they’re probably just more able to provide the model while Anthropic seems to be busting out the seams

Sonnet 5 today was incredibly slow for example

c0rruptbytes

14 hari yang laludiscuss

Deepseek isn't philanthropy, it's a hedgefund trying to short the western AI market by saying "hey we can do 90% of they can (arguably better at a density metric) for a 1/10th of the cost"

it's my theory at least, the Hindenburg Research of AI

c0rruptbytes

16 hari yang laludiscuss

[dead]

c0rruptbytes

16 hari yang laludiscuss

if they’re paying for the tokens, what’s the problem

c0rruptbytes

19 hari yang laludiscuss

i'm in, i think prices are gonna suck anyway, i own a playstation and that shit sucks, i want to do more couch co-op with my partner and the steam library opens up so much indie games

can i build a mini pc myself? probably but meh

c0rruptbytes

19 hari yang laludiscuss

TV is heavily subsidized from data collection and ads, not sure it's a perfect comparison

c0rruptbytes

19 hari yang laludiscuss

https://blog.kilo.ai/p/did-claude-opus-48-distill-alibabas

it happens to all models…when the internet is increasingly generated, things happen

c0rruptbytes

22 hari yang laludiscuss

MCP tool search fixes the major issue imo, MCP clears skills/clis in every other way

c0rruptbytes

25 hari yang laludiscuss

I would try a 6-bit MoE and maybe with unsloth's studio, they claim to have auto tool fixing which is where i see a lot of issues with MoEs

I'm on a 48gb M5 Pro right now and it's been okay, a lot of my rough experiences have been with MLX and I'm finding that GGUFs are okay now

c0rruptbytes

25 hari yang laludiscuss

large contexts degrade the performance - attention doesn't work will for large windows like that and cloud models are kind of hacking it

local models do involve some context engineering to get it okay, but it's not that rough

c0rruptbytes

25 hari yang laludiscuss

q4 isn't rubbish, but it's a compromise for a good value, q6 is essentially a no-compromise quantization and it's what i recommend for MoEs in my experience for agentic workflows

c0rruptbytes

25 hari yang laludiscuss

I'm talking about the common use case that I think hacker news people have:

you get a macbook for work, you run the macbook

they're not going to start giving GPUs to employees to run local models

c0rruptbytes

25 hari yang laludiscuss

I don't know about good, I use a lot of local models and they're still pretty painful to run locally

You have dense models (qwen 27b, gemma 31b) who are pretty smart, but pretty slow

You have MoE models (gemma 26b, qwen 35b, north mini code 30b) who are pretty fast, but make a lot of mistakes

You need a lot of memory to run these well, quantization makes tool calling weaker, so most run at 4 bit quants and are wondering why it kinda sucks and that's because you've essentially lobotomized the model (I recommend unsloth quants, i recommend 6bit for MoEs and 5bit for dense)

So you need a lot of compute to make the pre-fill fast, you need bandwidth to make the decode fast, you need a lot of memory to hold everything - lot of ifs

On top of that, your laptop becomes a loud hot churning machine, it's uncomfortable to work with.

So are they good? not really. Do they work? yes

edit: just wanna clarify - i think open models are the future, i think they're super important, i'm contributing constantly to the ecosystem - i think people should play around with these models, i think people should use `pi` and learn how it all works - but don't download a model expecting it to be good out of the box, you will have to tune and configure a lot of stuff to replace a "coding agent" that most people are using models for

c0rruptbytes

28 hari yang laludiscuss

Minimax M3 too, and huawei claims to be releasing non-nvidia dependent training software too. openPangu 2.0 could be a shake-up if it holds up as a good model

China may not care about open source, but they know they will personally fund AI through government investments while US relies on private investments, best way to scare private investments is a free capable alternative for everyone

Add on the fact that they actually invested in energy infrastructure and can offer AI very cheap to their citizens and you can get a population well versed in AI to reduce menial tasks and focus on more productive things (if we're to believe the claims of the technology)

c0rruptbytes

28 hari yang laludiscuss

i’m running m4 pro 48gb right now

omlx + gemma 12b 6 bit + pi

it’s feasible for sure

MoEs for speed (qwen 35b, cohere 30b, gemma 26b)

Dense for more methodical work (qwen 27b [reigning champ], gemma 31b, gemma 12b)

MoE i recommend 5bit+

Dense i think 4 bit is okay

Play with your context size, you don’t really need that much, have lazy loading for tools and mcps

my pi extensions for anyone looking for a skinny quick setup, i have use `--no-skills` right now too:

    "npm:pi-codex-goal",
    "npm:pi-simplify",
    "npm:pi-mcp-adapter",
    "git:github.com/elpapi42/pi-minimal-subagent",
    "npm:@wierdbytes/pi-statusline",
    "npm:@aliou/pi-guardrails",
    "npm:pi-lens",
    "npm:@juicesharp/rpiv-todo",
    "npm:pi-hashline-readmap",
    "npm:@mrclrchtr/supi-review",
    "npm:pi-cmux",
    "npm:@mrclrchtr/supi-context",
    "npm:pi-tool-search"

think of local models as "zero sugar" models and that's where we're at right now. I think it's crazy how good these models are compared to last year's frontier models

c0rruptbytes

bulan laludiscuss

it does not result in great results left unattended, it’ll start creating slop or hardcoding solutions

but overtime if you adjust your verification rubric, it’s not too bad, gets pretty good, if you do make it do TDD, it gets kinda crazy and you’ll have 2000-3000 tests after awhile, or on my common case, 6000-7000 lines of code in single files (i usually have a cron to audit files for decomposition and create tickets)

i wouldn’t use it at my job yet, but it’s been fun to use for personal projects - it’s like modded minecraft automation or factorio

c0rruptbytes

bulan laludiscuss

There's `honcho` for memory, i'm starting to play with it now, but I feel like I've seen a lot of projects pop up for it

c0rruptbytes

bulan laludiscuss

I like Zed...

but AI dev workflows get complicated fast

you start with claude code or codex and it's cute, but then you realize - hmm configuration is cheap, the AI can do it!

then you start looking into MCPs and skills, fuck it, oh-my-pi looks awesome!

wait a second? I can just have AI make my own personal AI harness! Next thing you know, you're writing the 5th version of "little-coder" or similar using the Pi library

ahh shit, you just read an article that `tools` are actually crazy important for AIs, using `sed` is dumb when `hashline` + ASTs are way better, lets just start writing our own tools!!

...anyway I just use Zed, simple agent on the left, code on the right

i have some pretty complicated automated workflows that use `linear` + a orchestrator -> implementer -> reviewer -> releaser workflow, but it's less a dev stack and an AI factory

comments