bjackman · HackerLangs

Even if nobody is "cheating" your particular definition of cheating, the benchmarks are _somewhere_ in the super-structural gradient descent. Models are benchmark-maximising machines at some level, so I think the benchmarks are inherently a bit useless.

This is not really surprising, benchmarking _people_ doesn't work. You can only get a decent measure of someone's coding abilities by personally interacting with them. Given that models are basically person simulators it would be weird if benchmarks kept being useful as the simulation got more accurate.

I think what I've just said is basically just a more roundabout way of what you said: "Goodhart's law at work". It really is a law.

bjackman

5 days agodiscuss

Great to see there are others doing the same.

It's a extremely cool that ESPHome is able, just by existing and being good, to create this little industry of no-bullshit products. What an awesome project that is!

bjackman

5 days agodiscuss

Yeah I was thinking airtightness might be the difference. My flat seems to be bizarrely hermetic (when you turn on the kitchen extractor fan, it struggles if you don't have a window open somewhere).

So maybe a few leaky cracks are enough that when you open a window you get a bit of a through-draft.

bjackman

6 days agodiscuss

That's interesting coz I found the opposite, at my place to keep the level below 1k I usually have to open a window in the room I'm in, or use a fan.

I live on a noisy street so I don't usually want to do that, if I open a window at the back and keep internal doors open it will stay reasonable but significantly elevated.

So yeah I think the lesson here is you probably need to buy a sensor, different homes are gonna differ.

My home is quite small (probably 80m²) and has literally zero ventilation built in (even in the bathroom!). I live in Switzerland where it's traditional to actively ventilate your home twice a day. But that doesn't do anything for CO2. Also it's such a fucking waste of time lol. Looking forward to moving into a modern building.

bjackman

7 days agodiscuss

IMO it's something where an intervention is often cheap enough that it's worth it even without great evidence.

But also bear in mind that regardless of "are we operating at max effectiveness", OSHA sets a legal limit of 5000ppm in a workplace, and that's about _safety_.

This article is talking about keeping levels below 1000 which is a very high standard IMO (still arguably justified given the studies mentioned). But if you are in a poorly ventilated home office you could easily hit 3000. At that point you are closer to "illegal in the US" than "earth's atmosphere".

So yeah even if you are unconvinced about micro-optimising your CO2 levels there's a very long established argument in favour of at least paying _some_ attention to it.

bjackman

7 days agodiscuss

As a middle ground I can also recommend this unit: https://apolloautomation.com/products/air-1

Looks like it's increased in price unfortunately but I like the idea, it's basically just what you would do as a DIY project but ready built. So you can either use it like a normal commercial product, or you can just fork the ESPHome config that's on GitHub and flash it exactly like any normal ESPHome project.

bjackman

7 days agodiscuss

And I believe the accuracy is also not great on these cheap ones. The product in the OP's photo costs $200 where I live! And ISTR finding the sensor itself contributes a lot to this cost.

IIUC they also need fans. The one I have in my home has one that's actually integrated into the sensor unit.

bjackman

8 days agodiscuss

Ah yeah I see. I guess the fallback there would be to split the service up into a remote encrypted storage layer that goes on the VPS and then host the actual service (with the decryption keys) locally?

But ISTR reading Immich kinda assumes the storage is on a plain local filesystem so you get perf issues if you do something clever under its feet. Could be out of date on that.

bjackman

8 days agodiscuss

I really don't think you want E2EE for this. I host storage for family and friends, I haven't set Immich up yet (don't think I'd have space for everyone's photos) but the choice is between:

1. "Hey just so you know, I have access to everything you upload here".

2. "Do NOT lose your password or your data will be GONE FOREVER and I CANNOT get it back".

I definitely prefer 1 and I'm sure my users do too. They shouldn't upload it if they didn't trust me anyway.

In my case I follow it up with "and I might actually go digging around in your files if I need to debug something or you're wasting disk space". But I think you could also follow it up with "but I do promise not to look" and that would be valid too.

This whole thing only makes sense for people you're pretty close to.

(I do tell people not to back up their password managers on my system though).

I guess maybe for Immich specifically it would be nice to have a "vault" feature where people can upload nudes etc where they are willing to trade risk of loss for privacy on a per-photo basis.

bjackman

9 days agodiscuss

No? Some subsets of the world has papyrus I guess but for most people during most of that time people were pressing text into clay and wax and stuff, it must have fucking sucked.

Then we got paper and pens and that was a pretty decent interim for a short period. Then about 100 years ago we got typing. Then about 20 years ago we reached a world where almost everyone is better at typing than they are at writing.

Obviously it's still important that people can write by hand, but expecting people to do it for more than a few hundred words at a time is idiotic. Would you like your clothes to be hand sewn too? That also worked for thousands of years (much longer than writing) but we stopped doing it for a very good reason.

bjackman

9 days agodiscuss

Horse carts are literally designed to transport people but they aren't very good at it compared to cars

bjackman

12 days agodiscuss

Well, how many times in that 9 years have you written on paper for 2 hours straight? Even as a kid who did it regularly, it sucked.

Doing it now I really don't think I could deliver my intellectual best while worrying about if anyone can read my handwriting and whether I'm gonna cramp up by the end of the exam.

Pen and paper is just not a very good way to produce text.

bjackman

12 days agodiscuss

... Including a university. Literally everywhere you work as a student is serenaded by keyboard sounds.

bjackman

16 days agodiscuss

As a $BIGCORP member I don't think this would be a great solution. I suspect there are plenty of vibe coding PR spammers that work for my company. And the admins of the GitHub org would not really care, making it easy for staff to contribute to third party projects is nowhere near their top priority (and policing the behaviour of their org members outside of org-owned repos is not in their mandate even if they wanted to).

bjackman

18 days agodiscuss

You are not evaluating those questions, you are evaluating the probability of that two things happening, and you need to evaluate it better than the other people to win.

There are no easy questions, the difficulty is set by the skill/investment level of your competition.

bjackman

20 days agodiscuss

[dead]

bjackman

20 days agodiscuss

Nothing major just a few little details:

- the /artifact thing is quite useful (don't think CC has it?)

- the /tasks is a bit better than CC's equivalent

- there are a few built-in skills that I haven't found CC equivalents for in the built in set (but the fact that I haven't sought out 3rd party versions shows you they aren't very important).

And more generally it does a better job of making the agent available. When Claude is debugging something complex and running a bunch of experiments it's often unavailable for like 20 minutes at a time, you only have /btw. Whereas AGY tends to more aggressively use timers and background jobs.

But now I wrote that out, I realised it's probably just as much of a system prompt thing as a harness design thing. Coz Claude _can_ operate that way too.

Anyway, like I said none of these come anywhere near balancing out the model quality gap.

bjackman

21 days agodiscuss

I'm a Brit living abroad, when I visit the UK I use a Tailscale network with an exit node at my home, and yeah this always seems to work for me.

Going the other way around to try and watch British TV I used to find with a normal hosted VPN services could still figure out I wasn't in the country, but now I have a Tailscale exit node at my mum's place in the UK it always works fine.

So I suspect it all comes down to the IP source, probably a residential IP is the best possible case and with commercial VPNs it depends on how hard they work on isolating their IP blocks from known datacentres.

bjackman

21 days agodiscuss

You don't save memory with code, you save it with architecture, platform decisions, and feedback loops.

Feedback loops are the important bit. If you want to reduce your service's memory footprint, don't at the code look at the memory profile and monitoring. You will find something like "oh shit 30% of our RAM is used by these buffers that we could basically eliminate if we tweak the flush frequency".

If you automate/regularise those investigations you will get an efficient service.

Same is true of every other performance metric, and reliability. It comes from your engineering processes (alerts, qualification, prod experimentation) not "write better code".

bjackman

21 days agodiscuss

For my personal use, I really fucking want a humanoid robot, coz my home and all the bullshit in it was built for humanoids, I want a robot to do the bullshit for me. I don't want to move into a new, robot-oriented home.

I've never been to a factory but I bet there's a lot of the same bullshit. Ditto in a mine.

On the other hand, I've been in a datacentre. I don't see much need for a humanoid form in there, everything is flat and predictable. Why don't we have robot DC techs? This is probably an interesting clue re the next 10 years of robotics and maybe the reason Boston Dynamics is only valued at $1.1B.

Seems we might still be pretty limited on usecases. Maybe a dexterity bottleneck.

comments