dandinu

24 dagen geledendiscuss

Correction - "Obviously, this is probably a bad idea."

24 dagen geledendiscuss

That's a good question. It started off as pure technical curiosity (in the realm of: will this even work?). but as a side note, I always think of my screen time and the amount we spend chatting away on social media.

Obviously, this is probably and idea. I could imagine I connect my calendar to Hermes, and automate myself into 12 dinner plans and a trip to Disneyland on a Thursday afternoon just because I once mentioned I'm a Mickey fan to my nephew.

dandinu

24 dagen geledendiscuss

The method I used in the article is real and is pretty standard. Also, I do a decent amount of distillation and tinkering with weights for work, so I can assure you I did try that before resorting to good 'ol RAG.

Overall, even with a finetuning-as-a-serice like Tinker (the one from Thinking Machines) which is pretty cheap, the economics didn't work out that well.

Also, you probably one-shot this with Claude, I agree. But, you need to have an expensive Max subscription, which not everyone is willing to shell out 200 bucks for, just to have some weekend fun.

dandinu

24 dagen geledendiscuss

Author here. This is a personal weekend project that grew into a working WhatsApp bot. It replies as me to two allowlisted contacts (myself + one friend who knew about the experiment). The interesting part is not the agent framework but the retrieval I eventually got the best results with: every reply gets generated by Claude after pulling 8 of my real past replies to that specific contact, filtered by recency.

Built on Hermes Agent + Baileys + Chroma + nomic-embed-text-v2-moe + Claude Sonnet 4.6 via Azure AI Foundry. About 2 hours of work plus an hour debugging a WhatsApp multi-device LID issue. Total runtime cost: ~$0.005 per reply.

The bot is not running on a dedicated number. It is hooked to my primary WhatsApp, which is a ban risk I accepted in exchange for being able to test with real contacts. The killswitch (Telegram command that empties the allowlist and restarts the gateway) takes about 10 seconds. There is also a hard kill: unlink the device from WhatsApp on the phone, ~5 seconds, severs the bridge session entirely.

Happy to answer questions.

dandinu

8 maanden geledendiscuss

So the deal with AWE (Address Windowing Extensions) is that it lets 32-bit apps access memory above 4GB by essentially doing manual page mapping. You allocate physical pages, then map/unmap them into your 32-bit address space as needed. It's like having a tiny window you keep sliding around over a bigger picture.

The problemis that llama.cpp would need to be substantially rewritten to use it. We're talking:

  cAllocateUserPhysicalPages()
  MapUserPhysicalPages()
  // do your tensor ops on this chunk
  UnmapUserPhysicalPages()
  // slide the window, repeat

You'd basically be implementing your own memory manager that swaps chunks of the model weights in and out of your addressable space. It's not impossible, but it's a pretty gnarly undertaking for what amounts to "running AI on a museum piece."

dandinu

8 maanden geledendiscuss

Oh darn, you're absolutely right (pun intended) about the 32-bit situation. SSE2 is really the "floor" there if you want any kind of reasonable compatibility. I was being a bit optimistic with SSE4.2 even for 64-bit - technically safe for most chips from that era but definitely not all.

The Pentium Pro challenge though... pure x87 FPU inference? That would be gloriously cursed. You'd basically be doing matrix math like it's 1995. `-mfpmath=387` and pray.

I'm genuinely tempted to try this now. The build flags would be something like:

  -DGGML_AVX=OFF -DGGML_AVX2=OFF -DGGML_FMA=OFF \
  -DGGML_F16C=OFF -DGGML_SSE42=OFF -DGGML_SSSE3=OFF \
  -DGGML_SSE3=OFF -DGGML_SSE2=OFF  # pain begins here

And then adding `-ffast-math` to `CMAKE_C_FLAGS` because at that point, who cares about IEEE 754 compliance, we're running a transformer on hardware that predates Google.

If someone actually has a Pentium Pro lying around and wants to see Qwen-0.5B running on it... that would be the ultimate read for me as well.

Thanks for the kind words. Always fun to find fellow retro computing degenerates in the wild.

dandinu

8 maanden geledendiscuss

That's pretty accurate. I'm always amazed how much we move forward with technology, just to later realize we already had it 15 years ago.

regarding your question:

I have a 32bit XP version as well, and I actually started with that one.

The problem I was facing was that it's naturally limited to 4GB RAM, out of which only 3.1GB are usable (I wanted to run some beefier models and 64bit does not have the RAM limit).

Also, the 32bit OS kept freezing at random times, which was a very authentic Windows XP experience, now that I think about it. :)

dandinu

8 maanden geledendiscuss

there is a full technical write-up in the Github repo in "WINDOWS_XP.md": https://github.com/dandinu/llama.cpp/blob/master/WINDOWS_XP....

Sorry for failing to mention that.

Link to vcredist thread: https://github.com/LegacyUpdate/LegacyUpdate/issues/352

dandinu

8 maanden geledendiscuss

I just finished testing the Tinker API launched by Mira Murati's Thinking Machines and I can honestly say, even though it feels unfinished, it's pretty great. The setup is really smooth and with some minimal coding (and using their examples in the Tinker Cookbook) I was able to fine tune a Llama 3.1 8B Base on the Romanian language in under 20 min. The result was pretty decent and eventually I got it to write better poetry than the base model, in the target language.

I mostly did it because I got some free credits from them and I was curious what a company valued at 50 billion has to offer. The product overall feels like it's half-baked since there is no real interface, but the API does a lot of the heavy lifting in the backend while maintaining this local development feeling, which I personally find pretty cool.

I put the code up on Github, if anyone is interested, but I am curious what y'all think about their approach top fine tuning.

Submissions

I built a WhatsApp clone of myself; a friend agreed on a password with it

Show HN: AlphaGenome Dashboard – Analyze Your Genome with DeepMind AlphaGenome

Show HN: I cross-compiled llama.cpp to run on Windows XP

The new Tinker API from Thinking Machines is half-baked but great at fine tuning

comments