theoli · HackerTrans

19 dagen geledendiscuss

Exactly this, it’s the loops. The first 50k tokens of a task is by far the most valuable. But when left to run independently, the agent will consume millions of tokens of error messages from running tests and discovering a minor syntax error, a missing import, a method call with incorrect parameters, etc. Then it will write some helper program while debugging the main task and get into the same loop debugging minor errors in the helper. From my experience, the vast majority of tokens consumed by Claude Code on totally independent tasks are consumed fixing minor mistakes it just made.

theoli

3 maanden geledendiscuss

There was this ROCm bug I was watching for awhile: https://github.com/ROCm/ROCm/issues/5706 - This is about the GPU clock remaining at max frequency, but that can drive the fan speed to increase.

It doesn't happen with Vulkan backends, so that is what I have been using for my two dual R9700 hosts.

EDIT: The bug is closed but there were mentions of the issue still occurring after closure, so who knows if it is really fixed yet.

theoli

3 maanden geledendiscuss

I have a dual R9700 machine, with both cards on PCIe gen4 x8 slots. The 256bit GDDR6 memory bandwidth is the main limiting factor and makes dense models above 9b fairly slow.

The model that is currently loaded full time for all workloads on this machine is Unsloth's Q3_K_M quant of Qwen 3.5 122b, which has 10b active parameters. With almost no context usage it will generate 59 tok/sec. At 10,000 input tokens it will prefill at about 1500 tok/sec and generate at 51 tok/sec. At 110,000 input tokens it will prefill at about 950 tok/sec and generate at 30 tok/sec.

Smaller MoE models with 3b active will push 70 tok/sec at 10,000 context. Dense models like Qwen 3.5 27b and Devstral Small 2 at 24b will only generate at around 13 - 15 tok/sec with 10,000 context.

This is all on llama.cpp with the Vulkan backend. I didn't get to far in testing / using anything that requires ROCm because there is an outstanding ROCm bug where the GPU clock stays at 100% (and drawing like 60 watts) even when the model is not processing anything. The issue is now closed but multiple commenters indicate it is still a problem. Using the Vulkan backend my per-card idle draw is between 1 and 2 watts with the display outputs shut down and no kernel frame buffer.

theoli

7 maanden geledendiscuss

I am doing this with good success parsing receipts with ministral3:14b. The first prompt describes the data being sought, and asks for it to be put at the end of the response. The format tends to vary between json, bulleted lists, and name: value pairs. I was never able to find a good way to get just JSON.

The second pass is configured for structured output via guided decoding, and is asked to just put the field values from the analyzer's response into JSON fitting a specified schema.

I have processed several hundred receipts this way with very high accuracy; 99.7% of extracted fields are correct. Unfortunately it still needs human review because I can't seem to get a VLM to see the errors in the very few examples that have errors. But this setup does save a lot of time.

theoli

7 maanden geledendiscuss

Epoch shift with 48-bit timestamp that has >12,000 years of range to get another 50 years of range is an amusing choice.

comments