benob · HackerTrans

21 dagen geledendiscuss

Longest definition and semi-columns are strong biases for right answer. Also, my run contained a lot of adjectives for which it is pretty obvious that noun definitions do not match.

benob

vorige maanddiscuss

It may be a clever move. By using the same models as android (contractually?), they can compete on the user experience which they typically handle better than android phone providers.

benob

vorige maanddiscuss

And papers on bias amplification in ML predate LLMs. I remember this specific one which was a spotlight paper at EMNLP:

Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraints, Zhao et al.

https://arxiv.org/abs/1707.09457

benob

2 maanden geledendiscuss

Does changing the date fix it?

benob

2 maanden geledendiscuss

Deployed it to a huggingface space: https://huggingface.co/spaces/benoitfavre/needle-playground

You can check the very simple docker file there.

benob

3 maanden geledendiscuss

Here is llama-bench on the same M4:

  | model                    |       size |     params | backend    | threads |            test |                  t/s |
  | ------------------------ | ---------: | ---------: | ---------- | ------: | --------------: | -------------------: |
  | qwen35 27B Q4_K_M        |  15.65 GiB |    26.90 B | BLAS,MTL   |       4 |           pp512 |         61.31 ± 0.79 |
  | qwen35 27B Q4_K_M        |  15.65 GiB |    26.90 B | BLAS,MTL   |       4 |           tg128 |          5.52 ± 0.08 |
  | qwen35moe 35B.A3B Q3_K_M |  15.45 GiB |    34.66 B | BLAS,MTL   |       4 |           pp512 |        385.54 ± 2.70 |
  | qwen35moe 35B.A3B Q3_K_M |  15.45 GiB |    34.66 B | BLAS,MTL   |       4 |           tg128 |         26.75 ± 0.02 |

So ~60 for prefill and ~5 for output on 27B and about 5x on 35B-A3B.

benob

3 maanden geledendiscuss

I get ~5 tokens/s on an M4 with 32G of RAM, using:

  llama-server \
   -hf unsloth/Qwen3.6-27B-GGUF:Q4_K_M \
   --no-mmproj \
   --fit on \
   -np 1 \
   -c 65536 \
   --cache-ram 4096 -ctxcp 2 \
   --jinja \
   --temp 0.6 \
   --top-p 0.95 \
   --top-k 20 \
   --min-p 0.0 \
   --presence-penalty 0.0 \
   --repeat-penalty 1.0 \
   --reasoning on \
   --chat-template-kwargs '{"preserve_thinking": true}'

35B-A3B model is at ~25 t/s. For comparison, on an A100 (~RTX 3090 with more memory) they fare respectively at 41 t/s and 97 t/s.

I haven't tested the 27B model yet, but 35B-A3B often gets off rails after 15k-20k tokens of context. You can have it to do basic things reliably, but certainly not at the level of "frontier" models.

benob

3 maanden geledendiscuss

I miss the comment tagging system: insightful, informative, interesting, funny. It would make sense for hn.

benob

3 maanden geledendiscuss

Space station tracking: https://flight-viz.com/cockpit.html?lat=40.64&lon=-73.78&alt...

benob

3 maanden geledendiscuss

I just realized that a hash function is nothing less than the output of a deterministic random number generator xored with some data

benob

3 maanden geledendiscuss

No, the failure is the human written prompt

benob

3 maanden geledendiscuss

The author emphasizes accessibility and coherence as a benefit but another interesting one is composability which does not emerge naturally in the world of UI. Create a UI for a pair of websites like a command line for grep and wc. LLMs already provide that but under the natural language interaction primitive. UI could allow for branded experiences, ad delivery and whatnot in ways that natural language doesn't.

benob

3 maanden geledendiscuss

"That allows us to license the open source project under the more permissive MIT license."

benob

3 maanden geledendiscuss

I would say:

- decomposition: discover a more general form of Fourrier transform to untangle the underlying factors

- memorization: some patterns are recurrent in many domains such as power low

- multitask: exploit cross-domain connections such as weather vs electricity

benob

3 maanden geledendiscuss

Ollama is a user-friendly UI for LLM inference. It is powered by llama.cpp (or a fork of it) which is more power-user oriented and requires command-line wrangling. GGML is the math library behind llama.cpp and GGUF is the associated file format used for storing LLM weights.

benob

4 maanden geledendiscuss

Maybe they quantized a bit too much the model parameters...

benob

4 maanden geledendiscuss

This is the worst lay-people explanation of an AI component I have seen in a long time. It doesn't even seem AI generated.

benob

4 maanden geledendiscuss

This reminds me of Intel talking about faster web browsing with the new Pentium

benob

4 maanden geledendiscuss

The real question is when will you resort to bots for rejecting low-quality PRs, and when will contributing bots generate prompt injections to fool your bots into merging their PRs?

benob

4 maanden geledendiscuss

Reminds me of "Universal pre-training by iterated random computation" https://arxiv.org/pdf/2506.20057, with bit less formal approach.

I wonder if there is a closed-form solution for those kinds of initialization methods (call them pre-training if you wish). A solution that would allow attention heads to detect a variety of diverse patterns, yet more structured than random init.

comments