isaacimagine

5개월 전discuss

I read this as "a shortage of tensors" and was curious as to the latest chicanery.

But yeah tenors right -- matches my experience.

isaacimagine

작년discuss

DT's reward-to-go vs. QL's Bellman incl. discount, not choice of architecture for policy. You could also do DTs with RNNs (though own problems w/ memory).

Apologies if we're talking past one another.

isaacimagine

작년discuss

TFP: https://arxiv.org/abs/2506.04168

The DT citation [10] is used on a single line, in a paragraph listing prior work, as an "and more". Another paper that uses DTs [53] is also cited in a similar way. The authors do not test or discuss DTs.

> hardness of the credit assignment ... data sparsity.

That is true, but not the point I'm making. "Bypassing credit assignment", in the context of long-horizon task modeling, is a statement about using attention to allocate long-horizon reward without horizon-reducing discount, not architecture choice.

To expand: if I have an environment with a key that unlocks a door thousands of steps later, Q-Learning may not propagate the reward signal from opening the door to the moment of picking up the key, because of the discount of future reward terms over a long horizon. A decision transformer, however, can attend to the moment of picking up the key while opening the door, which bypasses the problem of establishing this long-horizon causal connection.

(Of course, attention cannot assign reward if the moment the key was picked up is beyond the extent of the context window.)

isaacimagine

작년discuss

No mention of Decision Transformers or Trajectory Transformers? Both are offline approaches that tend to do very well at long-horizon tasks, as they bypass the credit assignment problem by virtue of having an attention mechanism.

Most RL researchers consider these approaches not to be "real RL", as they can't assign credit outside the context window, and therefore can't learn infinite-horizon tasks. With 1m+ context windows, perhaps this is less of an issue in practice? Curious to hear thoughts.

DT: https://arxiv.org/abs/2106.01345

TT: https://arxiv.org/abs/2106.02039

isaacimagine

작년discuss

The article is about using nalgebra to create an intuitive library for tranforming between Earth's various coordinate systems. Not "another Rust matrix library".

isaacimagine

작년discuss

Very cool, thank you for sharing!

isaacimagine

작년discuss

z3 has good python bindings, which I've messed around with before. My manual solution uses 42 gates, I would be interested to see how close to being optimal it is. I didn't ask the compiler to vectorize anything, doing that explicitly might yield a better speedup.

Re:neurosymbolics, I'm sympathetic to wake-sleep program synthesis and that branch of research; in a draft of this blog post, I had an aside about the possibility of extracting circuits and reusing them, and another about the possibility of doing student-teacher training to replace stable subnets of standard e.g. dense relu networks with optimized DLGNs during training, to free up parameters for other things.

isaacimagine

작년discuss

Thank you, I'm glad you enjoyed it!

Well, I'm not an expert. I think that this research direction is very cool. I think that, at the limit, for some (but not all!) applications, we'll be training over the raw instructions available to the hardware, or perhaps even the hardware itself. Maybe something as in this short story[0]:

> A descendant of AutoML-Zero, “HQU” starts with raw GPU primitives like matrix multiplication, and it directly outputs binary blobs. These blobs are then executed in a wide family of simulated games, each randomized, and the HQU outer loop evolved to increase reward.

I also think that different applications will require different architectures and tools, much like how you don't write systems software in Lua, nor script games mods with Zsh. It's fun to speculate, but who knows.

[0]: https://gwern.net/fiction/clippy

isaacimagine

작년discuss

+10 respect, thank you <3

isaacimagine

작년discuss

There are 163 lines of C. Of them, with -O3, 104 lines are present in the assembly output. So the C compiler is able to eliminate an additional ~36.2% of the instructions. It doesn't do anything fancy, like autovectorization.

I profiled just now:

          | instrs (aarch64) | time 100k (s) | conway samples (%) | 
    | -O0 |              606 |        19.10s |             78.50% |
    | -O3 |              135 |          3.45 |             90.52% |

The 3.45s surprises me, because it's faster than the 4.09s I measured earlier. Maybe I had a P core vs an E core. For -O0, the compiler is emitting machine code like:

    0000000100002d6c ldr x8, [sp, #0x4a0]
    0000000100002d70 ldr x9, [sp, #0x488]
    0000000100002d74 orn x8, x8, x9
    0000000100002d78 str x8, [sp, #0x470]

Which is comically bad. If I try with e.g. -Og, I get the same disassembly as -O3. Even -01 gives me the same disassembly as -O3. The assembly (-0g, -01, -03) looks like a pretty direct translation of the C. Better, but also nothing crazy (e.g. no autovectorization):

    0000000100003744 orr x3, x3, x10
    0000000100003748 orn x1, x1, x9
    000000010000374c and x1, x3, x1
    0000000100003750 orr x3, x8, x17

Looking more closely, there's actually surprisingly little register spilling.

I think the real question you're asking is, as I wrote:

> If we assume instruction latency is 1 cycle, we should expect 2,590 fps. But we measure a number nearly 10× higher! What gives?

Part of this is due to counting the instructions in the dissassembly wrong. In the blogpost I used 349 instructions, going off Godbolt, but in reality it's 135. If I redo the calculations with this new numbers, I get 2.11 instructions per bit, 0.553 million instrs per step, dividing out 3.70 gcycles/s gives 6,690 fps. Which is better than 2,590 fps, but still 3.6x slower than 24,400. But I think 3.6x is a factor you can chalk up to instruction-level parallelism,.

Hope that answers your questions. Love your writing Gwern.

isaacimagine

작년discuss

Glad you enjoyed it, and thanks for the tip!

isaacimagine

작년discuss

Thank you! Good question, Here are the NN stats, before lowering to C:

    total gates        | 2303 | 100.0%
    -------------------+------+-------
    passthrough        | 2134 |  92.7%
    gates w/ no effect | 1476 |  64.1%

Note the rows aren't mutually exclusive.

isaacimagine

작년discuss

Parameters and options, got it. I try to keep all configuration declarative and make building and running as deterministic as possible. Then I can commit whenever I do something interesting, that I can just checkout to revisit.

isaacimagine

작년discuss

Yes and no. I wasn't expecting to be able to reproduce the work, so I'm just content that it works. I was very surprised by how much hyperparameter finagling I had to do to get the DLGN converging; the tiny relu network I trained at the beginning, in comparison, converged with dead-simple SGD in a third of the epochs.

The speedup was surprising in the sense that the bit-level parallelism fell out naturally: that 64× speedup alone was unexpected and pretty sweet. There's likely still a lot of speed left on the table. I just did the bare minimum to get the C code working: it's single-threaded, there's no vectorization, lots of register spilling, etc. Imagine the speedup you'd get running the circuit on e.g. an FPGA.

But no, it was not surprising in the sense that yeah, multiplying billions of floats is going to be much slower than a handful of parallel bitwise ops. Physics is physics, doesn't matter how good your optimizer is.

isaacimagine

작년discuss

Agree, it's much better to write up a journal at times when your colleagues would be https://xkcd.com/303

isaacimagine

작년discuss

Chaotic energy haha, I like it. Thanks for the tips re: keeping a journal, I will do this more in the future. I usually keep development notes, though normally in markdown files scattered across the codebase or in comments, never by date in the README. In the future, I might make JOURNAL.md a standard practice in my projects? re:w&b, I used w&b when it first came out and I liked it but I'm sure it's come a lot further in the time since then. I will have to take a look!

Also lol "pretentious perfectionist" I'm glad to finally have some words to describe my design aesthetic. I like crisp fonts, what can I say.

isaacimagine

작년discuss

Author here. Any questions, ask away.

isaacimagine

작년discuss

https://dihulvhqvmoxyhkxovko.supabase.co/storage/v1/object/p...

isaacimagine

작년discuss

I tested this out with a promotional illustration from Neon Genesis Evangelion. The model works quite well, but there are some temporal artifacts w.r.t. the animation of the hair as the head turns:

https://goto.isaac.sh/neon-anisora

Prompt: The giant head turns to face the two people sitting.

Oh, there is a docs page with more examples:

https://pwz4yo5eenw.feishu.cn/docx/XN9YdiOwCoqJuexLdCpcakSln...

isaacimagine

4년 전discuss

Blockchain-like distributed concensus mechanisms do have a few practical applications: https://slightknack.dev/blog/chain/.

isaacimagine

Submissions

Druids: Batteries-included library to coordinate and deploy coding agents

Bad Apple but it's Lean Tactics

comments