dhampi · HackerTrans

2개월 전discuss

Well, with thinking models, it’s not that simple. The probability distribution is next token. But if a model thinks to produce an answer, you can have a high confidence next token even if MCMC sampling the model’s thinking chain would reveal that the real probability distribution had low confidence.

dhampi

6개월 전discuss

Canada has no domestic automaker and US automakers, under pressure from Trump, are closing some factories in Canada & relocating production to the US.

Yes, the Canadian auto industry will take a hit, but it already has from the US (and might take more).

dhampi

6개월 전discuss

Presumably a lot less expensive than pumped hydro to build.

dhampi

7개월 전discuss

The actual title is pretty buzzy given how limited the task described is. In one specific, very constrained and artificial task, you can find something like detailed balance. And even then, their data are quite far from being a perfect fit for detailed balance.

Would love it if I could use my least action principle knowledge for LLM interpretability, this paper doesn't convince me at all :)

dhampi

7개월 전discuss

No, I still don’t understand the analogy.

All of this burn-in stuff is designed to get your Markov chain to forget where it started.

But I don’t want to get from “how many apples does Bob have?” to a state where Bob and the apples are forgotten. I want to remember that state, and I probably want to stay close to it — not far away in the “typical set” of all language.

Are you implicitly conditioning the probability distribution or otherwise somehow cutting the manifold down? Then the analogy would be plausible to me, but I don’t understand what conditioning we’re doing and how the LLM respects that.

Or are you claiming that we want to travel to the “closest” high probability region somehow? So we’re not really doing burn-in but something a little more delicate?

dhampi

7개월 전discuss

I don't understand the analogy.

If I'm using an MCMC algorithm to sample a probability distribution, I need to wait for my Markov chain to converge to a stationary distribution before sampling, sure.

But in no way is 'a good answer' a stationary state in the LLM Markov chain. If I continue running next-token prediction, I'm not going to start looping.

dhampi

8개월 전discuss

I used to be befuddled by this too. Then I lived in the U.S. for a few years.

I think the answer is that the democrats are shockingly bad too, in many parts of the US. People expect grift and corruption from both parties.

Perhaps they didn’t expect the scale of this admin’s grift.

dhampi

8개월 전discuss

Guessing you’re a physicist based on the name. You don’t think automatically doing RG flow in reverse has beauty to it?

There’s a lot of “force” in statistics, but that force relies on pretty deep structures and choices.

dhampi

9개월 전discuss

I quit Julia after running into serious bugs in basic CSV package functionality a few years back.

The language is elegant, intuitive and achieves what it promises 99% of the time, but that’s not enough compared to other programming languages.

dhampi

10개월 전discuss

FOAG is probably the shortest readable introduction to serious algebraic geometry anyone has written. That's the nature of algebraic geometry.

dhampi

10개월 전discuss

Yes, Schrodinger’s equation is entirely deterministic. There is no randomness inherent in quantum mechanics. The perceived randomness only arises when we have incomplete information. (It is another matter that quantum mechanics to some extent forbids us from having perfect information.)

I mean no disrespect, but I don’t think it’s a particularly useful activity to speculate on physics if you don’t know the basic equations.

comments