flebron · HackerTrans

I think you might be thinking of "AGI" as some sort of point in time, where something happens and everyone all at once has some technology. Not only is the progress towards AGI gradual, it's also very jagged in both capabilities and especially who has access to it. It's irrelevant whether a particular company, like Cisco, Pepsi, or Tesla, has some capability, when there exists a different research lab that is at the frontier, approaching AGI from some direction.

flebron

25 天前discuss

Source for this? The notion of attention dates to a content-addressable lookup during sequence alignment (as well as, concurrently, memory lookups in neural Turing machines). Attention had been used in other models, like GRUs and LSTMs with attention. The Vaswani et. al. paper did not introduce attention, just removed everything _but_ attention (and FFW) from the network. Are you claiming the "critical idea" of removing the GRU and LSTM parts and just keeping attention was "truly" Noam's?

flebron

2 個月前discuss

The standard way of doing MTP is to run the drafter autoregressively for k steps, and then (not concurrently) use the larger model as a verifier for those k tokens at the same time. The larger model can then accept a prefix of those k tokens, and in any case generates one more token (which is needed in case you accepted zero tokens from the drafter). The larger model can effectively use this k as a "batch" dimension, reducing the penalty of large weight loading. Meanwhile the drafter is much smaller, so it's fine for _it_ to be autoregressive, as long as the main model is parallel.

flebron

3 個月前discuss

The website asks what they do in Haskell. The answer is property modification and reading, as well as very powerful traversal constructs, use lenses (https://hackage.haskell.org/package/lens , tutorial at https://hackage.haskell.org/package/lens-tutorial-1.0.5/docs...).

flebron

3 年前discuss

"Almost nobody except X does Y." and "Z does Y, with Z != X" are consistent. Is your disagreement entirely due to the (possibly nil) distinction between "Almost nobody except X does Y" and "Almost nobody does Y -- Except X", which is how the article is worded? If so, at least one native English speaker will disagree.

The poster you're replying to seems to have more of an emphasis in how the "Wrong." parent was worded. There was no need to be that confrontational, when one is adding a point of information ("Z does Y, with Z != X") that is most likely consistent with the thread title, and with the article content.

comments