michaelhartm

3 yıl öncediscuss

Btw, semantics and syntax is separated in the LLMs (the author is wrong). The embedding function (matmul) can map syntax and the proximity in the embedding (e.g. cosine similarity) is the semantics (that's attention). So not convinced. Chomsky might be wrong or right, but this author hasn't proven it.

michaelhartm

3 yıl öncediscuss

Nobody knows how the LLMs work under the hood. It's just lots of stacked transformers that encode various concepts. Nothing in this book refutes whether Chomsky's concepts are actually being encoded in LLMs or not. For all we know, Chomsky's concept of "binding principles", "binary branching" etc could be represented inside the inner layers of these many billion parameter models. In fact, I'd argue that this is the right research to do. Prove that no transform or feed-forward layer inside the neural net encodes, say "binding principles".

michaelhartm

3 yıl öncediscuss

I guess Databricks is now going after OpenAI?

michaelhartm

3 yıl öncediscuss

Btw, it's kinda crazy how bad the GPT4-J results in the blog are compared to the Dolly one, which seem pretty good. Do we know why it works so well to use this 50k dataset?

michaelhartm

3 yıl öncediscuss

They used the 6b GPT4-J, not 20B. That's what's interesting, it's a smallish large language model :).

michaelhartm

5 yıl öncediscuss

Data Wars: Snowflake vs Databricks (0 - 2)?

michaelhartm

5 yıl öncediscuss

* Databricks is unethical

* Nobody should benchmark anymore, just focus on customers instead

* But hey, we just did some benchmarks and we look better than what Databricks claims

* Btw, please sign up and do some benchmarks on Snowflake, we actually ship TPC-DS dataset with Snowflake

* Btw, we agree with Databricks, let's remove the DeWitt clause, vendors should be able to benchmark each other!

* Consistency is more important than anything else!!!

comments