eiz · HackerTrans

vor 3 Jahrendiscuss

https://arxiv.org/pdf/2302.13971.pdf table 15. 1770394 A100-80GB hours to train the entire model suite at the going rate for cloud 8xA100-80GBs (~$12/hr if you could actually get capacity) is ~$2.6M, under extremely optimistic assumptions. YMMV on bulk pricing ;) "the more you buy the more you save"

eiz

vor 3 Jahrendiscuss

> Where is the connection between computational details and the model's high-level behavior? Do we even know?

This is an active area of study ("mechanistic interpretability") and it's very early days. For instance here's a paper I read recently that tries to explain how a very simple transformer learns how to do modular arithmetic: https://arxiv.org/abs/2301.05217

Curious what interesting results people are aware of in this area.

eiz

vor 3 Jahrendiscuss

> 4. Describing positional embeddings as multiplicative. They are generally (and very counterintuitively to me, but nevertheless) additive with token embeddings.

Worth noting that rotary position embeddings, used in many recent architectures (LLaMA, GPT-NeoX, ...), are very similar to the original sin/cos position embedding in the transformer paper but using complex multiplication instead of addition

eiz

vor 3 Jahrendiscuss

Fair enough. In the GH100 architecture doc https://resources.nvidia.com/en-us-tensor-core/gtc22-whitepa... (page 18) they do mention retaining 2 graphics-capable TPCs but it's clearly not the focus.

eiz

vor 3 Jahrendiscuss

> They do not have any graphics rendering circuitry.

What? Not having a display output is not the same as not having graphics rendering circuitry. Here's vulkaninfo from an A100 box: https://gist.github.com/eiz/c1c3e1bd99341e11e8a4acdee7ae4cb4

eiz

vor 3 Jahrendiscuss

Here's a couple more quotes from Yann LeCun, their Chief AI Scientist:

https://twitter.com/ylecun/status/1643945742850031616

https://twitter.com/ylecun/status/1644503237699969026

pretty clear the non-release and extremely half assed response to the leak is just CYA

eiz

vor 4 Jahrendiscuss

I tried this:

> Solve the following word problem step by step:

> "John is in a train moving west at ten miles per hour. Jill is in a train moving north at twenty miles per hour. How fast are John and Jill moving apart from each other?"

To solve this problem, we first need to understand the concept of speed. Speed is a measure of distance traveled over time. In this problem, we are told that John and Jill are traveling at different speeds in different directions.

Since the two trains are moving in different directions, we cannot simply add or subtract their speeds to find the combined speed at which John and Jill are moving apart from each other. Instead, we need to use the Pythagorean theorem to find the combined speed.

The Pythagorean theorem states that in a right triangle, the square of the length of the hypotenuse (the longest side of the triangle) is equal to the sum of the squares of the lengths of the other two sides. In this problem, we can consider the trains to be moving along the two sides of a right triangle, with the distance between them as the hypotenuse.

We can use the Pythagorean theorem to find the distance between John and Jill as follows:

First, we square the speed of each train and add the results:

10 mph * 10 mph + 20 mph * 20 mph = 100 + 400 = 500

Next, we take the square root of the result to find the distance between John and Jill:

sqrt(500) = 22.36 mph

Therefore, the combined speed at which John and Jill are moving apart from each other is approximately 22.36 mph.

---

Results seem pretty inconsistent though. Other attempts have given complete nonsense.

comments