The major ML conferences all have pretty tight page limits, so more expository sentences usually get cut. This also means that papers usually only explain how their work is different from previous work, so they assume you are familiar with the papers they cite or are willing to read the cited papers.
This means that people who have an up-to-date knowledge of a given subfield can quickly get a lot out of a new papers. Unfortunately, it also means that it usually takes a pretty decent stack of papers to get up to speed on a new subfield since you have to read the important segments of the commonly cited papers in order to gain the common knowledge that papers are being diffed against.
Traditionally, this issue is solved by textbooks, since the base set of ideas in a given field or subfield is pretty stable. ML has been moving pretty fast in recent years, so there is still a sizable gap between the base knowledge required for productive paper reading and what you can get out of a textbook. For example, Goodfellow et al [1] is a great intro to the core ideas of deep learning, but it was published before transformers were invented, so it doesn’t mention them at all.
I think it’s worthwhile to distinguish between throughput and latency for these sorts of discussions, rather than just talking about performance since scratchpads are usually better for latency (even best-case latency) and caches are usually better for throughput. Though of course in this as in any sort of discussion of computer performance, caveats abound.
I think it’s important to note that you are assuming a gradual increase in activity over time as well as adequate food and rest between sessions. In the limit case things like rhabdomyolysis definitely exist and in the less-extreme case, you can have problems with connective tissue injuries from increasing the load or volume of exercise too fast. Over the long term, you can also have health problems and injuries caused by doing too much exercise relative to how much food and rest you are getting. All of these are real possibilities if someone started an intense exercise program after 20 years of completely sedentary life as an adult.
That said, we are discussing this in the context of walking a bit more, which should be very well-tolerated for pretty much everyone as long as they don’t have other significant health problems and work up gradually. For example, if you take 1000 steps a day, space out the increase by each week increasing your daily steps by 500-1000 steps a day until you reach 10000 steps a day over the course of a few months (temporarily pause or reverse the increases if you see anything other than passing muscular discomfort).
This also explains how ultramarathon (or other “extreme” physical activities) can be tolerated without much issue—bodies are very good at adapting to the loads that are placed on them as long as that increase is gradual and they have enough food and rest to rebuild after particularly intense bouts of activity.
A further corollary is that over the long term, being sedentary is not well-tolerated at all, so maintaining a certain baseline level of physical activity is definitely a better idea and as long as your body isn’t giving you any feedback to the contrary, you shouldn’t avoid high loads or intense activity just out of abstract fear of injury.
Jax is a great tool, but it’s really best for training and experimentation. The transformations outlined in this post (amongst others) make it easy to turn simple and straightforward code into high performance parallel code. While this is changing, inferences hasn’t been a historical area of emphasis for the project, so it wouldn’t be my first choice if that was your primary goal.
It really depends on what you want in terms of implementation. On one hand, a big chunk of digital logic is about implementing large, high-speed state machines using circuits.
On the other hand, regular expressions and things like lexers are largely based on finite-state machines so there is a ton of information related to those, but they often implement things more complex than finite-state machines in order to be more expressive.
In terms of manually implementing them yourself, I think the naive implementation based on the mathematical definition of a deterministic finite automata is pretty good for a small number of states for things like tracking program state. In particular, explicitly listing the total set of expected inputs/transition criteria, explicitly listing the states, and having a single function that takes the current state and current transition-relevant data and produces a new state.
This representation is nice because it makes the behavior inspectable in one location. This makes it easier to notice the edge cases and prevent the state representation and possible transitions from getting spread all over code. The transition function can be a switch statement or a table. Really any way of writing a two-input function with a finite number of inputs and outputs will work. Many people have also explored strongly-typed versions of this, which are worth a look as well.
The combination of lazy evaluation and state mutation/side effects can be pretty difficult to reason about. For example, if you have a function that changes a global variable as a part of a lazy computation, once that function could have been called you have no way of knowing if or when that global variable will change in the future. If you have other functions that depend on the value of that variable, their future behavior is now much more challenging to reason about than in a strict language. You can also imagine something akin to a race condition in which there are multiple lazy computations which could eventually set that variable to different values and the actual sequence of state transitions depends entirely on the dependency order of a possibly unrelated piece of code. In practice, this means that in languages that are strict by default, lazy computations are often forced to run in order to reason about the code, rather than because the actual results of the computation are required.
Since pure functions compute the same results under lazy or strict evaluation and require that any data dependencies they have are explicitly provided as inputs, they interact with lazy computations in a much more tractable way. This means that adding a strictness operator to a lazy language is much easier than adding a laziness operator a a strict language.
An alternate approach is what python did with generators where there is a data type for lazy computation, but it lives apart from the rest of the language, so it is mostly used for e.g. stream processing where a default-lazy approach is conceptually straightforward and is less likely to lead to extremely non-trivial control flow. This approach does, however, basically give up on having a laziness operator that will turn a strict computation into a lazy one.
This means that people who have an up-to-date knowledge of a given subfield can quickly get a lot out of a new papers. Unfortunately, it also means that it usually takes a pretty decent stack of papers to get up to speed on a new subfield since you have to read the important segments of the commonly cited papers in order to gain the common knowledge that papers are being diffed against.
Traditionally, this issue is solved by textbooks, since the base set of ideas in a given field or subfield is pretty stable. ML has been moving pretty fast in recent years, so there is still a sizable gap between the base knowledge required for productive paper reading and what you can get out of a textbook. For example, Goodfellow et al [1] is a great intro to the core ideas of deep learning, but it was published before transformers were invented, so it doesn’t mention them at all.
[1] https://www.deeplearningbook.org/