> At the core RL is just updating a table of values, and then using function approximation (aka, machine learning) for more complex cases.
this seems to be a common assertion about ml. other refrains include "ml is just matrix multiplication" and "ml is just affine transformations followed by nonlinearity".
while technically true, it is an unhelpful comment as it doesnt shed any light on the salient questions, such as:
* how do you reduce bias/variance, since youre sampling a minuscule slice of the state/action space?
* how do you construct a value function (or something like it) in a sparse reward environment, with possibly thousands of time steps?
* how do you know your policy network is exploring new states?
it is the equivalent of learning how to draw an owl: draw a few circles, then draw the fucking owl.
> 3) Test that the solution works on toy examples, like MNIST, simple block worlds, simulated data, etc.
youre right: mnist, imagenet, etc are toy examples that do not extend into the real world. but the point of reproducible research is to experiment on agreed upon, existing benchmarks.
> What are the use cases for adding yet another layer to the stack?
in my limited experience with horovod, horovod is most useful when youre running large clusters of workers/ps. in those situations, you typically have to manually find the appropriate balance of workers/ps. (otherwise youd run into blocking or network saturation issues.) horovod addresses this issue with their ring allreduce implementation.
having said all of that, im sticking with distributed tf for now.
> Usually you throw everything and see what sticks.
most practitioners start with the simplest possible learner, then gradually, and thoughtfully, increase model complexity while paying attention to bias/variance. this is far from a "kitchen sink" approach.
not exactly what you asked for, but if youre looking for a gentle academic introduction to the intersection of ai and games, i recommend [
AI Researchers, Video Games Are Your Friends!](https://arxiv.org/abs/1612.01608)
> Why should we expect there to be any mathematical foundation to this stuff?
i would be surprised that "this stuff" would be exception to the unreasonable effectiveness of mathematics. mathematics underpins virtually every observed phenomenon, including theoretical physics, computer science, economics. in fact, the mathematical structure of any physical theory often points the way to further advances in that theory and even to empirical predictions.
to not expect that "this stuff" should not have any mathematical foundation is a fantastically naive view.
the point is: anything complex can be dismissed as "just x,y,z" if you dont appreciate the massive body of work behind it.
i made that point because OP observed that "ml is just affine transformations" or something to that effect. yes, that's one way to frame it - if youre okay overlooking roughly 30 years of research.
how does this solve anything? if a simple decision tree could predict the outputs of more complex deep nets, why not use the decision tree in the first place? also, what do you do when a decision tree isnt powerful enough, as in the case of many interesting problems such as speech, computer vision, etc.
it was the ability of steve jobs to create an organizing principle (and its corresponding organizational structure) that deserves credit for apple's success.
> I am curious to know what major innovations in search engines happened since the page rank algorithm, or were there only incremental improvements?
a ton has happened! since pagerank, theres been a ton of advances around nlp that has changed the way queries are processed prior to information retrieval. for example, google's rankbrain seems to do a lot of the heavylifting around word similarity.