To be honest I think this may be actually an advantage: it explains concepts which otherwise are just weird parameters in code. Since it is pretty lengthy I would actually recommend to read the chapters relevant to a specific method you are interested in (maybe going a backwards to build the right context).
Good read. Does anyone know if any of the experimentation frameworks actually uses these methods to make the results more reliable (e.g. allow to automatically apply winsorization or attempt to make the split sizes even)?
Evolving neural networks with use of the fitness predictors (co-evolution). The project has stalled a bit due to lack of time (family etc) but overall I want to get back hacking around it.
There is one angle I really like in this concept: letting go and acknowledging that something is imperfect rather than continuing to try to make it ideal. I saw so many times people arguing and spending countless hours to reimplement or refactor something just to end up with a different version of the very same problem (i.e. imperfections in architecture).
To clarify: I've also done the same thing (i.e. refactor in pursuit of perfect architecture) and I'm not saying refactoring is bad. My point is that it takes some experience and humility to be able to say "stop" when it does not make more sense to continue to refactor.
Apart from NEAT/HyperNEAT there are also other approaches to neuroevolution (I think in this context it is referred to as "Evolutionary Neural Architecture Search" [0]). Evolution in general can be applied in different ways (e.g. optimizing the architecture, replacing training gradient descent etc.).
A while ago I co-authored a paper in this space [1] and released some code for interested folks [2].
I think the point is that the control group and treatment groups are in fact the same group (you were giving them the same "service" before starting the experiment) and any difference in the beta_C and beta_T is just random noise. The key part of the argument is that if N goes to infinity the beta_C should be equalt to beta_T (hence it doesn't make much sense to compare them).
If you I'd combine it with e.g. https://spinningup.openai.com/en/latest/ or doing some toy projects with https://stable-baselines3.readthedocs.io/en/master/ it would probably render the most value.