human_scientist

vor 7 Jahrendiscuss

> To submit to arxiv you need to be approved by someone as a legitimate researcher, or beg for reviews from people you’ve never met without any anonymity.

I did not have to do something like this.

human_scientist

vor 7 Jahrendiscuss

It depends on the number of evaluations. The more evaluations the stronger the model built by the TPE algorithm. With very few evaluations, we would expect TPE to match random sampling. This effect can for example be seen in the plots of the "Bayesian Optimization and Hyperband" paper [1, 2], where the plotted "Bayesian Optimization" approach is TPE.

Also, there might be model bias: for example if the objective function is stochastic (e.g., a reinforcement learning algorithm that only converges sometimes) or not very smooth, TPE might exploit areas that are not actually good based on one good evaluation. In those cases TPE might perform worse than random search! To alleviate the effect of model bias in model based hyperparameter optimization (e.g., TPE) and to obtain convergence guarantees, people often sample every k-th hyperparameter setting from a prior distribution (random search) (this is also the case for the plots in [1, 2].

If you are wondering which HPO algorithm you should use (or HPO in general), I would highly recommend the first part of the AutoML tutorial at NeurIPS2018 [3] given by my advisor.

[1] https://www.automl.org/blog_bohb/

[2] http://proceedings.mlr.press/v80/falkner18a.html

[3] https://nips.cc/Conferences/2018/Schedule?showEvent=10979

human_scientist

vor 7 Jahrendiscuss

HyperOpt is a library for hyperparameter optimization, my comment was about algorithms. From the homepage of hyperopt:

"Currently two algorithms are implemented in hyperopt: Random Search Tree of Parzen Estimators (TPE)"

(TPE is a Bayesian Optimization algorithm)

human_scientist

vor 7 Jahrendiscuss

What is the expected lifetime?

human_scientist

vor 7 Jahrendiscuss

Nice project!

You use gridsearch for hyperparamter optimization and state that at some point you would like to add a bayesian approach. One simple change that could boost the performance, would be to use random search inplace of gridsearch. Grid search is known to perform worse than random search in cases where not all hyperparameters are of similar importance [1]. Intuitively, in grid search a lot of evaluations evaluate the same setting of an important hyperparameter with changing settings of not important ones.

[1] Section 1.3.1; Automatic Machine Learning: Methods, Systems, Challenges; Chapter Hyperparameter Optimization

https://www.automl.org/wp-content/uploads/2018/11/hpo.pdf

human_scientist

vor 7 Jahrendiscuss

The NFL only applies to settings where your task distribution is uniform random over all possible tasks. It is my intuition that this kind of task distribution is almost surely not something we would encounter.

human_scientist

vor 7 Jahrendiscuss

Your algorithm description sounds like a variant of greedy local search with a NN heuristic. While I personaly do not know of a paper that does this kind of local search (I have not looked), I do want to add that there is a rich literature on "learning to learning", including "learning to optimize" approaches, e.g., https://arxiv.org/abs/1606.04474.

human_scientist

vor 7 Jahrendiscuss

> I don't see "Automatic design of novel algorithms" in this list. Can AutoML produce something as novel as a GAN, CapsNet, WaveNet, Transformer, Neural ODE, etc? Is that even considered to be one of its goals. In my opinion, there's a clear separation between a group of people trying to improve AutoML so that it's more useful in doing all those tasks on the list, and a group of people trying to invent next gen ML algorithms or DL architectures.

I agree with you from the view of the current state of the art methods and the current state of the AutoML / fundamental ML research communities. Current methods are very limited, but I can not think of a reason why a sufficiently general searchspace of architectures/pipelines could not produce something like a GAN or a WaveNet.

I do not think that designing algorithms as novel as the ones you listed is currently a goal of AutoML, as that is not something we have an attack for. However, I do think that with increasing capabilities, the field of AutoML will seek to automate every step of the machine learning pipeline - including the design of algorithms. E.g., once/if there are attacks to apply NAS for yielding truly novel architectures, I think NAS researchers will be happy to do just that -- wouldn't you call that AutoML then?

human_scientist

vor 7 Jahrendiscuss

The field of automatic machine learning (abbreviated as AutoML) concerns all endeavours to automate the process of machine learning. To provide a sense of what could constitute AutoML, let me post a list from the "Call for Papers" of the International Workshop on Automatic Machine Learning (ICML 2018) [1]:

    * Model selection, hyper-parameter optimization, and model search
    * Neural architecture search
    * Meta learning and transfer learning
    * Automatic feature extraction / construction
    * Demonstrations (demos) of working AutoML systems
    * Automatic generation of workflows / workflow reuse
    * Automatic problem "ingestion" (from raw data and miscellaneous formats)
    * Automatic feature transformation to match algorithm requirements
    * Automatic detection and handling of skewed data and/or missing values
    * Automatic acquisition of new data (active learning, experimental design)
    * Automatic report writing (providing insight on automatic data analysis)
    * Automatic selection of evaluation metrics / validation procedures
    * Automatic selection of algorithms under time/space/power constraints
    * Automatic prediction post-processing and calibration
    * Automatic leakage detection
    * Automatic inference and differentiation
    * User interfaces and human-in-the-loop approaches for AutoML

[1] https://sites.google.com/site/automl2018icml/

human_scientist

vor 7 Jahrendiscuss

Why? The concept of AutoML does include the design of novel algorithms.

human_scientist

vor 7 Jahrendiscuss

Could you give an example? I have a hard time understanding what you could mean, as Algorithm Configuration & Selection is such a general framework. If you are solely talking about the current state of the art, I would agree that techniques from AutoML do not have the generality and autonomity of an expert human.

human_scientist

vor 7 Jahrendiscuss

The parent did not specifically talk about NNs. As I understand it AutoML could apply to all statistical endeavours that involve estimation (classical or bayesian).

human_scientist

vor 7 Jahrendiscuss

Automatically building a scikit learn estimator might include many conditional hyperparameters and also a very large amount of them (<100) [1]. However, performing joint architecture and hyperparameter search can be framed to be on a much simpler search space, e.g., for a recent paper that aims to automate the design of RNA molecules, we formulated a 14 dimensional search space which includes very little conditional hyperparameters [2].

The tools included in the repository are very broadly applicable and only a few of them are specifically targeted at neural architecture search.

[1] https://www.kdnuggets.com/2016/08/winning-automl-challenge-a... [2] https://openreview.net/forum?id=ByfyHh05tQ

human_scientist

vor 7 Jahrendiscuss

What about approaches like auto-sklearn [1]? With these it is basicaly:

  >>> automl = autosklearn.classification.AutoSklearnClassifier()
  >>> automl.fit(X_train, y_train)
  >>> y_hat = automl.predict(X_test)

[1] https://automl.github.io/auto-sklearn/stable/

human_scientist

vor 8 Jahrendiscuss

https://eternagame.org/

This is RNA Design for humans as a game! Also: A trained human outperforms current AI approaches for RNA Design.

human_scientist

Submissions

Automatic Machine Learning Tutorial [NeurIPS 2018]

Learning to Design RNA

comments