You are right. Another detail is to find a metric which is meaningful in long term, e.g., the one which is in line with long-term happiness of your customers.
I guess it is up to you to define your metric to be measured and to be optimized. To define the right metric and targets is not easy and probably is business-specific, i.e., is not the same for every business.
I am wondering if it is based on their own Google HyperTune / Vizier [1, 2] which is modified to better deal with uncertainties or it is an absolutely independent in-house development.
Thank you for providing useful references. Hyperparameter tuning can accelarate CPLEX by 10x and more depending on problem instances (however, it it true that each new version of CPLEX is faster and faster).
What I meant is that in the case if your problem is formulated in a way that the CPLEX and Gurobi cannot treat it (e.g., stochastic and multiobjective) and is not very large-scale, then one can use heuristics. However, the efficiency of the latter will likely depend on hyperparameter settings which need to be set properly.
You may formulate it as a multi-objective problem where you minimize the estimated time of the trip and your uncertainty of this estimate. Often, the two objectives are conflicting. As a user, the software should show you a Pareto set of optimal solutions where each solution has its duration and uncertainty. Then it is up to you which solution to pick. I hope they have this feature at optibus.
You can use heuristics if/when the problem becomes ugly due to various constraints you may impose. The trick is to select heuristics depending on the problem at hand and in a way that their hyperparameter values are selected properly.
I think that in the context that he meant, 64 variables = 64 hyperparameters because it was about benchmark problems solved in black-box settings. In other words, it is not about the common misconception of comparing #hyperparameters and #variables on some ML model, where the latter can be in 10^7.
Please note that their work is primarily for expensive optimization where you cannot afford millions of function evaluations like you would need on rotated Rastrigin to solve it exactly for n>64.