New features includes config type safety, more powerful command line interface, new plugins for launching remotely and to perform hyper parameter optimization and more.
Hydra has some overlap with what MLFlow is already doing, but I am sure MLFlow can benefit from configuration composition.
Another way to look at it is:
What kind of Hydra plugins can be backed by MLFlow.
for example, remote launching to any cloud can be implemented as a Hydra Launcher plugin.
Currently the included launcher plugin is trivial (local execution), and the more interesting launchers to launch remotely are not open sourced as hey rely on internal FB infrastructure. Supporting the major cloud providers in Hydra through plugins is definitely something I would like to see.
2. It sure does have a lot of degrees of flexibility, and the PETS reference implementation is full of hard coded configuration. It would be nice to be able to combine configuration files somehow and override them from the command line.
3. Looking around, I didn't find anything that was quiet what I was after so I created OmegaConf (https://github.com/omry/omegaconf)
4. Building the framework, I evolved my understanding of what composing configuration actually means, and added additional features as I needed them.
5. The MBRL framework became really nice and useful, but was still primarily an implementation of PETS. I tried to make it a bit more generic to allow others in my team to use it for other things, and realized the better thing to do is to go for something totally generic and make it a standalone package, Thus Hydra was born.
It evolved into what you see today, and it was almost exclusively my work - so at no point did I have the cycles to really support development in other languages.
One of the challenges is to convince people that it's useful, It can do many things as some people point out and people can get uneasy because of that.
For Hydra to succeed, people should be using it, there is no point in spending years building something no one would use.
ML, which is where Hydra grew from is dominated by Python today, That's why Python is the first implementation.
Building Hydra in another language is possible, especially now that there is a reference implementation, but it's a lot of work.
If you are interested in trying it out I would be happy to discuss and advise.
Something like this could be implemented, but Hydra is using argparse for a few basic flags (--help, --run, --cfg etc) and argparse would not like us passing arbitrary undefined flags.
Philosophically speaking, I am not seeing a big advantage for Hydra overrides syntax to line up with getopt. What it does is different.
For example, unlike with getopt - you MUST specify a value with every override, there is no support for flags like --verbose. it would have to be verbose=true, the reason is that those are editing the configuration object, and in 99% of the cases you want to put in a value and not default to true.
Also, you should be thinking of those command line arguments as editing a complex object. db.driver=mysql means: in the db map, change value of the driver key to mysql.
Devops scripts are definitely a good use case, but so is machine learning research, which requires a lot of flexibility.
The origins of the framework is in ML, not devops.
Click let's you build really nice command line libraries, but those are rigid and their structure is hard coded.
A good example of what you wouldn't do with Click is logging configuration. You have a bunch of appenders, formatters and loggers and this is just not good with something like Click.
Hydra actually comes with built in Python logging configuration, it's just a part of the configuration object and Hydra is setting it up for the app:
https://cli.dev/docs/tutorial/logging
This kind of flexibility is a core part of how Hydra build the config, it's not limited to it's internal config (like logging) but extends naturally to the user config as well.
At the moment it's not as rich as all of the other command line libraries in term of types, enums etc but it's richer what it you can easily express with it.
For example, I challenge you do express logging configuration with oslo.config, or Click, org argparse.
You would be writing a whole lot of code to come up with something very rigid.
With Hydra composing hierarchical configs is trivial so logging configuration is not harder than anything else.
What you would do with those other CLI libraries is just is either either hard code your logging config (eew) or separate it into a different config file and add logic to deal with it. In my book this logic is an example of boilerplate that Hydra can handle cleanly.
Check the blog post for more details.