I've been using unikraft (https://unikraft.org/) unikernels for a while and the startup times are quite impressive (easily sub-second for our Rust application).
I think for a small configuration TOML might be fine. Where it breaks down in my opinion is in larger configuration files. It just becomes pretty unreadable.
Think about a Github Action being written in TOML ... would probably not look great!