Yglu: YAML augmented with an expression language(yglu.io)
yglu.io
Yglu: YAML augmented with an expression language
https://yglu.io/
30 comments
For large, repetitive configurations (Kubernetes configs, CloudFormation templates, CI job configurations, etc) you often want ways to abstract and/or DRY your configurations. Kubernetes folks use mustache/jinja/Go templates via Helm, which is a real mess. CloudFormation builds its own (shitty) evaluation language on top of YAML. This presumably aims to improve on those sort of hacky solutions. I feel strongly that the "right" solution is either (1) generate a YAML config with Python/JS/etc or (2) use a terminating expression language (I think that may be what this is). Since (2) implies a much steeper learning curve for average developers, I opt for (1).
That's what I've been finding a lot lately. Kubenetes manifests have very complete schemas, but there's not really any official concise way to express them or relations between them.
I've come to the conclusion that helm/templating is awful. Every helm chart I've ever seen suffers from the kitchen sink problem. Dependencies don't really address the problem either, because that's not really resolving a whole lot in the end.
What I've found so far is that Kustomize strikes a solid balance between an off the shelf working config provided from official sources, and making changes myself to suit my environment. Instead of the developer maintaining a kitchen sink approach, they can have something (or a set of somethings) that are straight forward, and I can maintain a set of tweaks that work for me. Unfortunately Kustomize's documentation is not that great, and it is incredibly awkward to work with.
I'm hoping that solid operators come out for most applications instead of helm charts. I'm not 100% sure that they're going to be substantially better than helm, but I'm hoping between operators and kustomize I can link applications together in a way that I get properly maintained applications for minimal effort (ala AWS), without having to maintain a large code base linking all the bits together.
I do think I need to look into pulumi as it might be in the direction I want to go, but I remain skeptical.
Terraform has some good ideas, but is too rigid for kubernetes, and I feel like I'm giving up the flexibility of kubectl. I like the idea of having a state and resolving differences between desired and actual state, but not enough to rewrite everything in HCL.
I've come to the conclusion that helm/templating is awful. Every helm chart I've ever seen suffers from the kitchen sink problem. Dependencies don't really address the problem either, because that's not really resolving a whole lot in the end.
What I've found so far is that Kustomize strikes a solid balance between an off the shelf working config provided from official sources, and making changes myself to suit my environment. Instead of the developer maintaining a kitchen sink approach, they can have something (or a set of somethings) that are straight forward, and I can maintain a set of tweaks that work for me. Unfortunately Kustomize's documentation is not that great, and it is incredibly awkward to work with.
I'm hoping that solid operators come out for most applications instead of helm charts. I'm not 100% sure that they're going to be substantially better than helm, but I'm hoping between operators and kustomize I can link applications together in a way that I get properly maintained applications for minimal effort (ala AWS), without having to maintain a large code base linking all the bits together.
I do think I need to look into pulumi as it might be in the direction I want to go, but I remain skeptical.
Terraform has some good ideas, but is too rigid for kubernetes, and I feel like I'm giving up the flexibility of kubectl. I like the idea of having a state and resolving differences between desired and actual state, but not enough to rewrite everything in HCL.
> I like the idea of having a state and resolving differences between desired and actual state, but not enough to rewrite everything in HCL.
i authored https://get-kapp.io based on my previous experiences managing iaas resources as cattle. it strongly revolves around idea of managing chnageset between actual and desired configuration.
it was also built as a tool that solely focuses on deployment to k8s, leaving configuration building (ie templating) to other tools.
take a look at it, would love to hear what you think.
on a side note, someone recently asked me to also wrap kapp as a terraform provider: https://github.com/k14s/terraform-provider-k14s
i authored https://get-kapp.io based on my previous experiences managing iaas resources as cattle. it strongly revolves around idea of managing chnageset between actual and desired configuration.
it was also built as a tool that solely focuses on deployment to k8s, leaving configuration building (ie templating) to other tools.
take a look at it, would love to hear what you think.
on a side note, someone recently asked me to also wrap kapp as a terraform provider: https://github.com/k14s/terraform-provider-k14s
> This presumably aims to improve on those sort of hacky solutions.
Alas, the ship of toothpaste has sailed out of barn squeezed all over the back of the horse.
We must retreat to Tom's Obvious, Minimal Language[1] and just ventilate anyone trying to throw it into the metaphorical blender.
[1] https://github.com/toml-lang/toml
Alas, the ship of toothpaste has sailed out of barn squeezed all over the back of the horse.
We must retreat to Tom's Obvious, Minimal Language[1] and just ventilate anyone trying to throw it into the metaphorical blender.
[1] https://github.com/toml-lang/toml
I agree with you that the job that Yglu is trying to do would be much better done (and a hell of a lot more readable) as Python code.
Automated config management.
As for YAML itself, it is handy to have some advanced functionality in a non-turing complete serialization format for the sake of things like security. It's not perfectly analogous, but PostScript is an example of a format that ideally should have been just a descriptive format for the purposes it was intended for, but being Turing complete makes it a security risk
As for YAML itself, it is handy to have some advanced functionality in a non-turing complete serialization format for the sake of things like security. It's not perfectly analogous, but PostScript is an example of a format that ideally should have been just a descriptive format for the purposes it was intended for, but being Turing complete makes it a security risk
https://get-ytt.io (i am one of the authors), one of the tools that inspired yglu, has been designed from ground up with an eye towards security even though its turing complete.
it is based on starlark which is a runtime that does not have facilities to do networking, fs access, etc. building on that, ytt does not allow/provide any kind of non-deterministic operations (access to time, disk, network, random, etc). it expects user to specify explicitly which files to load via -f flag (ie ytt template cannot load random template from fs).
there are several computation attacks that are currently possible (eg infinite loop) but that could be easily addressed thru global timeout for example.
check out interactive playground at https://get-ytt.io
it is based on starlark which is a runtime that does not have facilities to do networking, fs access, etc. building on that, ytt does not allow/provide any kind of non-deterministic operations (access to time, disk, network, random, etc). it expects user to specify explicitly which files to load via -f flag (ie ytt template cannot load random template from fs).
there are several computation attacks that are currently possible (eg infinite loop) but that could be easily addressed thru global timeout for example.
check out interactive playground at https://get-ytt.io
Dhall[1] is a similar concept, though instead of augmenting a particular format, it is an implementation of a simple pure functional programming language (with a very expressive type system which guarantees termination). Dhall programs can then be evaluated to several different formats (currently JSON, YAML and XML).
[1]: https://dhall-lang.org/
[1]: https://dhall-lang.org/
Dhall is awesome and rooted in the solid ML line. I wish pure functional programming langs were significantly growing in the industry. It would show signs of maturity but it is still currently a high-hanging fruit for many and traction is diluted in the impression of hobbyism or academism.
What can we bring for those dealing with YAML everyday? Hard to tell them "switch all your stuff to X". Easier to tell them "you can iteratively add functional expressions to your YAML". That's the point of Yglu. It's not about proposing a better Dhall or Jsonnet. It is about proposing better YAML-native tooling.
That is fascinating. I feel like I've seen this before and forgotten about. Really neat and I love that you can grab shared code from a CORS URI. That's great, especially for internal code sharing.
Some competing languages: https://news.ycombinator.com/item?id=20357079
I'm especially interested in jsonnet (https://jsonnet.org/) because of how thoroughly it's documented.
An example that produces three different config files:
I'm especially interested in jsonnet (https://jsonnet.org/) because of how thoroughly it's documented.
An example that produces three different config files:
local application = 'my-app';
local module = 'uwsgi_module';
local dir = '/var/www';
local permission = 644;
{
'uwsgi.ini': std.manifestIni({
sections: {
uwsgi: {
module: module,
pythonpath: dir,
socket: dir + '/uwsgi.sock',
'chmod-socket': permission,
callable: application,
logto: '/var/log/uwsgi/uwsgi.log',
},
},
}),
'init.sh': |||
#!/bin/bash
mkdir -p %(dir)s
touch %(dir)s/initialized
chmod %(perm)d %(dir)s/initialized
||| % {dir: dir, perm: permission},
'cassandra.conf': std.manifestYamlDoc({
cluster_name: application,
seed_provider: [
{
class_name: 'SimpleSeedProvider',
parameters: [{ seeds: '127.0.0.1' }],
},
],
}),
}Congratulations, you just overengineered a data format.
Yaml was overengineered to begin with.
There are not enough upvotes in the universe for this comment.
For when ordinary YAML isn't quite crazy enough.
Worth reading this if you are considering using it: https://www.arp242.net/yaml-config.html
Worth reading this if you are considering using it: https://www.arp242.net/yaml-config.html
That, and the Norwegian Trap[1].
[1] https://news.ycombinator.com/item?id=17359376
[1] https://news.ycombinator.com/item?id=17359376
Every data format secretly yearns to be XML, even if it loudly denies it when asked directly.
In features but not in eye cringe.
There are features in yaml compared to which xml is a thing of ageless beauty.
one of the ytt authors here.
looks like a very similar effort to ytt: https://get-ytt.io/
it seems like yaql is a different, potentially more limiting, scripting language, but otherwise similar efforts.
looks like a very similar effort to ytt: https://get-ytt.io/
it seems like yaql is a different, potentially more limiting, scripting language, but otherwise similar efforts.
actually glanced over the blog post. so this is inspired by ytt, which is pretty cool.
However, I am not totally sure that the use of exclamation mark and question mark to start expressions makes it any more readable than embedding code inside comments.
Also not sure what a "single source for template values" means , since in ytt you can have multiple input sources
However, I am not totally sure that the use of exclamation mark and question mark to start expressions makes it any more readable than embedding code inside comments.
Also not sure what a "single source for template values" means , since in ytt you can have multiple input sources
Yes, I liked the idea of ytt and after playing a bit, I felt that we could do YAML processing in some more YAML-native way.
YTT fulfills its mission, but I wanted to explore a more generic approach and have the loading of libs and values using the same mechanism. Also, allowing the reusable parts to be written in YAML and not only in the additional programming language.
I aim at bringing some ideas in this area and if they make sense, I am happy to see them live, be it in Yglu, YTT or any other tool.
Also, the fact that building Yglu was fast because of YAML parser and YAQL were already existing made me explore on my own, rather than developing these ideas in modifying or coding YTT libs, what I first imagined.
About the readability, I also don't see an argument here. !? vs. #@ does not matter a lot. It only matters that !? is already understood by the YAML parser as a tag, avoiding me to write a parser (I would not have started this work). The cool thing is that tags can also be used in mapping keys.
I will rephrase the "single source for template values" into something that gives better credit, because I really meant that import mechanisms should be the same for values and other stuff (libs, functions, ...). And aslo not only as overlay.
Thanks for the inspiration and let's continue to try and relieve the YAML pain around us.
YTT fulfills its mission, but I wanted to explore a more generic approach and have the loading of libs and values using the same mechanism. Also, allowing the reusable parts to be written in YAML and not only in the additional programming language.
I aim at bringing some ideas in this area and if they make sense, I am happy to see them live, be it in Yglu, YTT or any other tool.
Also, the fact that building Yglu was fast because of YAML parser and YAQL were already existing made me explore on my own, rather than developing these ideas in modifying or coding YTT libs, what I first imagined.
About the readability, I also don't see an argument here. !? vs. #@ does not matter a lot. It only matters that !? is already understood by the YAML parser as a tag, avoiding me to write a parser (I would not have started this work). The cool thing is that tags can also be used in mapping keys.
I will rephrase the "single source for template values" into something that gives better credit, because I really meant that import mechanisms should be the same for values and other stuff (libs, functions, ...). And aslo not only as overlay.
Thanks for the inspiration and let's continue to try and relieve the YAML pain around us.
Hold on. YAML gained popularity as a nice syntax for JSON, which itself was a subset of JavaScript, a full programming language. Now, we are adding some computational capabilities back to YAML? Wouldn't it be better to just execute JavaScript to generate a JSON object and skip building a new language inside of YAML?
Compared to Helm and other common kubernetes templating solutions that try to use blind textual replacement in whitespace-sensitive language, this (or something like it) looks like a god-send.
People that hate YAML must also dislike Markdown.
I despise YAML but like Markdown. They serve different purposes. Markdown is a light-weight way to format prose. YAML is a config language. Markdown is pretty good at formatting prose. YAML is a disaster confusing syntax unsuitable for configs past about 100 lines.
We have a 1500 line YAML file for our test config because YAML is the one true method for setting up tests. The issues I've run into:
- Indentation errors. Luckily CircleCI has a YAML validator, but the fact that a validator is required isn't great.
- true, false, yes, no. Why?
- date and number coercion vs strings.
- YAML has aliases which in theory let you reuse config. Except you can't compose aliases, like have JavaTest inherit config from GitHubReporter and SlackReporter.
My view is that complex configs should be generated from a reproducible binary, preferable in a real programming language with loops, conditionals, and some form of easy composition. Dhall, jsonnet [2], or Cue [3] are all interesting languages in this space.
[1]: https://dhall-lang.org/ [2]: https://jsonnet.org/ [3]: https://news.ycombinator.com/item?id=20362951
We have a 1500 line YAML file for our test config because YAML is the one true method for setting up tests. The issues I've run into:
- Indentation errors. Luckily CircleCI has a YAML validator, but the fact that a validator is required isn't great.
- true, false, yes, no. Why?
- date and number coercion vs strings.
- YAML has aliases which in theory let you reuse config. Except you can't compose aliases, like have JavaTest inherit config from GitHubReporter and SlackReporter.
My view is that complex configs should be generated from a reproducible binary, preferable in a real programming language with loops, conditionals, and some form of easy composition. Dhall, jsonnet [2], or Cue [3] are all interesting languages in this space.
[1]: https://dhall-lang.org/ [2]: https://jsonnet.org/ [3]: https://news.ycombinator.com/item?id=20362951
Compare the YAML spec [0] to the Markdown spec [1]
0. https://yaml.org/spec/1.2/spec.html
1. https://daringfireball.net/projects/markdown/index.text
0. https://yaml.org/spec/1.2/spec.html
1. https://daringfireball.net/projects/markdown/index.text
The link you listed for Markdown isn't the spec. Here is a more official spec:
https://spec.commonmark.org/0.29/
https://spec.commonmark.org/0.29/
In my experience also, YAML is already too feature-laden for it's own good in this regard, when what people usually want is just a more human-readable JSON.
Using Yglu for config seems like a bad idea (the whole point of config is that it's not code!) - so what is it for?