The perfect configuration format? Try TypeScript(reflect.run)
reflect.run
The perfect configuration format? Try TypeScript
https://reflect.run/articles/typescript-the-perfect-file-format/
147 comments
Sounds like the Configuration Complexity Clock
http://mikehadlow.blogspot.com/2012/05/configuration-complex...
http://mikehadlow.blogspot.com/2012/05/configuration-complex...
Oh I enjoyed that, thanks. Shared around at work.
Likely controversial, but I really like how Django does its settings. I think because Python code can start off just looking like a .ini file with variables and assignments, and get more complex only as required.
On the other end of my work, I'm a fan of having embedded firmware apply to as many devices as possible, and then specialise with configuration settings from EEPROM.
Likely controversial, but I really like how Django does its settings. I think because Python code can start off just looking like a .ini file with variables and assignments, and get more complex only as required.
On the other end of my work, I'm a fan of having embedded firmware apply to as many devices as possible, and then specialise with configuration settings from EEPROM.
Did I just hear ABAP@10? No, must be my brain predicting...
Those were my concerns as well. Your config file can stay simple, sure. But I'm pretty sure it won't.
And from some years of experience with configuration management systems, configuration creation can accrue a surprising amount of complexity over time. Mostly because unlike the code they support, a config management system has to support many versions of the code and multiple versions of a configuration at the same time.
And from some years of experience with configuration management systems, configuration creation can accrue a surprising amount of complexity over time. Mostly because unlike the code they support, a config management system has to support many versions of the code and multiple versions of a configuration at the same time.
I wonder if it would help if the final generated configuration was also easy to read.
Essentially you'll generate the config from the code, but you can always view the raw output side by side.
Essentially you'll generate the config from the code, but you can always view the raw output side by side.
Except ....
(...No one asked, but here are my $0.02 ;-)
You need typescript in your environment to use typescript!
The four formats the OP listed can be read by anything from Ada to EXCEL. But typescript can only be ready by a TS build flow.
In fact, I think JSON was a bridge too far. A config file shouldn't need a compiler.[1]
FOR GENERAL PURPOSE: big no.
BUT: If this is purely for TS projects where the tooling already exists, then OK. I'd use it.
EDIT [1] I'm referring to the article's early suggestion to run JSON through JSMin to support comments. So transpiling is more apt, not compiling.
(...No one asked, but here are my $0.02 ;-)
You need typescript in your environment to use typescript!
The four formats the OP listed can be read by anything from Ada to EXCEL. But typescript can only be ready by a TS build flow.
In fact, I think JSON was a bridge too far. A config file shouldn't need a compiler.[1]
FOR GENERAL PURPOSE: big no.
BUT: If this is purely for TS projects where the tooling already exists, then OK. I'd use it.
EDIT [1] I'm referring to the article's early suggestion to run JSON through JSMin to support comments. So transpiling is more apt, not compiling.
> In fact, I think JSON was a bridge too far. A config file shouldn't need a compiler.
JSON just needs a parser, like most other configuration formats. It's purely declarative.
JSON just needs a parser, like most other configuration formats. It's purely declarative.
> In fact, I think JSON was a bridge too far. A config file shouldn't need a compiler.
Does JSON need a compiler?
Does JSON need a compiler?
It does not. Easily implemented with combinators the whole way down.
Correct, but the very beginning of the original article proposes running JSON through JSMin to support comments. So in this context, a "compiler" is being proposed (or transpiler to more modern). My comment isn't clear.
Still easily implemented with combinators the whole way down. It’s building an ast, that’s it. Once you have a tree, you can do whatever you want with the result before passing it to a compiler for a language. Here though, just serialization stuff.
Like grandma used to say: "There's nothing combinators can't solve. Except consumption. Rest in peace Grandpa Neddie."
I'm dense: what's a combinator?
I'm dense: what's a combinator?
A functional programming technique that allow you to declaratively define your language when using parsers as values. If you have a good functional language implementation, the code look very similar to someone able to read BNF. And then you run it, and get out an AST the back end.
edit: Forgot partial application. Not strictly required, but super handy.
edit: Forgot partial application. Not strictly required, but super handy.
Ah, I see. I can't reply to that thread anymore, but it sounds like a combinator is like a first-class YACC grammar in the functional language that obviates the need to write the parser. That's really cool, thanks. I'll read up on it more.
Yea, that’s pretty accurate. So now I’m curious what you thought the site was named after?
Edit: It’s not that accurate. It’s just math used in a way to compose parsers using parsers.
Edit: It’s not that accurate. It’s just math used in a way to compose parsers using parsers.
Never gave it a moment's thought. Now after Googling it, I still wouldn't know because I don't program LISP (well I haven't since 1986, but that was only an intro in college). Interesting to read about it.
You can use most of the concepts in any language that allows first class functions, closures, pattern matching and types. Combinators are just ways to compose a function to create a new function that executes some aspect upon a set (usually).
That's one proposed solution but not the only one. You could just ignore comments, like most programming languages do.
[deleted]
Great little writeup ! After mangling YAML, HCL, JSON for years as an ops engineer, I have come to the same realisation. In fact, I have put this into practice in production pipelines by using: jkcfg[1] for the last couple of years. Two data points: 1. Zero developer support contract rate around YAML syntax and templating issues 2. High number of contributions in our private typescript configuration library from developers. Using typescript as an ops frontend has made operations a lot more approachable to folks.
Recently I took what learnt in the last 2 years using jkcfg/typescript and taken it to Deno in form of an opinionated port of jkcfg called: dxcfg[2]. Its early days, but I would bet on Deno/typescript for future ops configuration.
[1] https://jkcfg.github.io/#/ [2] dxcfg: https://github.com/dxcfg/dxcfg
Recently I took what learnt in the last 2 years using jkcfg/typescript and taken it to Deno in form of an opinionated port of jkcfg called: dxcfg[2]. Its early days, but I would bet on Deno/typescript for future ops configuration.
[1] https://jkcfg.github.io/#/ [2] dxcfg: https://github.com/dxcfg/dxcfg
Good to see people coming to the same conclusions we have had for a while using Clojure. There's an up and coming runtime data validation framework called Malli that I use for these sorts of files.
You can get many of the benefits but also run custom validations (dictate how two keys in a map relate to eachother, or force a be of a certain shape instead of just a string).
Here's the example from the script: [0]
[0] - https://malli.io/?value=%7B%20%3Alocale%20%22en-US%22%0A%20%...
You can get many of the benefits but also run custom validations (dictate how two keys in a map relate to eachother, or force a be of a certain shape instead of just a string).
Here's the example from the script: [0]
[0] - https://malli.io/?value=%7B%20%3Alocale%20%22en-US%22%0A%20%...
I’m not on board with using a language that only has a single implementation and is complex enough that implementing others is difficult.
Please use a language designed for configuration, but still sane. There are several if we leave the JSON-inspired family. Starlark, pystachio, Dhall. All are restricted, but allow functions and code reuse. All are simple enough to write a new interpreter for.
Please use a language designed for configuration, but still sane. There are several if we leave the JSON-inspired family. Starlark, pystachio, Dhall. All are restricted, but allow functions and code reuse. All are simple enough to write a new interpreter for.
Honorable mention for Cue[0].
[0] https://cuelang.org/
[0] https://cuelang.org/
CUE does look simple enough to be implemented in every language. I wonder if there's anything preventing that. Perhaps there's space for a CUE-like language with an even simpler spec.
TypeScript has at least 2 implementations: tsc and Deno, but I take your point and agree. That said, most languages probably have a JS runtime, so a basic implementation would be ts->js->interpreter. The ts->js is presumably pretty easy.
I'm pretty sure Deno just uses the open source TypeScript compiler: https://github.com/denoland/deno/tree/main/cli/tsc
They use swc for transpilation and bundling but, yes, for checking the only solution available now is tsc
I don't think Deno actually re-implements typescript, does it? Seems like a lot of wasted effort. They don't even use Typescript internally.
Reimplement in the way that they change certain rules to accommodate a more "modern JS" feel
And they absolutely use TS for the std library, just not in the runtime itself
And they absolutely use TS for the std library, just not in the runtime itself
> Reimplement in the way that they change certain rules to accommodate a more "modern JS" feel
What exactly do you mean by this?
> And they absolutely use TS for the std library, just not in the runtime itself
Yes, internally. The code that actually makes up deno itself is just plain Javascript.
What exactly do you mean by this?
> And they absolutely use TS for the std library, just not in the runtime itself
Yes, internally. The code that actually makes up deno itself is just plain Javascript.
[deleted]
Geniuenly curious, what downsides do you see with a single, reference implementation? I come from C++ world where we have several competing compilers, each slightly different from the other, and all this incurs a non-negligible maintenance cost while providing very little benefit in my eyes.
The problem is not a single, reference implementation, the problem is how hard it is to implement another one. "Implementing" JSON is relatively easy, most languages have a JSON library. Most people could probably write a JSON library, not a perfect one, but that would mostly work. And probably do it in a reasonable amount of time. All of that isn't true for Typescript.
Ignoring S-expr based configs for a moment, of the configuration languages, JSON parsing is probably the most straightforward. YAML is a superset of JSON, TOML is probably on par, TS and JS are obviously more complex than JSON, and XML is...XML. All of that is to say, I'd rather write a JSON parser than any of the above.
And JSON parsing is a minefield. http://seriot.ch/projects/parsing_json.html
I don't know of any notable S-expr based config languages, probably because they are so easy to parse, people just roll their own, leading to balkanization.
And JSON parsing is a minefield. http://seriot.ch/projects/parsing_json.html
I don't know of any notable S-expr based config languages, probably because they are so easy to parse, people just roll their own, leading to balkanization.
Windows INI format is going to be easier than any of those. You can parse it in a couple lines in a bash script.
I personally find it silly to need more than that. YAML, JSON, etc. What are we even doing here, besides trying to make life more difficult? Are we doing configuration or are we doing plugins and calling it configuration?
We'd all be better off if we just admitted we really want an extension language and went with embedded Lua or Scheme. With Dockerfile and others you have to run a separate lint program, assuming anyone even bothers doing that. At least with a proper language you would have built-in robustness. Most of these configuration formats/DSLs are half-assed and unnecessary anyway.
I personally find it silly to need more than that. YAML, JSON, etc. What are we even doing here, besides trying to make life more difficult? Are we doing configuration or are we doing plugins and calling it configuration?
We'd all be better off if we just admitted we really want an extension language and went with embedded Lua or Scheme. With Dockerfile and others you have to run a separate lint program, assuming anyone even bothers doing that. At least with a proper language you would have built-in robustness. Most of these configuration formats/DSLs are half-assed and unnecessary anyway.
As Zababa said, it is more about how easy it is to write another implementation, because config files need to be read in another language than (Typescript in this case).
The easier a config language (syntactically, but also in terms of capabilities), the easier it is to write a parser in Go, and Rust and Java etc. All of this particularly matters for libraries or in larger codebases that are multi-lingual.
Using starlark here because I'm most familiar with it.
Starlark's `load()` statement is "find the file at this exact literal path from the workspace root" (no relative paths allowed either). Typescript/Javascript's import is much more complicated. If you encounter an error in Starlark, you fail right there, no exception handling to implement.
Starlark doesn't allow recursion, has an extremely limited set of built-in types and functions, and that means it is very easy to reason about, and even a naive interpreter is pretty fast at evaluating it. I'm also reasonably certain about writing a simple interpreter for it in a week if I had to port it to a new language.
Of course, I view configuration languages on a continuum.
I'd definitely want a pomodoro timer to just use a .ini file. TOML/YAML is great for where you are just going to have a bunch of plain type assignments. No JSON, because comments.
Once you want to write larger configurations, and you need to provide APIs to the user in the language, I'd pick Starlark/Pystachio because they are still intentionally restrictive and Python is one of the most popular languages out there. At this point, you are probably trying to configure something like a large build system (Bazel) or some kind of deployment cluster (say Kubernetes) and you can start providing all sorts of utility libraries (either via bindings to your programming language, or as Starlark libraries). But due to their restricted nature you can still do all sorts of bounded analysis of the evaluation. You can remove things like random() because it's a bad idea.
Yes there are use cases where you want a full blown language to configure something, but those are extremely rare and at that point I think you are no longer "configuring" but just "programming". I very much prefer to keep these 2 things separate.
The easier a config language (syntactically, but also in terms of capabilities), the easier it is to write a parser in Go, and Rust and Java etc. All of this particularly matters for libraries or in larger codebases that are multi-lingual.
Using starlark here because I'm most familiar with it.
Starlark's `load()` statement is "find the file at this exact literal path from the workspace root" (no relative paths allowed either). Typescript/Javascript's import is much more complicated. If you encounter an error in Starlark, you fail right there, no exception handling to implement.
Starlark doesn't allow recursion, has an extremely limited set of built-in types and functions, and that means it is very easy to reason about, and even a naive interpreter is pretty fast at evaluating it. I'm also reasonably certain about writing a simple interpreter for it in a week if I had to port it to a new language.
Of course, I view configuration languages on a continuum.
I'd definitely want a pomodoro timer to just use a .ini file. TOML/YAML is great for where you are just going to have a bunch of plain type assignments. No JSON, because comments.
Once you want to write larger configurations, and you need to provide APIs to the user in the language, I'd pick Starlark/Pystachio because they are still intentionally restrictive and Python is one of the most popular languages out there. At this point, you are probably trying to configure something like a large build system (Bazel) or some kind of deployment cluster (say Kubernetes) and you can start providing all sorts of utility libraries (either via bindings to your programming language, or as Starlark libraries). But due to their restricted nature you can still do all sorts of bounded analysis of the evaluation. You can remove things like random() because it's a bad idea.
Yes there are use cases where you want a full blown language to configure something, but those are extremely rare and at that point I think you are no longer "configuring" but just "programming". I very much prefer to keep these 2 things separate.
If you can afford to wear only one hat, then sure, use one special language for configuration. But I can't, I am the entire infra department and I still need to do product and framework work. Not having to switch context, language wise, is a very nice thing to have.
Why use language designed for configuration? Is it the size concern or something else, cuz I think small langs like Lua make great config language.
Anyone have experiences to share about Dhall (https://dhall-lang.org)?
On their homepage they call it a "programmable configuration language that you can think of as: JSON + functions + types + imports"
On their homepage they call it a "programmable configuration language that you can think of as: JSON + functions + types + imports"
Yeah, I've used that a year ago or so.
Pros: not Turing complete but quite powerful, nice type system and functions. For example one can write a type safe config migration from v1 to v2 (and check statically that it works), exports to JSON, YAML etc.
Cons: uses Haskell like syntax, no bindings to various languages. You basically need to ship the dhall binary to convert the config to something else that your target application will consume.
Pros: not Turing complete but quite powerful, nice type system and functions. For example one can write a type safe config migration from v1 to v2 (and check statically that it works), exports to JSON, YAML etc.
Cons: uses Haskell like syntax, no bindings to various languages. You basically need to ship the dhall binary to convert the config to something else that your target application will consume.
> no bindings to various languages.
There are bindings to some languages: Haskell, Clojure, Go, Rust, Ruby and Purescript. https://docs.dhall-lang.org/howtos/How-to-integrate-Dhall.ht...
There are bindings to some languages: Haskell, Clojure, Go, Rust, Ruby and Purescript. https://docs.dhall-lang.org/howtos/How-to-integrate-Dhall.ht...
Sorta. I evaluated it a long time ago and decided against ever touching it due to its remote import functionality. Apparently it's crazy to think that your configuration files shouldn't be issuing HTTP requests.
> Apparently it's crazy to think that your configuration files shouldn't be issuing HTTP requests.
Depends on what language runtime can do with those responses. Arbitrary code execution in a config (a la Django settings and indeed OP’s idea) can do much more damage, but even that doesn’t scare many people as this kind of configuration is considered an inherently trusted source.
Depends on what language runtime can do with those responses. Arbitrary code execution in a config (a la Django settings and indeed OP’s idea) can do much more damage, but even that doesn’t scare many people as this kind of configuration is considered an inherently trusted source.
Probably important to mention that at the moment, TypeScript as a configuration format means anyone who writes a configuration blob for you has arbitrary code execution. This is rarely an issue in practice, though.
Deno is sandboxed by default and doesn't allow disk or network access, which is ideal for configuration.
It also runs Typescript directly.
It also runs Typescript directly.
Be careful with terms like "directly" as people seem to get very confused when talking about Deno. "Transparently" may be more accurate.
My understanding of the situation right now is that by default, Deno runs TS files by transpiling using SWC, which strips types without checking them, and then sends the result to V8. Optionally you can enable type checking which invokes TSC (with no emission) before SWC. All this is done "magically" from the user's point of view.
My understanding of the situation right now is that by default, Deno runs TS files by transpiling using SWC, which strips types without checking them, and then sends the result to V8. Optionally you can enable type checking which invokes TSC (with no emission) before SWC. All this is done "magically" from the user's point of view.
This could be solved by having some kind of sandbox (https://github.com/patriksimek/vm2), but I agree it complicates it.
It would be cool if tsc had a flag —sandboxed or similar that does not allow any sideeffects (fs access, output, forking, net requests, etc)
It would be cool if tsc had a flag —sandboxed or similar that does not allow any sideeffects (fs access, output, forking, net requests, etc)
One of such sandboxes is Deno.
Not even Deno can solve the halting problem, unfortunately.
Tcl's safe interpreters [0] have various limits like instruction count, command count, time, and memory [1].
[0] https://www.tcl.tk/man/tcl8.6/TclCmd/safe.html
[1] https://www.tcl.tk/man/tcl8.6/TclCmd/interp.html#M48
[0] https://www.tcl.tk/man/tcl8.6/TclCmd/safe.html
[1] https://www.tcl.tk/man/tcl8.6/TclCmd/interp.html#M48
No need to solve the halting problem, just put a timeout.
If you use it by running a V8 isolate that transforms the exported config into JSON, then sure. But TS-based config is largely for people using TS elsewhere. And in that case:
The point of having a config file in the first place was so that it could be replaced, at start time or better yet at runtime, without touching the rest of the code, such that the same code can be parametrized and instantiated as you wish. When you use a fully-powered language, this becomes a matter of discipline instead of something guaranteed by the format.
ALL of the other competing config file formats are attempts to walk this line between too little expressive power and too much expressive power, such that people naturally separate their code and config. Typescript-as-config just abandons this goal; it's not a new idea, just one that has been rejected over and over again by each new format because everyone else saw the benefit of limiting expressive power.
// this returns in 0 ms
export const config: any = {
get value() {
while (true) {}
}
} as const;
// accessing config.value runs forever
console.log(config.value);
The point of having a config file in the first place was so that it could be replaced, at start time or better yet at runtime, without touching the rest of the code, such that the same code can be parametrized and instantiated as you wish. When you use a fully-powered language, this becomes a matter of discipline instead of something guaranteed by the format.
ALL of the other competing config file formats are attempts to walk this line between too little expressive power and too much expressive power, such that people naturally separate their code and config. Typescript-as-config just abandons this goal; it's not a new idea, just one that has been rejected over and over again by each new format because everyone else saw the benefit of limiting expressive power.
Agreed. It's a matter of discipline. It is a a balancing act between:
1. A constrained specification where its difficult(but not impossible) to shoot yourself in the foot.
2. Devops: as in equitable participation of operators and developers in maintaining configuration on a build it, run it basis.
Languages like skylark/starlark, dhall, cuelang, jsonnet promise a constrained spec(for operators) while being expressive enough(for developers). But after using skylark, jsonnet and dhall in production, personally I haven't seen enough adoption in terms of developer contribution so the promise of expressiveness which is attractive enough for developers hasn't been true for me.
At my current workplace, we have taken a middle approach. We use typescript to generate/emit yaml/json and use "official" validators to validate this configuration. The developers get a familiar and "blessed" frontend to the configuration while the operators can still enforce schema validation, resources and security policy on the generated config.
But yes, if one could import this base typescript package directly in their application, and generate the configuration which can be then picked up by the operator pipeline and isapplied to the infrastructure. That would be bad. Really bad.
1. A constrained specification where its difficult(but not impossible) to shoot yourself in the foot.
2. Devops: as in equitable participation of operators and developers in maintaining configuration on a build it, run it basis.
Languages like skylark/starlark, dhall, cuelang, jsonnet promise a constrained spec(for operators) while being expressive enough(for developers). But after using skylark, jsonnet and dhall in production, personally I haven't seen enough adoption in terms of developer contribution so the promise of expressiveness which is attractive enough for developers hasn't been true for me.
At my current workplace, we have taken a middle approach. We use typescript to generate/emit yaml/json and use "official" validators to validate this configuration. The developers get a familiar and "blessed" frontend to the configuration while the operators can still enforce schema validation, resources and security policy on the generated config.
But yes, if one could import this base typescript package directly in their application, and generate the configuration which can be then picked up by the operator pipeline and isapplied to the infrastructure. That would be bad. Really bad.
Very clear example, thank you.
I would like to see an option to whitelist syntax/types/keywords in Typescript. Allowing only const, dicts, arrays, assignments (but not while). It could just abort if something other than the whitelisted appears.
It could get rid of most of this problems, and not allow network or infinite looping (or file access, and others). Of course it would not be perfect, but could be good enough.
I would like to see an option to whitelist syntax/types/keywords in Typescript. Allowing only const, dicts, arrays, assignments (but not while). It could just abort if something other than the whitelisted appears.
It could get rid of most of this problems, and not allow network or infinite looping (or file access, and others). Of course it would not be perfect, but could be good enough.
Rhai can do this, obviously not for Js but for its own very customizable Lua-competitive language. https://rhai.rs/
It also means your configuration must be transpiled into JS first before execution unless run through a typescript runtime interpreter.
Dhall seems to gather some praises, and not fall into the pits mentioned here.
https://news.ycombinator.com/item?id=17523623
Dhall is awesome. Only downside is that the included formatter doesn't preserve comments. Super well-designed and with interesting features like semantic hashing.
My money is on https://cuelang.org in this space.
As I understand it it's essentially the Google Configuration Language (GCL) author responding to Hashicorp Configuration Language (HCL), with language development modeled after Golang. Still in early stages but very promising.
As I understand it it's essentially the Google Configuration Language (GCL) author responding to Hashicorp Configuration Language (HCL), with language development modeled after Golang. Still in early stages but very promising.
I feel like the AWS CDK is a great example of this working well. Rather than puzzling over what config options are valid, I get autocomplete and compiler errors when I insist on using an incorrect option. You also get stuff like deprecation warnings showing up in your editor.
It also becomes trivial to parameterise config so I can setup the same general Cloudfront distribution once and tweak it based on the individual application. It's open to abuse but it's better than interpolating values into Cloudformation templates.
It also becomes trivial to parameterise config so I can setup the same general Cloudfront distribution once and tweak it based on the individual application. It's open to abuse but it's better than interpolating values into Cloudformation templates.
This is so cool, I feel like other similar projects, like k8cdk, are going in the same direction for that reason.
The only problem with AWS CDK is that sometimes the type hints aren't clear enough in Python, that part could have been a lot smoother imo.
The only problem with AWS CDK is that sometimes the type hints aren't clear enough in Python, that part could have been a lot smoother imo.
Is evaluation lazy in TypeScript ?
For me, one of the most important features of a configuration language is the ability to directly refer to other values of the configuration, and more importantly, in the case of a configuration split in separate modules, to refer to values that could be provided by the other modules.
Essentially, this means that the final configuration should be the fixed point of a function, that the user would provide the configuration as a definition of this function (or in the case of multiple modules, as multiple definitions that would be merged using function composition), and that the evaluator would be able to determine the fixed point of this function.
I feel like this kind of configuration (used notably in NixOS) provides a lot of what's missing in traditional configuration languages, makes it easier to compose configuration modules, without requiring to write the configuration in a Turing-complete language.
For me, one of the most important features of a configuration language is the ability to directly refer to other values of the configuration, and more importantly, in the case of a configuration split in separate modules, to refer to values that could be provided by the other modules.
Essentially, this means that the final configuration should be the fixed point of a function, that the user would provide the configuration as a definition of this function (or in the case of multiple modules, as multiple definitions that would be merged using function composition), and that the evaluator would be able to determine the fixed point of this function.
I feel like this kind of configuration (used notably in NixOS) provides a lot of what's missing in traditional configuration languages, makes it easier to compose configuration modules, without requiring to write the configuration in a Turing-complete language.
Typescript is definitely eagerly evaluated unless you were to use a first class function to defer execution
The line between configuration and code is always a little blurry. There are some projects where using a .js file for the "configuration" of another operation is incredibly useful. TS makes even more sense in many of those situations. Lots of configuration files require pre-processing already, like merging shell env vars in with textutil or something like that. So, I wouldn't dismiss this opinion outright.
Using programming languages for configuration is a bad idea.
Couldn't agree more — a hard requirement for a configuration language is that it must be inert.
Under some circumstances, configuration needs to be generated dynamically so you need a programming language — and TypeScript would be fine for that. But a "perfect" configuration format isn't perfect if it can run arbitrary code.
Under some circumstances, configuration needs to be generated dynamically so you need a programming language — and TypeScript would be fine for that. But a "perfect" configuration format isn't perfect if it can run arbitrary code.
Many years ago, I had the idea of using Ruby as a configuration language. It lends itself nicely to representing tree structures and has some convenient features like ranges. It all worked well as long as we stuck to the subset of Ruby I intended.
One day one of the other developers introduced a change to the configuration that had a side effect. It made sense; after all, the configuration is read at startup, so surely it makes sense to put code there that you want to have run at startup? You can imagine my surprise when I made change that read the configuration again after startup.
Configuration should be purely functional and idempotent, with no side effects, or as you say, inert.
One day one of the other developers introduced a change to the configuration that had a side effect. It made sense; after all, the configuration is read at startup, so surely it makes sense to put code there that you want to have run at startup? You can imagine my surprise when I made change that read the configuration again after startup.
Configuration should be purely functional and idempotent, with no side effects, or as you say, inert.
Why? The only reasons I can think of are:
* They can be non-deterministic (do a different thing each time you run them).
* They can be non-hermetic (access stuff in the environment you don't know about).
* They can do naughty security things.
* You can't present GUIs of them because they aren't declarative.
All but the last one don't exclude programming languages. Here's an interesting project to make hermetic deterministic Javascript (Typescript support is planned):
https://github.com/jkcfg/jk
For the sorts of places where you don't have a GUI for the settings anyway (which is the common case) I think it makes loads of sense. It beats making the kind of declarative programming languages you see in YAML files.
* They can be non-deterministic (do a different thing each time you run them).
* They can be non-hermetic (access stuff in the environment you don't know about).
* They can do naughty security things.
* You can't present GUIs of them because they aren't declarative.
All but the last one don't exclude programming languages. Here's an interesting project to make hermetic deterministic Javascript (Typescript support is planned):
https://github.com/jkcfg/jk
For the sorts of places where you don't have a GUI for the settings anyway (which is the common case) I think it makes loads of sense. It beats making the kind of declarative programming languages you see in YAML files.
Static declarations without a reason are equally bad.
Then should we should do what the industry does instead — start with a flat configuration language like YAML and then add modules, variables and scopes, iteration...
And what do we do when we want the Cartesian product of two sets? Ahem Terraform.
And what do we do when we want the Cartesian product of two sets? Ahem Terraform.
Indeed, configuration should specify only state, never new behavior. For instance:
[plugin "foo"]
binary = "C:\\Program Files\\Foo\\Foo.DLL"
Except that a programming language provides:
- types if you want them
- IDEs/tooling/a kind of UI-out-of-the box
- often other useful toolchain programs for packaging, management, etc
- a well defined syntax
- and the big bertha for REALLY COMPLICATED config: autocomplete in your tool/ide/etc
- and yeah, turing complete dropouts, meta configuration frameworks for generating config
- types if you want them
- IDEs/tooling/a kind of UI-out-of-the box
- often other useful toolchain programs for packaging, management, etc
- a well defined syntax
- and the big bertha for REALLY COMPLICATED config: autocomplete in your tool/ide/etc
- and yeah, turing complete dropouts, meta configuration frameworks for generating config
Using static file formats for configuration is wishful thinking for projects that are perfectly specified on day 1 and whose requirements never, ever change.
I'd rather choose a bad idea than having to reimplement a Turing-complete programming language in YAML a few years down the line.
I'd rather choose a bad idea than having to reimplement a Turing-complete programming language in YAML a few years down the line.
Using pure programming languages for configuration is fine. The trick is that the config "program" should basically just output the final - static - configuration, and set of inputs should be very strictly controlled (i.e. no opening random files etc).
why? It can't be because then if something goes wrong you need to debug your configuration, because you need to do that often enough with a non-programming language format for even less power offered.
If your config is so complex that it needs "debugging" then it's too complex.
The reason you don't want to use a programming language for configuration is that configuration shouldn't be programming - configuration should be idempotent, it should describe a state, and it should contain only data, and structures of data (arrays, maps, sets, what have you.) No functions, no classes, no mutability.
If you use a programming language as configuration, the temptation to make your configuration more and more complex, and add features and "shortcuts" to integrate it more deeply with your application will be irresistible, and eventually your config will need its own config, which will eventually grow to need it's own config, ad infinitum.
And I say this as someone who advocates using Lua tables for config in gamedev, if you're not willing to be disciplined with avoiding Turing creep at all costs, don't do it.
The reason you don't want to use a programming language for configuration is that configuration shouldn't be programming - configuration should be idempotent, it should describe a state, and it should contain only data, and structures of data (arrays, maps, sets, what have you.) No functions, no classes, no mutability.
If you use a programming language as configuration, the temptation to make your configuration more and more complex, and add features and "shortcuts" to integrate it more deeply with your application will be irresistible, and eventually your config will need its own config, which will eventually grow to need it's own config, ad infinitum.
And I say this as someone who advocates using Lua tables for config in gamedev, if you're not willing to be disciplined with avoiding Turing creep at all costs, don't do it.
Emacs would like a word. :)
gradle has entered the chat
You are not supposed to write actual build logic inside gradle though, as many people do. It is supposed to be a DSL for build scripts the same way maven is, with slight ease of use scripting.
Gradle does suck though.
Why do you think so? I couldn’t disagree more. Task caching/avoidance, rich build domain model (source sets, configurations, tasks, extensions), huge plugin ecosystem.
My opinion is based on my experience integrating artifact generators (both code and config) into a build. I recall being funneled into building a plugin by documentation to deal with some idiosyncrasies of the generator. Whereas I just wanted to write a small script. In general I felt like there were a lot of extra hoops and concepts to learn to do basic easy shit. And there really isn't any payoff. For example I recall scripts and plugins have different capabilities. And that some things can't be expressed in untyped Gradle.
I also found that the user documentation was just a skosh above useless. It felt like it was more concerned with not being wrong than being helpful. Reminiscent of older javadocs in the java collections library.
The same could be said of many build systems, I suppose.
I also found that the user documentation was just a skosh above useless. It felt like it was more concerned with not being wrong than being helpful. Reminiscent of older javadocs in the java collections library.
The same could be said of many build systems, I suppose.
I think this is kind of a no-brainer, if your project is already based on TypeScript.
Not just for configuration, but for things like test fixture data, and even data assets like i18n translations, or basically anything that projects might commonly use JSON for, internally. It's just so much easier to create, edit, manage, and deal with in TypeScript.
But it is hard to imagine TypeScript-as-config taking root in projects that don't already use TypeScript.
Not just for configuration, but for things like test fixture data, and even data assets like i18n translations, or basically anything that projects might commonly use JSON for, internally. It's just so much easier to create, edit, manage, and deal with in TypeScript.
But it is hard to imagine TypeScript-as-config taking root in projects that don't already use TypeScript.
I've really come to prefer json5 as a config format: https://json5.org/
It's basically Json with comments, trailing commas, and unquoted object keys (and some other minor things).
Much nicer than real json as a config format, but still simple and declarative, not requiring an interpreter or being turing-complete.
It's basically Json with comments, trailing commas, and unquoted object keys (and some other minor things).
Much nicer than real json as a config format, but still simple and declarative, not requiring an interpreter or being turing-complete.
Starlark or Cue, but not a general purpose language please. Side effects in a configuration language is an atrocity...
It's possible to sandbox most languages, and with some work you can probably make them deterministic too.
Here's an example: https://github.com/jkcfg/jk
That beats having to learn an entirely new language.
Here's an example: https://github.com/jkcfg/jk
That beats having to learn an entirely new language.
Really interesting but, sincerely, I would rather use sqlite.
I know: It's a binary format. And you need to add a library to read it.
BUT
- Your configuration _has_ data types.
- You can have a copy with an old config within the file easily (create table backup_config as select * from actual_config; ).
- The file is small.
- Encryption is built-in if you need it.
- And it's programable
I know: It's a binary format. And you need to add a library to read it.
BUT
- Your configuration _has_ data types.
- You can have a copy with an old config within the file easily (create table backup_config as select * from actual_config; ).
- The file is small.
- Encryption is built-in if you need it.
- And it's programable
I have used TypeScript for config in a few projects, with node-config[1], and it’s substantially better IME than JSON/YAML/etc. Yeah it’s unconventional to write configs in a general purpose language, and I understand why that would be undesirable. But having configs validated at compile time is great at preventing bugs.
I’m not sure I’d recommend node-config (it was chosen by a past team). Its magic undermines the confidence in type safety provided by choosing TypeScript in the first place. But it does give some good baseline conventions one can apply without all the magic.
1: https://github.com/lorenwest/node-config
I’m not sure I’d recommend node-config (it was chosen by a past team). Its magic undermines the confidence in type safety provided by choosing TypeScript in the first place. But it does give some good baseline conventions one can apply without all the magic.
1: https://github.com/lorenwest/node-config
TypeScript needs to be compiled, right? So what use is a configuration file if I need to run it through the build pipeline?
The point of having configuration is to gain the ability to configure software without rebuilding in my mind. Maybe I'm missing something.
The point of having configuration is to gain the ability to configure software without rebuilding in my mind. Maybe I'm missing something.
I think we got here because everyone's mixed up describing a state with describing the process to describe a state. Or, configuration versus scripts that apply configuration. Most of the popularly config-as-code systems mix the two up, including Ansible, my favorite one. You end up with "config files" that are, all at once in the same file: 1) configuration, 2) templates for configuration, and 3) programs that modify config when they're run.
I think the whole approach is just fundamentally misguided. Each of these systems should probably use 2-3 different languages & formats, not just one. And especially not YAML. Ugh.
So you're right, config files should not be code. Unfortunately, much of "modern" config management involves writing programs in config file formats, which is also extremely bad in a whole different way.
I think the whole approach is just fundamentally misguided. Each of these systems should probably use 2-3 different languages & formats, not just one. And especially not YAML. Ugh.
So you're right, config files should not be code. Unfortunately, much of "modern" config management involves writing programs in config file formats, which is also extremely bad in a whole different way.
> TypeScript needs to be compiled, right?
Yes.
> So what use is a configuration file if I need to run it through the build pipeline?
Compilation doesn't necessarily happen in a “build pipeline”; typescript is available as a library.
> The point of having configuration is to gain the ability to configure software without rebuilding in my mind.
There are many different purposes of configuration, and they are sometimes in tension, which is why there are so many configuration formats (spanning from limited languages like TOML through general purpose programming languages, whether interpreted or compiled) with an ebb and flow as to which approaches are more and less popular over time.
Yes.
> So what use is a configuration file if I need to run it through the build pipeline?
Compilation doesn't necessarily happen in a “build pipeline”; typescript is available as a library.
> The point of having configuration is to gain the ability to configure software without rebuilding in my mind.
There are many different purposes of configuration, and they are sometimes in tension, which is why there are so many configuration formats (spanning from limited languages like TOML through general purpose programming languages, whether interpreted or compiled) with an ebb and flow as to which approaches are more and less popular over time.
The fact that it is compiled does not mean you need to run it through an additional step in your build process. In practice, it would just mean that you include TypeScript in your package, which can either be used as an executable or as a library. It can be completely transparent to the user.
You can even get by without including TypeScript, if you are willing to just strip out the type information (there are various libraries that do this).
You can even get by without including TypeScript, if you are willing to just strip out the type information (there are various libraries that do this).
> To use this approach today requires that your project also be written in Typescript. It also means you’ll need to forgo a separate flat-file for your configuration.
You're right, the summary presents this configuration as a stop-gap, a possible pre-cursor to a better, future JSON.
You're right, the summary presents this configuration as a stop-gap, a possible pre-cursor to a better, future JSON.
What would be the problem with using something like protobufs? To write the configuration, you can use whatever language you want, what matters are the outputs anyway. The outputs are stored by automation in text format so that they can be better inspected. The programs that use such configuration have the advantage of knowing the schema of the configuration and directly parse it avoiding possible typing problems. The nice thing is that you can use the same language to write the programs and the configuration.
protobuf may very well be total overkill. What you wanted to do was strongly type your configuration. What you get is a complete RPC layer.
This might not even be warranted when inter-process communication is a primary concern. After all, there's nothing inherently wrong with using whatever data transfer mechanism your standard lib provides.
It's not just protobuf though. Almost every data format for serializing structured data suffers from pushing a jungle on you when all you asked for was a banana.
The least invasive format I know of is FlatBuffers[0]. Doesn't push RPC on you. Allows you to parse a message even if you do not know this specific message's data structure in advance (like JSON).
[0]: https://google.github.io/flatbuffers/
This might not even be warranted when inter-process communication is a primary concern. After all, there's nothing inherently wrong with using whatever data transfer mechanism your standard lib provides.
It's not just protobuf though. Almost every data format for serializing structured data suffers from pushing a jungle on you when all you asked for was a banana.
The least invasive format I know of is FlatBuffers[0]. Doesn't push RPC on you. Allows you to parse a message even if you do not know this specific message's data structure in advance (like JSON).
[0]: https://google.github.io/flatbuffers/
1. As said in the original comment, it doesn't necessarily have to be protobuf, but just a data serialization system.
2. Protobuf per se is just a serialization system, it does not push RPC "Protocol buffers are Google's language-neutral, platform-neutral, extensible mechanism for serializing structured data – think XML, but smaller, faster, and simpler." [0]. It can be used in combination with RPC systems, but does not have to.
[0] https://developers.google.com/protocol-buffers
[0] https://developers.google.com/protocol-buffers
I think parsing YAML or JSON into typed structures is the easier way to go. I e.g. do that in Golang using a little form validation and coercion library I've written. The end result is a nested, strongly typed data structure. Here's an example: https://github.com/iris-connect/eps/blob/master/settings.go (the accompanying form validation configuration: https://github.com/iris-connect/eps/blob/master/forms/settin...). Hashicorp has some libraries that do similar stuff, I wrote my own though since I need it quite often and wanted to have full control over it. The library is just a few hundred lines of Golang code (https://github.com/kiprotect/go-helpers/tree/master/forms). Undocumented as of now as it's nothing I want to explicitly share with the world.
In my experience, a lot of the validation needs to be done at runtime anyway as type checking alone won't allow you to e.g. validate if a string is a valid regular expression. Also, I think using TypeScript for configuration requires you to compile & interpret the configuration file in order to check it and obtain the actual configuration values. Not sure if I like that as it requires bundling the entire TypeScript compiler with your program.
In my experience, a lot of the validation needs to be done at runtime anyway as type checking alone won't allow you to e.g. validate if a string is a valid regular expression. Also, I think using TypeScript for configuration requires you to compile & interpret the configuration file in order to check it and obtain the actual configuration values. Not sure if I like that as it requires bundling the entire TypeScript compiler with your program.
TypeScript is limiting for describing schema in my experience. Try requiring an array of certain length, for example.
Also, it’s Turing-complete, so you are enabling future developers or users to shoot themselves in the foot and write something that does not halt already at configuration stage.
To me the perfect language for configuration seems to be Dhall, but it doesn’t seem to have gained any traction.
Also, it’s Turing-complete, so you are enabling future developers or users to shoot themselves in the foot and write something that does not halt already at configuration stage.
To me the perfect language for configuration seems to be Dhall, but it doesn’t seem to have gained any traction.
Dhall is too unfamiliar for most developers, IMO. Developers don’t want to have to grok an entirely new syntax just for config. I’m personally surprised no one has added type annotations to Starlark yet.
The Rust implementation has it as an experimental extension (https://github.com/facebookexperimental/starlark-rust/blob/m...)
Oooh, very cool! I'll check it out and maybe port to Go.
Everything is unfamiliar, until one adopts it.
I didn't think it needed to be explained, but not all things are equally unfamiliar nor equally easily adopted.
[deleted]
Depending on what you're looking to do, you can use a tuple type to get an array of a certain length.
[number, number, number]
[number, number, number]
I meant length that depends on another type, not length known beforehand (e.g., if N widgets are defined somewhere, [up to] N elements can exist somewhere else, or configuration should not compile). Naturally, this had to involve generics (similar to [0], except I don’t recall why but the `length` property workaround didn’t work or didn’t provide sufficient type safety in my use case). Eventually, after reading workarounds in a related issue[1], I gave up. TypeScript’s type system is great for its intended use, but when applied to expressing schema constraints it quickly falls short and runtime code execution is required to validate a given structure with all the corresponding drawbacks.
[0] https://stackoverflow.com/q/56034226/247441
[1] https://github.com/microsoft/TypeScript/issues/26223
[0] https://stackoverflow.com/q/56034226/247441
[1] https://github.com/microsoft/TypeScript/issues/26223
> not length known beforehand
Is that a feature of other languages?
Is that a feature of other languages?
Haven’t had much exposure to other statically typed languages recently, so can’t say off the top of my head. Intuitively it looks like something modern generics should allow easily though, doesn’t it?
As to Dhall, for example, I could validate such a constraint (and more) using a function I can be reasonably sure is safe and won’t halt (i.e. without all the drawbacks of TS and other general-purpose languages).
As to Dhall, for example, I could validate such a constraint (and more) using a function I can be reasonably sure is safe and won’t halt (i.e. without all the drawbacks of TS and other general-purpose languages).
Kind of interesting but now I need to run arbitrary code to read a config?
To be fair, you already have to execute arbitrary code to understand what config is doing.
Where execute is to sequence a series of effects that are described in a file.
Where execute is to sequence a series of effects that are described in a file.
Typescript has a lot of punctuation and thus it's not really human-editable. I'd prefer a ini-style format or dockerfile style (with one directive per line of text, clearly identified by a single uppercase word). Moreover, these formats are so easy to parse that you do not even need a library to do so.
Except there’s the ini where strings need to be quoted and the one where they don’t. The one where white space is allowed after the equal sign and the one where it’s not. The one where keys are case sensitive and the one where they’re not.
TOML cleans up most of that, though.
TOML cleans up most of that, though.
I'm puzzled what you mean by this. Do you mean it's not easily editable by non-engineers?
It's annoying to edit by everybody. I guess engineers will especially abhor the unnecessary complexity.
This is the proposal of TFA:
It is beyond me how could any serious engineer honestly find any advantage in the first version.
This is the proposal of TFA:
export const config: MyAppConfig = {
locale: 'en-US',
timezone: 'America/New_York',
logLevel: 'info',
environment: 'local',
}
LOCALE en-US
TIMEZONE America/New_York
LOGLEVEL info
ENVIRONMENT local
It is beyond me how could any serious engineer honestly find any advantage in the first version.
Well, if you want nesting on your map then you need something slightly more complicated than your second version, but I will grant that you could do something like:
A.B value
C value
D.E.F value
D.E.G valueYou know what we don't need? Yet Another Configuration Format! Also, don't tell me I can't add comments to JSON because I sure can - add it as a field! If a comment is that important then it's important enough to transfer along with the dataset and durably persist.
I've done that before and found that my comment got reordered by a tool that thought the order of objects didn't matter.
Personally I wouldn't use Typescript for config. It breaks my #1 dev rule which is KISS. Keep it simple stupid. Simplicity is orders of magnitude more important than other things like DRY or config schema.
I'm curious what folks who agree with this think about setup.py then? There's been a push to move to setup.cfg to specifically avoid the fact that python libraries run arbitrary code on install.
The problem with setup.py is different. Because it can run arbitrary code, package writers go crazy and import extra dependencies, but you specify your package dependencies in setup.py, so without some out-of-band way to tell your users that the setup.py file in itself also has dependencies, package installation on machines other than your own might fail unless by sheer luck your users already had the packages being used in setup.py.
That problem doesn't apply to arbitrary configuration, but only to configuration files that represent package build and installation metadata. You do see this issue with things like Gradle, though, where your build.gradle file can represent the dependencies of the package being built, but the build file itself may have dependencies since you can run arbitrary Groovy code and import any package that can be contained in a jar file. It gets around this because Gradle itself can just download those dependencies before trying to execute the build script, which would be perfectly possible for a real Python build system, but not when setup.py is just being executed by the Python interpreter, which doesn't do dependency management and expects imports to already be available in the system import path.
That problem doesn't apply to arbitrary configuration, but only to configuration files that represent package build and installation metadata. You do see this issue with things like Gradle, though, where your build.gradle file can represent the dependencies of the package being built, but the build file itself may have dependencies since you can run arbitrary Groovy code and import any package that can be contained in a jar file. It gets around this because Gradle itself can just download those dependencies before trying to execute the build script, which would be perfectly possible for a real Python build system, but not when setup.py is just being executed by the Python interpreter, which doesn't do dependency management and expects imports to already be available in the system import path.
Heck, for what it's worth, even Make has this issue, as people quite often write Makefiles assuming you have a bunch of tools installed that may or may not be part of the POSIX specification. I wish I could find it now, but somewhere in the GNU Make guide book is a list of more or less "approved" utilities you're supposed to be able to safely assume are installed on any GNU based system that you can use in a Makefile, and you're not supposed to go beyond that.
I can't believe what i am reading...
JS people gonna JS.
> JS people gonna JS.
Also reinvent the wheel. They can list four other configuration formats du jour, but conveniently leave out s-expressions, which have solved most problems people bring up over half a century ago.
Also reinvent the wheel. They can list four other configuration formats du jour, but conveniently leave out s-expressions, which have solved most problems people bring up over half a century ago.
[deleted]
My only real criticism is that I believe it's generally an anti-pattern to have configuration that necessitates DRY. It has essentially the same problems as OOP inheritance. Because configuration isn't and shouldn't be an application in and of itself, avoiding potential issues and misdirection from avoiding DRY in this case seems like the right tradeoff to having to use multi-select to change the config format in your IDE.
Cuelang enables DRY while forbidding inheritance.
I like the idea of the IDE assisting with the schema-correctness of the config (which you get for free with TypeScript coupled with any competent IDE configured to use the typechecker to validate the code).
... but as a rule of thumb, if your config language is Turing complete, you're setting yourself up for future pain as complexity of config grows with time and someone makes the bad decision to take advantage of that Turing completeness...
... but as a rule of thumb, if your config language is Turing complete, you're setting yourself up for future pain as complexity of config grows with time and someone makes the bad decision to take advantage of that Turing completeness...
Great idea.
And because you need many constants that often change in such a configuration file you could put that all in another file and include that in the 'real' configuration file. Why not call that, hmm, `config_config.txt`. But as plain text is a bit hard to parse, we could give that file some structure, so that it is like, well, another markup language, just without markup and easier to read and write.
I think typescript as a config format alone may get too complex.
But json backed by a typescript interface definition could be a useful middle ground.
You're still writing json but you get nice code completion and realtime validation of the config file.
But json backed by a typescript interface definition could be a useful middle ground.
You're still writing json but you get nice code completion and realtime validation of the config file.
I use TypeScript and like the ecosystem.
For broader use, I would like to see native support or an official extension for creating dependent types or specific constraints.
Things like number ranges, or the basics from JSON Schema, etc.
For broader use, I would like to see native support or an official extension for creating dependent types or specific constraints.
Things like number ranges, or the basics from JSON Schema, etc.
I've done a similar thing with Scala in the past - ship the compiler jar and then compile your configuration (which is just an abstract class implementation) as part of the startup process.
Not typed support but I find jsonnet [0] pretty useful when you have tons of configuration files.
[0] - https://jsonnet.org/
[0] - https://jsonnet.org/
Apple takes a similar approach for swift. Package.swift files run real swift code and export an object that specifies the package’s configuration.
In practice I think it works fairly well.
In practice I think it works fairly well.
How to add comments in json or TypeScript config files? Commenting is very much needed for any manually written code or config we keep in repositories.
Typescript is perfect for stuff like this because it's so much less verbose than most things out there
As a bonus, you might not need validation and schemas, too.
As a bonus, you might not need validation and schemas, too.
Should this be a future ECMAScript enhancement or is the better route for browsers to natively support TypeScript?
YAML is beautiful when it's simple and horrible when you need to dynamically generate it. The indentation...
The idea of allowing logic from a programming language in a config format is.... at best very horrifying...
OK, so we're coming back around to the "configuration as code" trend again then.
Protocol Buffers text format ?
Composability in configuration is really important beyond a small scale. If you're running a replicated server in n locations, it's far better to keep one config file that has some form of conditional logic based on location than to maintain n textprotos that need to be kept synced.
Protobuffs remain very underrated outside of Google.
or json-schema with jsonc or js files
One of the arguments for TS in the post is that it can encourage DRY. It was DRY that made this TS-based kubernetes config so hard to follow, because simple repetitive statements got abstracted into parameterised functions, or classes to inherit from, or other things that are resolved dynamically at runtime. Not to mention that you're also going to start mixing in random dependencies from NPM to help abstract things further. This is great for application development but not so great for pure configuration.
A language designed for config, like Dhall or CUE, can give you much of what a typed language offers without letting you go crazy with a full-blown language runtime.