Designing a package manager from the ground up(circleci.com)
circleci.com
Designing a package manager from the ground up
https://circleci.com/blog/designing-a-package-manager-from-the-ground-up/
17 comments
Orb is actually a Docker image (executor) plus few defined commands. Say, if you want to build a custom Slack notification for multiple projects, you'd create an Orb with an executor and a command, e.g.
executors:
alpine:
docker:
- image: alpine:3.6
parameters:
webhook_url:
description: URL to Slack webhook
type: string
message:
description: Message to post to Slack
type: string
commands:
notify:
description: "Notify Slack"
executor: alpine
steps:
- run:
name: Notify Slack
command: |
curl -X POST -H 'Content-Type: application/json' --data "{ \
\"attachments\": [ \
{ \
\"fallback\": \"<<parameters.message>>\", \
\"text\": \"<<parameters.message>>\", \
\"mrkdwn\": true, \
\"color\": \"#1cbf43\", \
} \
] \
}" <<parameters.webhook_url>>
orbs:
slack_notify: sirn/slack@dev
workflow:
foo:
jobs:
- slack/notify:
webhook_url: "..."
message: "..."
Man, that's terrible. How the hell are you supposed to syntax check that, or version control it, or collaborate on it? Extract it from yaml, do a thing, test it, then put it back in yaml, then run it and test it again? And if anyone else wants to make a change to your change, they now have to do the same? What a pain in the ass.
If this is all Docker containers anyway, why aren't they just creating Docker images and referencing them? They're just shoving a Dockerfile into YAML, without the benefit of it actually being a Dockerfile.
If this is all Docker containers anyway, why aren't they just creating Docker images and referencing them? They're just shoving a Dockerfile into YAML, without the benefit of it actually being a Dockerfile.
At least yaml allows for comments. This is not supported with JSON (at least not with most parsers).
I recently migrated a javascript project to typescript. We had some yaml based localization file and I decided to turn it into a typescript file by translating the yaml into json and then turning that into a ts file that exports constants. Crucially, typescript can do type inference and multiline strings.
The benefits of this are that I can now autocomplete keys in the file elsewhere in my code and get their type inferred correctly.
Similarly, gradle now supports kotlin in addition to groovy as a language for specifying build files in. The gradle dsl is more or less a declarative DSL. Using a statically compiled language such as Kotlin means you gain type safety and better tool support (autocomplete, correctness checking). Relative to Groovy you don't actually lose anything in terms of expressiveness but you gain a lot.
Neither typescript nor Kotlin are particularly more verbose than JSON though the lack of supporting a colon to map keys to values in a dictionary remains an unfortunate language choice in Kotlin.
Probably typesafe configuration DSLs are going to be a thing in the future. Neither JSON nor Yaml really do this. And schemas just add a lot of cruft.
I recently migrated a javascript project to typescript. We had some yaml based localization file and I decided to turn it into a typescript file by translating the yaml into json and then turning that into a ts file that exports constants. Crucially, typescript can do type inference and multiline strings.
The benefits of this are that I can now autocomplete keys in the file elsewhere in my code and get their type inferred correctly.
Similarly, gradle now supports kotlin in addition to groovy as a language for specifying build files in. The gradle dsl is more or less a declarative DSL. Using a statically compiled language such as Kotlin means you gain type safety and better tool support (autocomplete, correctness checking). Relative to Groovy you don't actually lose anything in terms of expressiveness but you gain a lot.
Neither typescript nor Kotlin are particularly more verbose than JSON though the lack of supporting a colon to map keys to values in a dictionary remains an unfortunate language choice in Kotlin.
Probably typesafe configuration DSLs are going to be a thing in the future. Neither JSON nor Yaml really do this. And schemas just add a lot of cruft.
> Probably typesafe configuration DSLs are going to be a thing in the future. Neither JSON nor Yaml really do this. And schemas just add a lot of cruft.
You kind of need schemas. Or you need better type systems. I want at least regexp validation, interval validation, etc. from my configuration files.
You kind of need schemas. Or you need better type systems. I want at least regexp validation, interval validation, etc. from my configuration files.
That's the point of typescript. It has a type system. And regular expressions are supported in javascript/typescript as well. Interval validation might be a bit harder.
In any case, you don't get any of that with most json/yaml based configuration languages. I seem to rarely encounter anything that comes with a schema.
In any case, you don't get any of that with most json/yaml based configuration languages. I seem to rarely encounter anything that comes with a schema.
I know that Typescript has a type system :)
My point was that the current config file formats are not good enough - I agree with that - but if we want to switch away from them, we need even better formats. They need to be at least good as the previous generation (XML), which had schemas. JSON and YAML are a regression which became popular because of their simplicity.
We need the simplest config format that comes with a schema. They should either come with a separate schema system or with a type system that is powerful enough to implement what I said before: regexp validation, interval validation. Without making you write more than validate="[a-z]*", not writing a whole class or such.
My point was that the current config file formats are not good enough - I agree with that - but if we want to switch away from them, we need even better formats. They need to be at least good as the previous generation (XML), which had schemas. JSON and YAML are a regression which became popular because of their simplicity.
We need the simplest config format that comes with a schema. They should either come with a separate schema system or with a type system that is powerful enough to implement what I said before: regexp validation, interval validation. Without making you write more than validate="[a-z]*", not writing a whole class or such.
Yeah I had a similar experience recently moving our elastisearch mapping from plain JSON into typescript. Boy does it help immensely, especially if you have a lot of duplicated fields.
You are not alone, I had to use Yaml for some Swagger related work, hoping never to deal with it again.
I rather have my json, but preferably XML with schema validation, having the joy of auto-complete and proper documentation.
I rather have my json, but preferably XML with schema validation, having the joy of auto-complete and proper documentation.
JSON is a bare data format, YAML is a feature-filled data format, TOML is a configuration format, XML is punishment from God for our sins of over-engineered design by working group. They'll be useful for different things (simple data transfer, complex data objects represented in applications, configuring an application, and sadism/masochism, respectively)
I've been using HJSON lately and it's been a joy.
I really like their "volatile" decision. More package managers need to adopt it, in my opinion.
Not another vendor-lock-in package manager. And, very predictably, no end-to-end integrity and nonrepudiation. smh.
Yeah, it's unfortunate. Rather than a solution that could be adopted outside of CCI, this is only useful in those configs. So as soon as you use them in your org with this product, you have to reinvent the wheel to get the same functionality in any other CI that a different team might be using.
Which package manager do you feel gets it right?
I’m not the OP but Nix/NixOS is the closest I’ve ever seen to a “Platonic ideal” package manager
I think you'll like AppFS
http://AppFS.rkeene.org/
http://AppFS.rkeene.org/
Am I the only one who finds YAML to be difficult to reason about? I find that if the config is that complex, JSON or XML to be much easier to deal with. I think of TOML as the best balance between ease of use and expressiveness.
These ORBs sound similar to docker images, I’m surprised they didn’t offer a comparison, like “why not docker?”, or did I miss it?