Faster and more flexible pipelines with a Directed Acyclic Graph(about.gitlab.com)
about.gitlab.com
Faster and more flexible pipelines with a Directed Acyclic Graph
https://about.gitlab.com/blog/2020/05/12/directed-acyclic-graph/
39 comments
The fact that everyone goes out of their way to reinvent this really goes to show how awful the Make experience is.
I have a pretty awesome experience with make. If you use it for what it's designed for i.e. building files from a graph of dependencies, it's pretty handy. I use for auto-generated code, binary code as well as tasks that depend on other tasks, such as logging into system A requires getting this key from AWS etc. Most people have bash shortcuts for simple scripts to automate things, I instead write Makefiles, since it seems closer to my mental model, you want to do X but doing X depends on resources Y Z T, make figures out the rest.
Unless you're working on something complex, Make is pretty great. My only complaint as a regular use is the lack of support for spaces in target names.
I didn't read too deeply, but I'm glad to see all these sorts of systems slowly converge on a sane model.
The part that saddens me is that they all end up looking like a glorified distributed make.
The part that saddens me is that they all end up looking like a glorified distributed make.
I'd hardly call any of them sane. There used to be debates about checking compilers into the VCS and now no one bats an eye when the entire build system is constantly churning underneath you. Today you have GitLab Runner 12.x. Next month it'll be 13.x. I imagine most other CI systems are similar, the hosted variants being especially bad because you can't even decide when to update.
Modern CI feels like a shell script using a bunch of external APIs got tossed into a blender and poured into a YAML config. Then everyone claims builds are repeatable because they run in a container, but ignore the fact that if your Dockerfile has a single command that isn't working with exact versions for resources (apt-get anyone?) you can't rely on the build being repeatable between jobs in the same pipeline.
A big problem I think we'll face in the future is there's no value in giving developers a product they can control. GitLab CI will evolve so you need a top tier plan to do anything interesting and they'll eventually own your build environment. GitHub will push things like Codespaces and they'll own your development environment. The idea that anyone would even touch GitHub Actions blows my mind.
What happened to the developers that want to control their environments? Do you actually own your codebase if it's useless without a bunch of paid subscriptions you're using as part of your development workflow? What happens when your CI provider triples the price or shuts down entirely?
Modern CI feels like a shell script using a bunch of external APIs got tossed into a blender and poured into a YAML config. Then everyone claims builds are repeatable because they run in a container, but ignore the fact that if your Dockerfile has a single command that isn't working with exact versions for resources (apt-get anyone?) you can't rely on the build being repeatable between jobs in the same pipeline.
A big problem I think we'll face in the future is there's no value in giving developers a product they can control. GitLab CI will evolve so you need a top tier plan to do anything interesting and they'll eventually own your build environment. GitHub will push things like Codespaces and they'll own your development environment. The idea that anyone would even touch GitHub Actions blows my mind.
What happened to the developers that want to control their environments? Do you actually own your codebase if it's useless without a bunch of paid subscriptions you're using as part of your development workflow? What happens when your CI provider triples the price or shuts down entirely?
Make is a 44 year old implementation of the DAG model. It is not the only possible one, and it has plenty of flaws. At least in principle, there is room for a better implementation of the DAG model, and even different implementations tailored for different ecosystems.
The Mother of all Demos was amazing 5 decades ago. And yet we're not stuck using the exact same UI toolkit as Douglas Engelbart.
The Mother of all Demos was amazing 5 decades ago. And yet we're not stuck using the exact same UI toolkit as Douglas Engelbart.
Why does that sadden you? Make doesn't do the job. Therefore a new tool is required. What's the problem?
The problem of DAG is solved by make very well. So well that it is why people insist on using it. In theory they would have made a good choice.
The issue is that "make" made sure to put enough traps so that people with less knowledge of the pitfalls tend to fall through. How many times I have seen recursive make calls without the $(MAKE) variable lead to funny races or lost compilation flags? How many paralelizable targets running in single thread because somebody did not understand some strange out of order execution(due to wrong Dependency mapping)? How many programs which are only partially built? How many projects were not actually built for huge periods of time because somebody forgot to put PHONY target? How many Frankenstein build systems exist which create Meta languages around make(kbuild) ? My latest adventure was a dormant fork bomb for 4 years in a daily updated project in the clean target! Whole development server farms being resource starved because the clean target started being executed as part of the normal build. How did the fork bomb happen? The usual recursive make.
Make is perfect actually,but we are not perfect in its usage.
The issue is that "make" made sure to put enough traps so that people with less knowledge of the pitfalls tend to fall through. How many times I have seen recursive make calls without the $(MAKE) variable lead to funny races or lost compilation flags? How many paralelizable targets running in single thread because somebody did not understand some strange out of order execution(due to wrong Dependency mapping)? How many programs which are only partially built? How many projects were not actually built for huge periods of time because somebody forgot to put PHONY target? How many Frankenstein build systems exist which create Meta languages around make(kbuild) ? My latest adventure was a dormant fork bomb for 4 years in a daily updated project in the clean target! Whole development server farms being resource starved because the clean target started being executed as part of the normal build. How did the fork bomb happen? The usual recursive make.
Make is perfect actually,but we are not perfect in its usage.
My problems with this feature:
- You can't have a DAG of stages. Sometimes, especially with a monorepo which this touts, the maintenance of the "needs" becomes burdensome and you want to just block on a stage rather than explicitly named jobs
- The visualization of this is subpar.
So far my favorite implementation of this feature is Azure Pipelines. No idea if its coped into Github Actions yet or not (haven't switched over). I hate that is the case because of my underlying caution about Microsoft after the 90s and early 2000s though supposedly they are better now.
- You can't have a DAG of stages. Sometimes, especially with a monorepo which this touts, the maintenance of the "needs" becomes burdensome and you want to just block on a stage rather than explicitly named jobs
- The visualization of this is subpar.
So far my favorite implementation of this feature is Azure Pipelines. No idea if its coped into Github Actions yet or not (haven't switched over). I hate that is the case because of my underlying caution about Microsoft after the 90s and early 2000s though supposedly they are better now.
Can't you do something like:
-Stage1 needs A, B, C
-D needs Stage1Basically faking a stage by making a job?
That helps but the "definition" of a stage is still far away from the job definition when you have enough of them and people won't know to update this when they copy/paste a job definition for adding a new test.
Plus, visualization is still terrible :)
That helps but the "definition" of a stage is still far away from the job definition when you have enough of them and people won't know to update this when they copy/paste a job definition for adding a new test.
Plus, visualization is still terrible :)
I don't disagree. Just trying to help.
So, just like Concourse, Drone and any other modern CI system. About time.
Yeah so I just want to point out that most other CI systems are harder to understand, less documented, harder to bring your own hardware to bear on, and do less for the general software community than GitLab does.
I've used CircleCI, Travis, Drone (I don't think I've used Concourse actually) -- and I by far prefer GitLab CI, it is more featureful and more easy to use.
I've used CircleCI, Travis, Drone (I don't think I've used Concourse actually) -- and I by far prefer GitLab CI, it is more featureful and more easy to use.
This feature is currently broken. It can let deploys run without builds/tests passing.
https://gitlab.com/gitlab-org/gitlab/-/issues/31526
https://gitlab.com/gitlab-org/gitlab/-/issues/31526
Hi! GitLab Community Advocate here: I noticed the issue you linked is closed. I believe the conversation is continued here: https://gitlab.com/gitlab-org/gitlab/-/issues/213080
Hope that is helpful. If so, feel free to add a comment so the team can see.
Hope that is helpful. If so, feel free to add a comment so the team can see.
Due to that issue being closed maintainers assumed it was fixed and went around closing related issues without testing if the problems were fixed.
https://gitlab.com/gitlab-org/gitlab/issues/198570#note_2936...
I have had a really negative experience reporting bugs to GitLab in the last 12 months. I have spent tons of time investigating bugs in new features only to have the devs ignore it. It feels like they are sweeping things under the rug so they can hit deadlines for officially announcing new features.
https://gitlab.com/gitlab-org/gitlab/-/issues/39534#note_283...
https://gitlab.com/gitlab-org/gitlab/issues/198570#note_2936...
I have had a really negative experience reporting bugs to GitLab in the last 12 months. I have spent tons of time investigating bugs in new features only to have the devs ignore it. It feels like they are sweeping things under the rug so they can hit deadlines for officially announcing new features.
https://gitlab.com/gitlab-org/gitlab/-/issues/39534#note_283...
Hello Jared!
GitLab Senior Backend Engineer - Verify (CI) here - Thanks for reaching out! I'll try to explain current situation of these issues.
1. https://gitlab.com/gitlab-org/gitlab/-/issues/198570
We know that this bug with DAG/needs is quite annoying. That's why we've been actively discussing about this bug in other issue: https://gitlab.com/gitlab-org/gitlab/-/issues/213080
Can you please take a look at the discussions there?
2. https://gitlab.com/gitlab-org/gitlab/-/issues/39534
Unfortunately we had no choice but to give an option to fix this behavior, instead of fixing it. That's because, if we fixed this bug immediately, it would probably break existing pipelines.
That's why we allowed users to fix this behavior themselves (https://gitlab.com/gitlab-org/gitlab/-/merge_requests/24605).
Hope this is helpful and please feel free to contact us if you have any questions/feedbacks. Thanks!
GitLab Senior Backend Engineer - Verify (CI) here - Thanks for reaching out! I'll try to explain current situation of these issues.
1. https://gitlab.com/gitlab-org/gitlab/-/issues/198570
We know that this bug with DAG/needs is quite annoying. That's why we've been actively discussing about this bug in other issue: https://gitlab.com/gitlab-org/gitlab/-/issues/213080
Can you please take a look at the discussions there?
2. https://gitlab.com/gitlab-org/gitlab/-/issues/39534
Unfortunately we had no choice but to give an option to fix this behavior, instead of fixing it. That's because, if we fixed this bug immediately, it would probably break existing pipelines.
That's why we allowed users to fix this behavior themselves (https://gitlab.com/gitlab-org/gitlab/-/merge_requests/24605).
Hope this is helpful and please feel free to contact us if you have any questions/feedbacks. Thanks!
These two issues illustrate that GitLab has a quality control problem. If devs aren't dog fooding these features, the community should be given more time to beta test them before the APIs are declared stable.
Hi! GitLab Community Advocate here - I agree that we need to get better how we improve the quality and performance of existing features/bugs. I am sorry for the negative experience you had. Please ping me on any long standing issues/bugs that are important to you and I'll do my best to route it to the directly responsible individuals.
For the new features part, going fast is a core part of our strategy, as is focusing on breadth over depth (https://about.gitlab.com/company/strategy/#breadth-over-dept...). Not only does this provide an easier path to collaborate and contribute for the wider community, it allows us to shorten the feedback loop with everyone that uses GitLab so that we can invest our time and effort in the areas that matter most to the wider community.Hope this is helpful.
For the new features part, going fast is a core part of our strategy, as is focusing on breadth over depth (https://about.gitlab.com/company/strategy/#breadth-over-dept...). Not only does this provide an easier path to collaborate and contribute for the wider community, it allows us to shorten the feedback loop with everyone that uses GitLab so that we can invest our time and effort in the areas that matter most to the wider community.Hope this is helpful.
I'm confused, a DAG is a model for the relationships between data. What makes this a separate feature for the product? And it's pretty fundamental to the problem you try to solve, usually, so what were they using before, a list? Tree?
> I'm confused, a DAG is a model for the relationships between data.
Here they're talking about a DAG as a model for the relationships between computation rather than data.
Many systems that model computation use DAGs. For example it's how many compilers understand your code when you compile it.
Here they're talking about a DAG as a model for the relationships between computation rather than data.
Many systems that model computation use DAGs. For example it's how many compilers understand your code when you compile it.
> Here they're talking about ... computation rather than data.
The lisp people would like a word...
In seriousness, computation is data, and this is perhaps especially obvious in a CI context where your “computation” is increasingly bash scripts in some string field inside of a YAML file.
The lisp people would like a word...
In seriousness, computation is data, and this is perhaps especially obvious in a CI context where your “computation” is increasingly bash scripts in some string field inside of a YAML file.
> The lisp people would like a word...
Why do you feel the need to say things in such a snarky way as this, as if I'm ignorant?
That was the point - that computation can also be represented in a 'data' structure like this. I already gave that exact example - a compiler representing computation as data.
Why do you feel the need to say things in such a snarky way as this, as if I'm ignorant?
That was the point - that computation can also be represented in a 'data' structure like this. I already gave that exact example - a compiler representing computation as data.
I think they're replying that way because you seem to miss my point, I didn't think I needed to say that "the operations are the data" explicitly.
My confusion was not on what they were doing or why they chose to use a DAG, but why they're presenting it this way and why they weren't using a DAG to begin with.
Like I've written task runners in various forms before. They've always been based on DAGs and topological sorting from the beginning. I wouldn't advertise it as such because the choice of model is obvious to me, I'd be more interested in reading about alternatives that provide other benefits.
My confusion was not on what they were doing or why they chose to use a DAG, but why they're presenting it this way and why they weren't using a DAG to begin with.
Like I've written task runners in various forms before. They've always been based on DAGs and topological sorting from the beginning. I wouldn't advertise it as such because the choice of model is obvious to me, I'd be more interested in reading about alternatives that provide other benefits.
Whoa. I wasn't trying to be snarky; if anything it was intended to be a humorous jab at how "computation is data" is such a thing in the lisp community.
I agree that your example illustrates the "computation is data" concept; I was just reconciling that with the apparent contradiction ("computation rather than data") earlier in your post.
I wasn't trying to call "gotcha" on you or anything; sometimes I say things that are unclear and I appreciate it when others help me clarify.
I agree that your example illustrates the "computation is data" concept; I was just reconciling that with the apparent contradiction ("computation rather than data") earlier in your post.
I wasn't trying to call "gotcha" on you or anything; sometimes I say things that are unclear and I appreciate it when others help me clarify.
They were using a list-of-sets; the CI pipeline is composed of stages which run in series, each of which runs a set of jobs in parallel. The new dependency DAG allows jobs to explicitly declare dependencies on each other and avoid waiting at stage boundaries.
Agreed. First sentence: "A Directed Acyclic Graph will let you run pipeline steps out of order." That's simply false.
Interpret "order" as "the order in which the steps are defined in your YAML file", and it's simply true.
I'm nitpicking but my point is that a DAG doesn't let you run anything. It might represent something, though, such as a bunch of tasks and a "must be executed before" relation. It's sloppy, imprecise use of language.
By that logic, no code lets you run anything, it just "represents" a program that actually runs. That's just pedantry.
Code runs and possibly lets you run something (for example in case of an OS or interpreter). I don't see a problem there.
[deleted]
Historically gitlab had a stages system: "Do all your builds, then all your tests, then all your deployments"
Which part of the explanation of what they mean in the article is unclear?
Sounds good. My guess as to why this has been late coming is because git monorepos aren't that popular. The seem to be growing in popularity as LFS and submodules stabilize.
Anyone want to share their experience with git and monorepos these days?
Anyone want to share their experience with git and monorepos these days?
They are rediscovering the basic features of Make et al.