Run end-to-end tests faster with Firecracker(webapp.io)
webapp.io
Run end-to-end tests faster with Firecracker
https://webapp.io/blog/github-actions-10x-faster-with-firecracker/
60 comments
> You can also get comparable performance out of https://buildkite.com/ which lets you self-host runners on AWS
you can self-host github runners as well, with a few caveats, the most serious one being that then you are responsible for cleaning up the state of your self-hosted runner between runs
https://docs.github.com/en/actions/hosting-your-own-runners/...
structural isolation guarantees of the form "build execution during run N cannot possibly impact build execution of run N+1" are tremendously helpful -- they reduce the number of weird CI failures and the cost to triage and fix each weird CI failure (by reducing the space of possible interactions). If you cannot offer similar guarantees when self hosting your own CI infrastructure then it may not be wise to self host.
you can self-host github runners as well, with a few caveats, the most serious one being that then you are responsible for cleaning up the state of your self-hosted runner between runs
https://docs.github.com/en/actions/hosting-your-own-runners/...
structural isolation guarantees of the form "build execution during run N cannot possibly impact build execution of run N+1" are tremendously helpful -- they reduce the number of weird CI failures and the cost to triage and fix each weird CI failure (by reducing the space of possible interactions). If you cannot offer similar guarantees when self hosting your own CI infrastructure then it may not be wise to self host.
> structural isolation guarantees of the form "build execution during run N cannot possibly impact build execution of run N+1" are tremendously helpful
Yes although the flip side is that it can make caching much less efficient. My experience is that caching layer can often take several minutes to download, dominating CI run time, whereas this might be instant on a self-hosted box that just leaves the previous build artefacts in place.
Yes although the flip side is that it can make caching much less efficient. My experience is that caching layer can often take several minutes to download, dominating CI run time, whereas this might be instant on a self-hosted box that just leaves the previous build artefacts in place.
I tried to get docker layer caching working within GHA for a second benchmark, but it seems like none of the approaches work particularly well for a "docker-compose build" - I'd happily amend the post with a second benchmark if you wouldn't mind opening a PR based on the existing one [1]
https://github.com/webappio/livechat-example/blob/be7c9121c1...
The point still stands for 2c - you can super easily parallelize with firecracker (by taking a snapshot of the state right before the test runs, then loading it a bunch of times)
https://github.com/webappio/livechat-example/blob/be7c9121c1...
The point still stands for 2c - you can super easily parallelize with firecracker (by taking a snapshot of the state right before the test runs, then loading it a bunch of times)
I did something similar w/VMware nearly 8-10 years ago, just separate storage from OS in your thinking and automate provisioning as needed.
It’s possible to do something similar with ec2 but if you want speed then move your build farm on-prem to utilise storage provider technologies eg https://tintri.com/blog/. Portworx+Stork would probably get you most of the way if you went down the k8 route but not something I’ve seen or look at in detail.
In my experience if your OS boot time is your main bottle neck then you are likely over optimising before expanding your testing environments. Fun but not the best use of development time, it is easier to have a minimum amount of system online before they are needed and just attach new storage to them as needed.
It’s possible to do something similar with ec2 but if you want speed then move your build farm on-prem to utilise storage provider technologies eg https://tintri.com/blog/. Portworx+Stork would probably get you most of the way if you went down the k8 route but not something I’ve seen or look at in detail.
In my experience if your OS boot time is your main bottle neck then you are likely over optimising before expanding your testing environments. Fun but not the best use of development time, it is easier to have a minimum amount of system online before they are needed and just attach new storage to them as needed.
I have to mention GitLab here. Their runners are extremely easy to self-host.
There’s an even faster strategy than this and it’s easier to setup.
You’re going to deploy 4 CI pipelines (so make sure you’re not manually putting together ci pipelines configs, use automation):
Pipeline 1: A conveyor belt of environments. All this pipeline does is spin up fresh environments then run a short automated smoke test. Hydrate the env with the most recent mask from prod. The trigger condition is there’s less than <Threshold> environments available. I did 8 on a whim and never saw a need to change it.
Pipeline 2: Normal garden variety CI pipeline triggered on merges to main. Output of this will be two artifacts persisted: a built package and your unit test evidence
Pipeline 3: Test your automated deployment by deploying the package build from #2 into the first of the queue of free envs from #1 trigger your end to end and integration and contract tests. Don’t run your security or operability tests here.
Pipeline 4: Async pipeline triggered on a 6hr schedule, do your long running stuff like fuzz testing here, your security tests etc. do these outside of the dev cycle.
Release candidates can only be signed after a successful run through 2, 3 & 4. That means prod deploys are on a predictable cadence which users and ops are usually appreciative of rather than we drop it in when it’s ready.
The DevEx is pretty sweet - you don’t see pipeline 1 or 4 in your build loop. Only the runtime of 3 would be comparable to the article - slightly faster than the article because no firecracker bringup overhead, no matter how small that is.
You’re going to deploy 4 CI pipelines (so make sure you’re not manually putting together ci pipelines configs, use automation):
Pipeline 1: A conveyor belt of environments. All this pipeline does is spin up fresh environments then run a short automated smoke test. Hydrate the env with the most recent mask from prod. The trigger condition is there’s less than <Threshold> environments available. I did 8 on a whim and never saw a need to change it.
Pipeline 2: Normal garden variety CI pipeline triggered on merges to main. Output of this will be two artifacts persisted: a built package and your unit test evidence
Pipeline 3: Test your automated deployment by deploying the package build from #2 into the first of the queue of free envs from #1 trigger your end to end and integration and contract tests. Don’t run your security or operability tests here.
Pipeline 4: Async pipeline triggered on a 6hr schedule, do your long running stuff like fuzz testing here, your security tests etc. do these outside of the dev cycle.
Release candidates can only be signed after a successful run through 2, 3 & 4. That means prod deploys are on a predictable cadence which users and ops are usually appreciative of rather than we drop it in when it’s ready.
The DevEx is pretty sweet - you don’t see pipeline 1 or 4 in your build loop. Only the runtime of 3 would be comparable to the article - slightly faster than the article because no firecracker bringup overhead, no matter how small that is.
There are times when some corner of software development speaks a specialized language and this is an example.
1. Conveyor belt(?) of environments. Hydrate(?) the env(ironment). Mask(?) from prod(uction)
2. I think I got this. Typical "merge to main pipeline" with built product and test results as outputs.
DevEx(?). And not sure why I wouldn't see pipeline #4 in my build loop because I can't deploy unless 2, 3 and 4 pass.... Maybe you mean I don't wait to see it.
Also not sure how it's faster because environments still need to be brought up. Unless you are trying to say that the environment is already running when the merge to master pipeline succeeds.
1. Conveyor belt(?) of environments. Hydrate(?) the env(ironment). Mask(?) from prod(uction)
2. I think I got this. Typical "merge to main pipeline" with built product and test results as outputs.
DevEx(?). And not sure why I wouldn't see pipeline #4 in my build loop because I can't deploy unless 2, 3 and 4 pass.... Maybe you mean I don't wait to see it.
Also not sure how it's faster because environments still need to be brought up. Unless you are trying to say that the environment is already running when the merge to master pipeline succeeds.
> Unless you are trying to say that the environment is already running when the merge to master pipeline succeeds.
I read this as implying that there would be a "warm pool" of environments ready to go at any given time. This is an approach I've used in the past when environment construction was expensive (e.g. Windows servers), but there are certainly tradeoffs.
I read this as implying that there would be a "warm pool" of environments ready to go at any given time. This is an approach I've used in the past when environment construction was expensive (e.g. Windows servers), but there are certainly tradeoffs.
Ahh :-)
>> Conveyor belt
Not a technical term, i just meant a repeatable stream of environments being created
>> Hydrate
This one is a pretty common term, if you hydrate a schema then you just populate it with data
>> Mask
In many industries (health, finance, even more since GDPR and the ilk) production data can be sensitive, it might have Personally Identifiable Information for example. Masking is the process of taking a value from production and obscuring it but with plausible-looking data. It can be time consuming and gnarly.
>> DevEx
Developer Experience - usually i say DX but i stopped myself on that one when i was writing the comment!
>> not sure why I wouldn't see pipeline #4 in my build loop
It's part of your deployment loop for sure, but there's no reason it has to be part of your inner dev loop. You can continue on with your next task and be confident you'll start receiving telemetry from prod for your new feature's behaviour soon.
>> Unless you are trying to say that the environment is already running when the merge to master pipeline succeeds
Exactly this. The time for creating that env was spent async outside of your inner dev loop.
>> Conveyor belt
Not a technical term, i just meant a repeatable stream of environments being created
>> Hydrate
This one is a pretty common term, if you hydrate a schema then you just populate it with data
>> Mask
In many industries (health, finance, even more since GDPR and the ilk) production data can be sensitive, it might have Personally Identifiable Information for example. Masking is the process of taking a value from production and obscuring it but with plausible-looking data. It can be time consuming and gnarly.
>> DevEx
Developer Experience - usually i say DX but i stopped myself on that one when i was writing the comment!
>> not sure why I wouldn't see pipeline #4 in my build loop
It's part of your deployment loop for sure, but there's no reason it has to be part of your inner dev loop. You can continue on with your next task and be confident you'll start receiving telemetry from prod for your new feature's behaviour soon.
>> Unless you are trying to say that the environment is already running when the merge to master pipeline succeeds
Exactly this. The time for creating that env was spent async outside of your inner dev loop.
May I ask what stack you employ to meet these goals?
Many tend to reach for Gitlab CI or Github Actions but these piles of "executable yaml" never appear to be up to the task of complex deployment logic you describe in your post, not including that they don't account for multi-repo or composed artifact workflows naturally. The state of the art, if you can call it that, is Jenkins where you can drop into raw-ish groovy/java for the logic pieces when you need to. But then you run into the constant struggle of working around Jenkin's leaky abstractions and peculiarities.
You can patch together a pile of bash, python, go et al but you land in a worse place where there is no guiding structure to the automation for onboarding, enhancement, and maintenance.
I'm curious of other's experiences building complex build / deployment pipelines where up-front you have consistent entry structure to the automation but have all the escape hatches one would need to implement custom logic when required, in a type safe, potentially compiled, testable way (ie: pipelines as 'actual' code).
Of course one could write their own automation engine to avoid yaml hell and all that. However I am not seeing any pervasive solutions being presented that don't present "yet another (yaml | json | xml | cue | whatever) task dag launching containers running random scripts from wherever".
Many tend to reach for Gitlab CI or Github Actions but these piles of "executable yaml" never appear to be up to the task of complex deployment logic you describe in your post, not including that they don't account for multi-repo or composed artifact workflows naturally. The state of the art, if you can call it that, is Jenkins where you can drop into raw-ish groovy/java for the logic pieces when you need to. But then you run into the constant struggle of working around Jenkin's leaky abstractions and peculiarities.
You can patch together a pile of bash, python, go et al but you land in a worse place where there is no guiding structure to the automation for onboarding, enhancement, and maintenance.
I'm curious of other's experiences building complex build / deployment pipelines where up-front you have consistent entry structure to the automation but have all the escape hatches one would need to implement custom logic when required, in a type safe, potentially compiled, testable way (ie: pipelines as 'actual' code).
Of course one could write their own automation engine to avoid yaml hell and all that. However I am not seeing any pervasive solutions being presented that don't present "yet another (yaml | json | xml | cue | whatever) task dag launching containers running random scripts from wherever".
>> May I ask what stack you employ to meet these goals?
Nothing fancy :-) Ansible + Jenkins pipelines
>> these piles of "executable yaml" never appear to be up to the task of complex deployment
Agreed, there's an untapped market here - somewhere between your enterprise GitOps model like i'm alluding to (usually heavily integrated with Kubernetes) but repackaged for smaller scale use.
Nothing fancy :-) Ansible + Jenkins pipelines
>> these piles of "executable yaml" never appear to be up to the task of complex deployment
Agreed, there's an untapped market here - somewhere between your enterprise GitOps model like i'm alluding to (usually heavily integrated with Kubernetes) but repackaged for smaller scale use.
Firecracker is great and all, but the core idea here described works also with plain docker; i.e. there is nothing inherently firecracker specific to the basic technique
Yeah, I don’t like that the article itself treats building the DB seed data, etc, into the Firecracker VM image like this is impossible to do in Docker. The techniques are good things to do — but it’s very tenuous how the techniques are connected to Firecracker.
I’ve do all of the above using multi-layered Docker files and a cron CI job to rebuild the base integration test image every 6 hours. Sure if you need the isolation, Firecracker is the way to go. But if you invest primarily in container shenanigans to speed up CI with Docker, it’s not too much extra work to wrap it in a Firecracker VM, plain QEMU, or whatever once you start wanting more isolation.
Also, maybe I’m holding it wrong but Docker in Docker had not bitten us yet on our GitHub action runners.
I’ve do all of the above using multi-layered Docker files and a cron CI job to rebuild the base integration test image every 6 hours. Sure if you need the isolation, Firecracker is the way to go. But if you invest primarily in container shenanigans to speed up CI with Docker, it’s not too much extra work to wrap it in a Firecracker VM, plain QEMU, or whatever once you start wanting more isolation.
Also, maybe I’m holding it wrong but Docker in Docker had not bitten us yet on our GitHub action runners.
Yep, buildkit does incremental builds quite well
We find the dominating factor in (our) incremental builds / CI to be network/io caching, which has less to do with firecracker/docker and more with the surrounding hw/sw (gha topology & smarts, IO speed, ...). It's a real problem in GPU/AI CI where we get monster image sizes. There were some cool blog posts ~last year on caching and routing tricks happening at GH (joint with MSR?), but they've seemingly gone silent..
We find the dominating factor in (our) incremental builds / CI to be network/io caching, which has less to do with firecracker/docker and more with the surrounding hw/sw (gha topology & smarts, IO speed, ...). It's a real problem in GPU/AI CI where we get monster image sizes. There were some cool blog posts ~last year on caching and routing tricks happening at GH (joint with MSR?), but they've seemingly gone silent..
You don't need a management daemon running though, and get a complete virtualized kernel that can be customized if needed.
Ok, so IIUC, the main difference with firecracker versus docker is that processes are better separated from each other ("micro VM" instead of namespaces) and that one can run a customized kernel. But for e2e tests I've written, neither of these advantages mattered.
I do love the idea of taking a snapshot of a prebuilt database image and can see where this would really speed up the tests.
I do love the idea of taking a snapshot of a prebuilt database image and can see where this would really speed up the tests.
Author here!
The three big differences are:
1. Docker doesn't deal with running processes (like postgres or redis), only the filesystem state
2. Docker doesn't have enough isolation, so you'd probably need to run it within qemu or firecracker for compliance in bigger teams
3. Docker-in-docker is still pretty painful, if you need to do anything nonstandard like change the size of /dev/shm, access /dev/kvm, or load kernel drivers, it'll take custom configuration.
The three big differences are:
1. Docker doesn't deal with running processes (like postgres or redis), only the filesystem state
2. Docker doesn't have enough isolation, so you'd probably need to run it within qemu or firecracker for compliance in bigger teams
3. Docker-in-docker is still pretty painful, if you need to do anything nonstandard like change the size of /dev/shm, access /dev/kvm, or load kernel drivers, it'll take custom configuration.
I’m confused. Why do you need to snapshot live processes? Are we concerned about startup time of Postgres or whatever? Also, why is isolation needed for e2e tests? Lastly, why is docker-in-docker a requirement, and how is that easier than qemu in qemu or qemu in docker or whatever?
> Why do you need to snapshot live processes?
Often times there are long-living processes which rarely change but take a long time to warm up. The Bazel [1] agent for C++ projects, the buildkit [2] state for docker, or the running Postgres or Redis server for a cloud native app for example.
It's why running "docker build" twice on your laptop is so fast, but running "docker build" in CI seems glacially slow.
> why is docker-in-docker a requirement, and how is that easier than qemu in qemu or qemu in docker or whatever?
The example given was running "docker-compose build", so you'd need either docker-in-firecracker (this post), docker-in-docker, or docker-in-qemu. You'd almost never run docker-compose build on bare metal in practice, because you'd immediately need to send the images you built somewhere else in order to use them.
[1] https://bazel.build/ [2] https://docs.docker.com/develop/develop-images/build_enhance...
Often times there are long-living processes which rarely change but take a long time to warm up. The Bazel [1] agent for C++ projects, the buildkit [2] state for docker, or the running Postgres or Redis server for a cloud native app for example.
It's why running "docker build" twice on your laptop is so fast, but running "docker build" in CI seems glacially slow.
> why is docker-in-docker a requirement, and how is that easier than qemu in qemu or qemu in docker or whatever?
The example given was running "docker-compose build", so you'd need either docker-in-firecracker (this post), docker-in-docker, or docker-in-qemu. You'd almost never run docker-compose build on bare metal in practice, because you'd immediately need to send the images you built somewhere else in order to use them.
[1] https://bazel.build/ [2] https://docs.docker.com/develop/develop-images/build_enhance...
But that's state on disk, not process state. It should not affect startup time in buildkit.
I'm not experienced enough with Bazel to comment on that.
I'm not experienced enough with Bazel to comment on that.
> 1. Docker doesn't deal with running processes (like postgres or redis), only the filesystem state
Docker in practice usually doesn't, but in theory it can and should.
A checkpoint/restore demo, of live-migrating running docker containers between servers, including running process state, was shown at Dockercon 2015. I can't find a video right now, but it was demoed then.
The "docker checkpoint" command has existed since docker 1.13.0 (released 2017), which allows checkpointing and restoring the running state of containers, including running processes, with some limitations. https://docs.docker.com/engine/reference/commandline/checkpo...
The reality is just unfortunately that these features are experimental, have been for the last 7+ years, and have taken far longer to mature than would be expected.
Docker in practice usually doesn't, but in theory it can and should.
A checkpoint/restore demo, of live-migrating running docker containers between servers, including running process state, was shown at Dockercon 2015. I can't find a video right now, but it was demoed then.
The "docker checkpoint" command has existed since docker 1.13.0 (released 2017), which allows checkpointing and restoring the running state of containers, including running processes, with some limitations. https://docs.docker.com/engine/reference/commandline/checkpo...
The reality is just unfortunately that these features are experimental, have been for the last 7+ years, and have taken far longer to mature than would be expected.
Hi, offtopic but: is webapp.io a pivot from layerci, or just a rebranding?
Interesting that you're folks now use firecracker. I assume it now fills in adequately for the previously homegrown tech at layerci [0]?
[0] https://news.ycombinator.com/item?id=25979941
Interesting that you're folks now use firecracker. I assume it now fills in adequately for the previously homegrown tech at layerci [0]?
[0] https://news.ycombinator.com/item?id=25979941
Just a rebranding! (The technology's gotten better as well, of course - we didn't used to use firecracker at all)
https://webapp.io/blog/layerci-has-rebranded-to-webapp-io/
https://webapp.io/blog/layerci-has-rebranded-to-webapp-io/
Docker does handle snapshots of running processes.
It's called checkpoint/restore, it utilizes the CRIU tooling to do this.
In terms of doing this in a CI env like actions where you may have different types of machines serving you, it may be problematic as the machine specs need to pretty closely match.
In terms of doing this in a CI env like actions where you may have different types of machines serving you, it may be problematic as the machine specs need to pretty closely match.
Gee, why not just go straight to step 3 via fork/exec? Bound to shave off a few milliseconds beyond that 10x. And no firecracker required.
If you a cloud host, you need a way to sandbox hostile code. Firecracker allows you to do that (it is a configuration of the traditional KVM virtualization system except lighter and faster, instead of booting a VPS which can take minutes, you can now spawn one in under a second).
To clarify my post: I see the reason for Firecracker to exist in general, it's great. But does "e2e tests" include untrusted code? I think it really shouldn't.
So why use firecracker here? Invoking your tests in a bare VM or container is great for making sure that you are controlling the environment and enumerating your system dependencies. But this post proposes discarding those things and instead using some saved state as the entry point into your Firecracker. So now you are booting from Your Image instead of a { Official Distro Image + Dependency Recipe }. It seems like a step backward.
So why use firecracker here? Invoking your tests in a bare VM or container is great for making sure that you are controlling the environment and enumerating your system dependencies. But this post proposes discarding those things and instead using some saved state as the entry point into your Firecracker. So now you are booting from Your Image instead of a { Official Distro Image + Dependency Recipe }. It seems like a step backward.
Author here!
I think there's always been a push/pull of "fat base images" versus "install everything every time" - It's obviously subjective, but I think it's more important to run the tests on every commit than it is to start the environment from scratch.
It's also not necessarily mutually exclusive, you could have a "staging branch" where you make something that looks a lot like production and then re-run end-to-end tests there, while running the per-branch tests with this method to avoid slowing down developers.
I think there's always been a push/pull of "fat base images" versus "install everything every time" - It's obviously subjective, but I think it's more important to run the tests on every commit than it is to start the environment from scratch.
It's also not necessarily mutually exclusive, you could have a "staging branch" where you make something that looks a lot like production and then re-run end-to-end tests there, while running the per-branch tests with this method to avoid slowing down developers.
> But does "e2e tests" include untrusted code?
What other possible way could CI work?
What other possible way could CI work?
Continuous integration can -- and does -- happen with local development teams using their own equipment. In general, those are typically pretty trusting environments where Firecracker style isolation is not as critical.
Not just sandboxing, but just ensuring that each test runs in a clean environment, without interference from files/processes left behind by a previous or even concurrent test.
Because process isolation under unix is pretty lax. Processes have by default have all the rights of the user. And you might end up with a system different from the initial state
They seem to be comparing CI runner starting from scratch to always on VM with firecracker preconfigured.
Used to do something similar with vsphere a while back. The servers took ages to get into the right state to test so much easier to just revert to snapshot to get a clean state.
interesting, I wonder if you can use this with GitHub self hosted runners?
Interesting. What other cool things are people doing with Firecracker?
Fly built a whole platform with Firecracker VMs: https://fly.io/
I have been using Weave Ignite [1] recently to run Firecracker micro vm(s) instead of containers for a multitude of tasks!
1. https://github.com/weaveworks/ignite
1. https://github.com/weaveworks/ignite
Always amaze me to see the new trend of DevOps that will be happily following such a tutorial, wget and running random code from the internet in production...
I don't think this is production, this is for running your tests. Your code in the "tests haven't run yet" state probably leak all the secrets they have access to and destroy the machine they're running on, so you don't let them have any secrets and create a new machine each time. "curl | bash" here just injects potential flakiness (as does "npm install" when npm dies, etc.)
Obviously a lot of people treat their CI system as their CD system, and do things like letting tests have highly privileged access to their production k8s cluster. That's a terrible idea even if you aren't installing software with "curl | bash".
So overall, I don't think this is worth a HN comment to complain about. People are going to install software in non-auditable non-reproducible ways.
Obviously a lot of people treat their CI system as their CD system, and do things like letting tests have highly privileged access to their production k8s cluster. That's a terrible idea even if you aren't installing software with "curl | bash".
So overall, I don't think this is worth a HN comment to complain about. People are going to install software in non-auditable non-reproducible ways.
It always amazes me that people seem to think this is a _new_ trend.
Pretty disingenuous to compare building vs not building. Those firecracker containers need to be built too..
Sure it’s faster startup, but the rest is nonsense.
Sure it’s faster startup, but the rest is nonsense.
Great article. Firecracker has been an amazing addition to my toolkit and it is good to see succeeding in solving real world problems.
Sounds like having an actual non-ephemeral computer with extra steps...
But why does it require firecracker and not qemu?
QEMU takes much longer to save/restore snapshots, and it's much harder to do via the API
[deleted]
What does this e2e tests in webapp ?
I don't understand why you need to rebuild docker image every app build, this seems like really wasteful.
I don't understand why you need to rebuild docker image every app build, this seems like really wasteful.
If the app itself is part of the image you need to rebuild the image every time a dev wants to test their change.
Is that the same as redeploying your app to an existing container?
That doesn’t really have a meaning. Are you asking if it updates an existing Docker image, or even updates an existing Docker container (i.e. a running process)? If so, the answers would be ‘yes’ and ‘no’ respectively.
Example.
ImageA: ubuntu + apache + yourPhpApplication ImageB: ubuntu + apache
To deploy ImageA all you need to do is run the image. To deploy image B you need to run ImageB along with a copy of your application
Above I was describing ImageA.
ImageA: ubuntu + apache + yourPhpApplication ImageB: ubuntu + apache
To deploy ImageA all you need to do is run the image. To deploy image B you need to run ImageB along with a copy of your application
Above I was describing ImageA.
This has been done successfully using VMs since 2 decades.
1. It doesn't show off the unique capabilities of firecracker very well.
2. The comparison not very fair.
2a. The docker-build step (which dominates the runtime) is run without any caching, just by adding 2 lines to your build-push-action, "cache-from: type=gha, cache-to: type=gha,mode=max" you can make it a lot faster.
2b. ~1m20s of the time is just "VM start". GitHub Actions has had a rough time recently, but you should never wait that long to get your CI running in day-to-day operation.
2c. The tests are unrealistically short at 20s which allows the author to get to their 10x faster number.
Let's say the GitHub Action starts in 5 seconds, the GitHub Actions cache reduces the build time to 2 minutes and the tests take 10 minutes to run. Now Firecracker is 20% faster ...
You can also get comparable performance out of https://buildkite.com/ which lets you self-host runners on AWS meaning you're almost guaranteed to get a hot docker cache (running against locally attached SSDs). You can now start running your tests (almost) as fast with much more mature tooling.