Use GitHub actions at your own risk(julienrenaux.fr)
julienrenaux.fr
Use GitHub actions at your own risk
https://julienrenaux.fr/2019/12/20/github-actions-security-risk/
64 comments
While I understand the "fork everything you use" policy, github should make this process easier. Even though it is "free", it pollutes one's account/profile with hundreds of packages that are not really 'forks' with independent or alternate functionality.
Many automated vendor/package management tools will not automate pulling newer changes from upstream. If this was instead a simple a read-only version/tag, it is easy for those tools to provide upgrade/update path.
Many automated vendor/package management tools will not automate pulling newer changes from upstream. If this was instead a simple a read-only version/tag, it is easy for those tools to provide upgrade/update path.
A reasonably good solution to mitigate the “profile pollution” is to create a Github organization for these kinds of forks (I use <username>-forks)
As a side effect this adds more organization icons to your Github profile, making it appear as if you're such an important OSS contributor that you're an official member of the Username Forks Alliance :)
Do you fork your own programming languages too?
We definitely standardize runtime and compiler versions across runtime environments and build servers and have planned upgrade paths with rollback options.
Not so much a fork as a package cache - but usually for anything important, absolutely.
Yes. That is, you check in the binaries.
Do I do this for hobby projects? No, of course not. If my toy projects break, meh. But for production? Where breakage actually costs money? Damn right I do.
It's not only breakage, though. You want a hermetic build system, so you can build software reproducibly and consistently. There are few things more "fun" than building "the same" code on two machines, and one executable has a bug while the second doesn't. (OK. Compiler bugs are even more fun)
Do I do this for hobby projects? No, of course not. If my toy projects break, meh. But for production? Where breakage actually costs money? Damn right I do.
It's not only breakage, though. You want a hermetic build system, so you can build software reproducibly and consistently. There are few things more "fun" than building "the same" code on two machines, and one executable has a bug while the second doesn't. (OK. Compiler bugs are even more fun)
That's the sort of tough-guy prepper-with-a-slide-rule speech that people (apparently) just can't get enough of. But does it mean anything?
There is no bright line delimitating what's on the "critical path" and what's not. Your servers? The electric grid? The datacenter? The backbone(s) it's connected to? The judicial system making contracts enforceable? NATO? Name any two that straddle the line, and anybody can come up with something in between them, because it's a continuum.
There is no bright line delimitating what's on the "critical path" and what's not. Your servers? The electric grid? The datacenter? The backbone(s) it's connected to? The judicial system making contracts enforceable? NATO? Name any two that straddle the line, and anybody can come up with something in between them, because it's a continuum.
Welcome to a world were engineers aren't paid to copy code from Stack Overflow, but to make actual judgment calls.
What's the chance of the grid going down? What's the cost of it? Is the expected value significantly higher than a power generator? Congrats, you just bought a power generator.
Backbone down would cause you huge financial losses? Well, whaddaya say, let's build more backbone.[1]
Any redundancy that costs significantly less than the losses it mitigates will be built. The only reason we're not seeing a private NATO yet is that it costs more than pretty much any company makes. But 5,000 people security forces? Damn skippy those exist[2]
[1] https://engineering.fb.com/data-center-engineering/building-... [2] https://www.businessinsider.com/facebook-off-duty-police-off...
What's the chance of the grid going down? What's the cost of it? Is the expected value significantly higher than a power generator? Congrats, you just bought a power generator.
Backbone down would cause you huge financial losses? Well, whaddaya say, let's build more backbone.[1]
Any redundancy that costs significantly less than the losses it mitigates will be built. The only reason we're not seeing a private NATO yet is that it costs more than pretty much any company makes. But 5,000 people security forces? Damn skippy those exist[2]
[1] https://engineering.fb.com/data-center-engineering/building-... [2] https://www.businessinsider.com/facebook-off-duty-police-off...
The lack of a bright line doesn't make the idea meaningless. It's just a judgement call.
It's not unreasonable to be more concerned about an open source library disappearing than say, postgresql or mysql disappearing. No macho posturing required.
It's not unreasonable to be more concerned about an open source library disappearing than say, postgresql or mysql disappearing. No macho posturing required.
But people who complain about this stuff usually don't frame it that way-- absolutist statements like from the GP: "Anything you use in a critical path you should control." are common. The message is typically "I realize something fundamental that you don't and I'm owning all my dependencies and you don't own any of yours," rarely "I made a different choice about dependency ownership and here are the reasons I think that's appropriate for you too."
You seemed to take a pretty holistic, system-wide view of "critical path" which is fine. Do we know if the other person meant that?
Let's say I'm working front-end web development. My "critical path" might include: js bundler, some kind of application framework, various libs, etc. The scope of my work (and authority/responsibility) informs what I can reasonably have under my control.
Let's say I'm working front-end web development. My "critical path" might include: js bundler, some kind of application framework, various libs, etc. The scope of my work (and authority/responsibility) informs what I can reasonably have under my control.
GitHub could completely disappear and it wouldn't cause mass societal chaos or the breakdown of civilization. Those other things you listed, if they disappeared, there's quite a good chance they would cause mass societal chaos or the breakdown of at least certain civilizations. If the judicial system suddenly disappears, I reckon we'll have other things to worry about besides how it affects our build-chains.
Don’t forget payments, which are arguably the most critical part of any software that costs money.
> Anything you use in a critical path you should control.
https://en.wikipedia.org/wiki/Not_invented_here
?
> It's not a programming library where optimizations are created or we fixed a security vulnerability (though I suppose that can come up ).
But it's a service provided by a paid-for entity which has support which you pay for. If it doesn't work, you let support know....
https://en.wikipedia.org/wiki/Not_invented_here
?
> It's not a programming library where optimizations are created or we fixed a security vulnerability (though I suppose that can come up ).
But it's a service provided by a paid-for entity which has support which you pay for. If it doesn't work, you let support know....
Unlike git commit hashes, docker image tags can be overwritten by the maintainer at any time, even if they map to a hit commit hash.
You can address image layers by sha256 hash.
Do a docker pull of the image and the log will tell you the hash. Then you can do docker pull myimage@sha256:abc123def456......
I usually use these in my deployments since they can't be modified, it will require a new deploy.
came here to say this.
My team and I work on Actions at GitHub and I just wanted to stop by and add a bit more context. We definitely understand the concerns the article brings up and it's actually why we recommend in our documentation using SHA references when consuming third-party Actions. We also introduced the concept of verified Actions that extend an Organization's verification to Actions in the marketplace. We know there's more to do here to help prevent any malicious Actions and we're planning to continue to iterate. Thanks so much for the feedback and keep it coming.
I’m excited about Actions generally, so I’m glad to hear your team is aware of the challenges of building a reliable ecosystem of third party dependencies by reference to repos.
Still, recommending git SHAs has real ergonomic and maintainability issues, and while it’s more defensive it doesn’t prevent left-pad style broken builds due to disappearing content.
And at the same time, GitHub is investing heavily in package management infrastructure. I don’t say this lightly because designing good dependency management systems is much more subtle and difficult than people typically recognize, but from where I sit this appears to be fundamentally a dependency management concern. It seems a shame not to try to put GitHub Package Registry to good work here. Is that something potentially on the horizon?
Still, recommending git SHAs has real ergonomic and maintainability issues, and while it’s more defensive it doesn’t prevent left-pad style broken builds due to disappearing content.
And at the same time, GitHub is investing heavily in package management infrastructure. I don’t say this lightly because designing good dependency management systems is much more subtle and difficult than people typically recognize, but from where I sit this appears to be fundamentally a dependency management concern. It seems a shame not to try to put GitHub Package Registry to good work here. Is that something potentially on the horizon?
This is an excellent point, and should solve the issue I put below about builds taking forever when pulling in a slow-building action.
If I'm using a marketplace action in my workflow, I want "foo/[email protected]" to be treated like a binary package dependency, not a build from sourcecode.
If I'm using a marketplace action in my workflow, I want "foo/[email protected]" to be treated like a binary package dependency, not a build from sourcecode.
All great points and I think you're onto something with the idea of using GitHub Packages. Definitely something we're considering, stay tuned.
In addition to the security aspect, it would be really great if you have the option to lock down or cache the Docker images used for actions. It was weird for me when I pulled in an action from the marketplace for a workflow, and now about 80% of my total build time is building of that action for each and every build I run (e.g. the Dockerfile is compiled for that action EVERY time I run my build). I should have the option to at least cache that built image for me after the first time I run a workflow.
It sounds like you should add an abstraction layer that makes the default depending on the SHA. Your product should be secure by default and it clearly is not right now.
Likely, you can still do this and migrate people fairly painlessly to a secure version.
Edit: even if you select a tag or something, it should use the underlying reference instead of the tag. You can add an escape hatch if you REALLY want to target the tag (there are obviously workflows where that makes sense), but it should be off by default and warm users of the security risks.
Likely, you can still do this and migrate people fairly painlessly to a secure version.
Edit: even if you select a tag or something, it should use the underlying reference instead of the tag. You can add an escape hatch if you REALLY want to target the tag (there are obviously workflows where that makes sense), but it should be off by default and warm users of the security risks.
The same could be said for any use of docker images. Seems a little unfair to single out Github.
That's definitely true, but given the sensitivity of having access to private source and secrets I think its fair to call out a warning.
couldn't a malicious docker image also be tooled to dump all of that stuff to an external destination?
Absolutely. CI systems tend to get broad access to everything sacred. Giving that level of access to community code is risky in the least.
Totally agreed. However, when do we stop making this mistake? I think it's worth a callout when a large organization designs a dependency management system with such an obvious flaw on the "happy path".
It should be possible to import specific docker imager versions into a private repository and use them for production.
With Gitlab.com:
https://docs.gitlab.com/ee/user/packages/container_registry/
https://docs.gitlab.com/ee/ci/docker/README.html
With Gitlab.com:
https://docs.gitlab.com/ee/user/packages/container_registry/
https://docs.gitlab.com/ee/ci/docker/README.html
Sorry, that whole github actions concept gives me the screaming heeby-jeebies.
We're really not even pulling the dependency locally, we're just going to run someone else's code from their repo, on github's server, with our data and secrets?
What could possibly go wrong with that? /s
We're really not even pulling the dependency locally, we're just going to run someone else's code from their repo, on github's server, with our data and secrets?
What could possibly go wrong with that? /s
This isn't much different than other CI/CD pipelines. Docker images come to mind.
This isn't much different than other SaaS CI/CD pipelines.
Right, such as travis, circleci, gitlab, etc. All of which are pretty popular, so clearly there is interest in SaaS CI/CD.
Yes, they are popular. But on-premises ones also exist, are popular too, and mostly do not have this one problem.
It's a relevant distinction on this context.
It's a relevant distinction on this context.
[deleted]
On-premise CI/CD may still (and often does) use someone else's code, which can expose data and secrets in the same way as outlined. The only difference is that it's on your own infrastructure.
There's something to be said for on-premise solutions, but they won't save you from a malicious maintainer.
There's something to be said for on-premise solutions, but they won't save you from a malicious maintainer.
How does this different from any other hosted CI/CD solution?
I am thinking specifically about: * Gitlab's CI * CircleCI * TeamCity
etc....
I am thinking specifically about: * Gitlab's CI * CircleCI * TeamCity
etc....
I think the main difference is actions is built around the idea of this ecosystem of "just include other people's code in your pipeline." With Team City or GitLab you're typically only running code you wrote in their respective configuration files.
That's super valuable because you get started faster - e.g. "oh here's how I can deploy a lambda" but brings up this security concern of running other people's code. As discussed in the article, this is something NPM solved after the left-pad debacle. And I am sure (I hope) that GitHub will either change the documentation or add features to prevent this.
That's super valuable because you get started faster - e.g. "oh here's how I can deploy a lambda" but brings up this security concern of running other people's code. As discussed in the article, this is something NPM solved after the left-pad debacle. And I am sure (I hope) that GitHub will either change the documentation or add features to prevent this.
Can you give them read only access?
This is correct, using a commit hash is safer.
However, you'll sometimes want to update the commit hash after the action's maintainer releases new code. If you don't then you run the risk of running an old action that has vulnerable components or bugs.
The easiest workflow for that should be Dependabot's updater for GitHub actions: https://dependabot.com/github-actions/ Has anyone tried that approach for GitHub actions?
You still need to review the action's code before you use it, and every time you update to a new commit hash. But this approach protects you from automatically running new malicious code pushed to master/re-tagged.
However, you'll sometimes want to update the commit hash after the action's maintainer releases new code. If you don't then you run the risk of running an old action that has vulnerable components or bugs.
The easiest workflow for that should be Dependabot's updater for GitHub actions: https://dependabot.com/github-actions/ Has anyone tried that approach for GitHub actions?
You still need to review the action's code before you use it, and every time you update to a new commit hash. But this approach protects you from automatically running new malicious code pushed to master/re-tagged.
> Each hash is supposed to be unique and you cannot rewrite history with the exact same SHA-1.
I thought SHA-1 wasn't considered secure?
I thought SHA-1 wasn't considered secure?
There are collision attacks against SHA-1. But the currently known collision attacks aren't a big problem for git for a variety of reasons.
Git uses hashes that include more data than just the hash of the content of an object. Meaning the existing known collisions don't work and you'd have to develop collisions specifically to target git.
Additionally, the current known collision attacks require generating the two colliding contents as a pair. You can't take an existing document and then generate a collision for that document. Further, the technique is not particularly good for text documents because you need to include some data to force the collision. This is easily missed in binary documents but not the text documents most people are committing into git repos (granted you can put binary files into git and that would still work but the other reasons for this not being very useful still apply).
Git also has code to deal with possible collisions and will fall back to full comparison and will fail a commit that creates a collision. Finally, generating these document pairs leaves detectable patterns in the content. GitHub scans for these patterns as part of their process.
https://github.blog/2017-03-20-sha-1-collision-detection-on-...
Git uses hashes that include more data than just the hash of the content of an object. Meaning the existing known collisions don't work and you'd have to develop collisions specifically to target git.
Additionally, the current known collision attacks require generating the two colliding contents as a pair. You can't take an existing document and then generate a collision for that document. Further, the technique is not particularly good for text documents because you need to include some data to force the collision. This is easily missed in binary documents but not the text documents most people are committing into git repos (granted you can put binary files into git and that would still work but the other reasons for this not being very useful still apply).
Git also has code to deal with possible collisions and will fall back to full comparison and will fail a commit that creates a collision. Finally, generating these document pairs leaves detectable patterns in the content. GitHub scans for these patterns as part of their process.
https://github.blog/2017-03-20-sha-1-collision-detection-on-...
You hand-wave the difference between collision resistance and second-preimage resistance most away...
Just a recap: generating two files who's hashes collide takes 2^80 tries if SHA-1 wasn't broken, whereas finding a file with the same hash as a specific, externally-determined file takes 2^160 tries.
Just a recap: generating two files who's hashes collide takes 2^80 tries if SHA-1 wasn't broken, whereas finding a file with the same hash as a specific, externally-determined file takes 2^160 tries.
It's not secure for many applications. There are various practical collision attacks on it.
However, to rewrite history you need a preimage attack. In other words, you need to find a value X which hashes to the same hash value y which already exists h(X)=y
These are not currently practical. So SHA1 is certainly not recommended, but isn't completely broken yet.
However, to rewrite history you need a preimage attack. In other words, you need to find a value X which hashes to the same hash value y which already exists h(X)=y
These are not currently practical. So SHA1 is certainly not recommended, but isn't completely broken yet.
A known collision brings a hash algorithm to be "not secure" for cryptographic use. It doesn't mean there is a reliable way to come up with an equivalent hash for different data.
That said, hashes like md5 are still plenty used in ETags for http 304s all the time despite being considered insecure for cryptographic use.
That said, hashes like md5 are still plenty used in ETags for http 304s all the time despite being considered insecure for cryptographic use.
Depends on what attacks are relevant for your application. Even MD5 is still secure for some use cases. https://security.stackexchange.com/a/19908
It's insecure for certain applications.
Here is Linus on SHA-1 in git: https://www.theregister.co.uk/2017/02/26/git_fscked_by_sha1_...
Here is Linus on SHA-1 in git: https://www.theregister.co.uk/2017/02/26/git_fscked_by_sha1_...
There are collision attacks against it but GitHub should be detecting and protecting against those. See https://github.blog/2017-03-20-sha-1-collision-detection-on-...
See https://marc.info/?l=git&m=148787047422954 for some of linus thoughts about this.
It isn’t, according to NIST.
Title should be adjusted to mention branches & tags specifically.
I think GitHub is making it extremely easy to make this mistake by making the Actions Marketplace so easily accessible when you're editing a workflow in their UI. This is horrible design. They should at least point out the real consequences of pulling in these other actions and letting you fork right there.
I mean, there is an action in the marketplace that runs git-crypt to decrypt your files. With this issue, no user should ever be using this action without forking it to their repo first.
I mean, there is an action in the marketplace that runs git-crypt to decrypt your files. With this issue, no user should ever be using this action without forking it to their repo first.
It's to reduce the friction to lock-in. GitHub should Just Work™, regardless if that sacrifices security, because the smoother the experience the more processes you'll build around it, so the harder it'll be to migrate away from it.
Looks like you can now run actions on your own servers instead https://help.github.com/en/actions/automating-your-workflow-... which might help with things quite a bit considering that you shouldn't have to expose any secrets if you are running them on your own machines
One thing I'm missing from Actions is a way to manually trigger an action from the UI.
Currently the closest equivalents are making a request via the API (but then you need a token with it, and all the security concerns it opens) or sending a special message using an empty commit (but I haven't found a simple way to do that from inside VSCode, unless I missed it somewhere).
Currently the closest equivalents are making a request via the API (but then you need a token with it, and all the security concerns it opens) or sending a special message using an empty commit (but I haven't found a simple way to do that from inside VSCode, unless I missed it somewhere).
if you're going to use memes in a blog post at least use them correctly
What meme would you use then?
Instead of an incorrectly applied tapping head meme I would have used the Big Brain meme to sarcastically suggest that using a package’s reported version on the marketplace was smarter than a git SHA
I guess the same holds for Azure DevOps plugins, over which users don’t even have control over the version used. So what’s preventing plugin owners from doing something harmful?
What about hash collision?
You could write an obfuscator to keep appending useless code or comments to the payload to continually generate commit hashes until you get a collision with the same original hash.
A lot of work of course, and the hash is so f-ing huge it might take a few thousand compute-years, but you know, hackers find a way.
If nothing else, use the full 40-char hash to ensure that your attacker has to find the same atom twice across 10 moons.
Anything you use in a critical path you should control. If you're managing your own fork, nobody will inject bad code except yourself.
Just my 2 cents. Also, the things that github 3rd party actions are doing is usually not that complicated. I mean how many different ways can you publish a docker image or deploy an artifact to S3. Once it works, it either works or doesn't. It's not a programming library where optimizations are created or we fixed a security vulnerability (though I suppose that can come up ).
Sorry minor rant. food for thought.