How the Trivy supply chain attack harvested credentials from secrets managers(vaultproof.dev)
vaultproof.dev
How the Trivy supply chain attack harvested credentials from secrets managers
https://vaultproof.dev/blog/trivy-supply-chain-attack
16 comments
Well...obviously secrets available in the runtime environment of a CI job are vulnerable to attacks that can compromise the runtime environment. I think everyone knew that. Also GitHub actions that come from less than unreproachable sources (GitHub themselves, ?) have always been an obvious attack vector. In places I've worked where we were concerned about this we forked all the actions repos into our own org so we could never pick up mystery meat in our jobs.
You're right and forking actions is the correct mitigation.
The gap is operational discipline. Most teams know they should fork upstream actions and review updates before pulling them in. Almost none actually do it consistently. The Trivy attack is useful not because it revealed something unknown but because it made the abstract cost of that gap concrete.
The gap is operational discipline. Most teams know they should fork upstream actions and review updates before pulling them in. Almost none actually do it consistently. The Trivy attack is useful not because it revealed something unknown but because it made the abstract cost of that gap concrete.
The technical breakdown of the Trivy attack is a great reminder that our security is only as strong as our version pinning. We all know we should pin to SHA-256 hashes instead of mutable tags, but the UX for managing and updating those hashes is still painful enough that most teams default to tags. Until the tooling makes 'doing the right thing' as easy as @v1, these supply chain leaks will continue to be a high-ROI path for attackers.
Exactly. It is a UX problem as much as a security problem.
Leaving aside the fact that this is an ad thinly veiled as an article, OneCli does the same, and recently NanoClaw made OneCli setup their default.
Fair criticism on the framing and it reads more promotional than I intended.
Took a look at OneCLI after your comment. The approaches are different.
OneCLI stores the real key encrypted and decrypts it at request time, injecting it as a header through their gateway. The full key exists in plaintext at the moment of injection.
VaultProof splits the key into cryptographic shares using Shamir Secret Sharing. No complete key exists anywhere at rest. The proxy reconstructs it transiently for the duration of the API call then zeros it immediately.
Different trust models and different threat coverage. OneCLI is a solid approach for agent credential management. The Shamir splitting is specifically for teams where the key existing as plaintext even transiently on a third party server is a concern.
Took a look at OneCLI after your comment. The approaches are different.
OneCLI stores the real key encrypted and decrypts it at request time, injecting it as a header through their gateway. The full key exists in plaintext at the moment of injection.
VaultProof splits the key into cryptographic shares using Shamir Secret Sharing. No complete key exists anywhere at rest. The proxy reconstructs it transiently for the duration of the API call then zeros it immediately.
Different trust models and different threat coverage. OneCLI is a solid approach for agent credential management. The Shamir splitting is specifically for teams where the key existing as plaintext even transiently on a third party server is a concern.
Author here. Built VaultProof after analyzing the Trivy attack
the credential harvesting worked specifically because the keys existed
as plaintext in the CI/CD environment after retrieval from the secrets
manager. Happy to go deep on the Shamir architecture or the attack
mechanics if useful.
Why use a Shamir architecture at all, instead of giving the CI run an ephemeral token that will be exchanged on the proxy?
Ephemeral tokens are a valid approach and some systems use exactly that.
The difference with Shamir is what happens if the proxy itself is compromised. With token exchange the proxy holds or can reconstruct the real key server side. A compromised proxy is game over for the credential.
The difference with Shamir is what happens if the proxy itself is compromised. With token exchange the proxy holds or can reconstruct the real key server side. A compromised proxy is game over for the credential.
Can you explain what this is please? "Exploits mutable Git tags and self-declared commit identity"
Two things combined.
First: Git tags are not immutable. When you write actions/[email protected] in your pipeline you are not pinning to a fixed commit. The tag is just a pointer and whoever controls the repo can silently move it to point to different code. Most teams assume a version tag means a fixed version. It does not.
Second: Git does not verify who makes a commit. Anyone can set their name to any Aqua Security developer they want. The malicious commit looked like it came from a trusted author because Git has no identity enforcement at all.
The practical fix for the first problem is pinning to a full commit hash instead of a tag name. That hash cannot be moved.
Almost nobody does this by default which is why the attack worked at scale. its very common supply chain failure pattern.
First: Git tags are not immutable. When you write actions/[email protected] in your pipeline you are not pinning to a fixed commit. The tag is just a pointer and whoever controls the repo can silently move it to point to different code. Most teams assume a version tag means a fixed version. It does not.
Second: Git does not verify who makes a commit. Anyone can set their name to any Aqua Security developer they want. The malicious commit looked like it came from a trusted author because Git has no identity enforcement at all.
The practical fix for the first problem is pinning to a full commit hash instead of a tag name. That hash cannot be moved.
Almost nobody does this by default which is why the attack worked at scale. its very common supply chain failure pattern.
> The practical fix for the first problem is pinning to a full commit hash instead of a tag name
If the underlying project in turn uses named tags, i.e. if the hash pinning doesn't apply transitively, then the protection appears incomplete, doesn't it?
If the underlying project in turn uses named tags, i.e. if the hash pinning doesn't apply transitively, then the protection appears incomplete, doesn't it?
Correct. As an attacker you just move one level deeper.
If the target pins their direct actions to commit hashes you compromise a dependency of the action instead. They pinned the top of the tree but you own something in the middle of it.
SolarWinds was not attacked directly. The attackers compromised Orion, a build tool SolarWinds depended on. SolarWinds had decent security on their own code. It did not matter because the attack came through a dependency they trusted and did not control.
The defender has to secure the entire chain. The attacker only has to find one weak link anywhere in it. That asymmetry is why supply chain attacks keep working.
If the target pins their direct actions to commit hashes you compromise a dependency of the action instead. They pinned the top of the tree but you own something in the middle of it.
SolarWinds was not attacked directly. The attackers compromised Orion, a build tool SolarWinds depended on. SolarWinds had decent security on their own code. It did not matter because the attack came through a dependency they trusted and did not control.
The defender has to secure the entire chain. The attacker only has to find one weak link anywhere in it. That asymmetry is why supply chain attacks keep working.
Um ok, but the "setting the name to any string" is not the real problem, which is that an attacker had the ability to write to the repository at all, regardless of the name they choose, no?
As I mentioned below, another mitigation for this kind of supply chain attack is to fork the action repos into your own organization to allow tighter control over their content.
As I mentioned below, another mitigation for this kind of supply chain attack is to fork the action repos into your own organization to allow tighter control over their content.
Regarding GitHub actions and it's secret manager. Any decently organized company would do well to stay away from well known secret interfaces.
Instead use oidc auth to fetch secrets just in time, all short-lived for the duration of the pipeline.
OIDC is the right call for cloud provider credentials like AWS, GCP,
Azure all support it well and short lived tokens are genuinely better
than static keys.
The gap is third party APIs. OpenAI, Stripe, Anthropic, GitHub. None of them support OIDC. You still end up with a static API key that has to exist somewhere in the pipeline as a plaintext string. That is exactly what the Trivy payload targeted.
The gap is third party APIs. OpenAI, Stripe, Anthropic, GitHub. None of them support OIDC. You still end up with a static API key that has to exist somewhere in the pipeline as a plaintext string. That is exactly what the Trivy payload targeted.