The recent Trivy incident is a textbook example of how modern supply chain attacks work.
Not by exploiting your application.
Not by bypassing your firewall.
But by entering through your CI/CD pipeline—using tools you already trust.
This post breaks down:
- What actually happened (technically)
- Why it worked
- What signals we should have seen
- How to detect and contain similar attacks
1. The Attack Path: Step by Step
At a high level, the attack had three components:
A. Compromised Distribution
Attackers gained the ability to:
- Publish a malicious Trivy release
- Modify GitHub Action references (trivy-action, setup-trivy)
This is critical:
👉 They didn’t need to target your environment directly
👉 They targeted the distribution layer of a trusted tool
B. Exploiting Mutable References
Most pipelines reference GitHub Actions like this:
- name: Run Trivy scan
uses: aquasecurity/trivy-action@v1
The problem:
- @v1 is a mutable tag
- It can be repointed to a different commit at any time
Attackers leveraged this by:
- Updating tags to point to malicious commits
- Without changing the visible reference
👉 Pipelines continued to run — but executed different code
C. Execution in Trusted Context
Once the pipeline ran:
- The malicious action executed inside CI/CD
- Before or alongside the actual scan
This gave the attacker:
- Access to environment variables
- Access to filesystem
- Access to credentials and tokens
- Ability to make outbound network calls
👉 In other words: full access to the pipeline runtime
2. What the Malicious Code Likely Did
While specifics can vary, patterns in similar attacks include:
Credential Harvesting
- Reading environment variables (tokens, keys)
- Accessing:
- GitHub tokens
- cloud credentials (AWS, GCP, Azure)
- Docker registry credentials
Filesystem Scanning
- Looking for:
- SSH keys
- kubeconfig files
- .env files
- cached credentials
Memory Access (Advanced)
- Attempting to read process memory (/proc/…/mem)
- Extracting in-memory secrets
Data Exfiltration
- Sending collected data to attacker-controlled endpoints
- Often via:
- HTTPS requests
- DNS-based exfiltration
- disguised API calls
👉 All of this happens within a trusted execution boundary
3. Why This Worked
This attack succeeded because of three systemic gaps.
1. Implicit Trust in Tooling
Pipelines assumed:
- If it’s from a known repo, it’s safe
- If the name is unchanged, behavior is unchanged
There was no verification of:
- commit integrity
- publisher identity
- code changes behind tags
2. Use of Mutable References
Using:
@v1
@latest
means:
👉 You are not controlling what code executes
👉 You are delegating that control externally
SHA-pinned references (immutable commits) would not have been affected.
3. Over-Privileged Execution Environments
CI/CD pipelines often run with:
- Broad environment variables
- Long-lived credentials
- Network access
- File system access
This creates an ideal environment for:
- credential theft
- lateral movement
4. What Signals Should Have Been Detected
Even if the malicious code entered the pipeline, it still had to behave in detectable ways.
Here are the key signals:
A. Unexpected File Access
Security scanning tools typically:
- analyze images or code
- do not read arbitrary system files
Suspicious signals include:
- access to /home/runner/.ssh/
- reading kubeconfig files
- scanning .env or credential directories
B. Abnormal Secret Access
Indicators:
- access to credentials not required for the scan
- reading all environment variables indiscriminately
A scan step should not need:
- cloud admin credentials
- deployment tokens
C. Suspicious Network Activity
Watch for:
- outbound connections to unknown domains
- first-time connections during a pipeline run
- data transfers unrelated to expected tool behavior
D. Behavioral Mismatch
A key detection vector:
👉 The action says “run security scan”
👉 But behavior includes credential access + network exfiltration
This mismatch is a strong indicator of compromise.
3. Why This Worked
This attack succeeded because of three systemic gaps.
1. Implicit Trust in Tooling
Pipelines assumed:
- If it’s from a known repo, it’s safe
- If the name is unchanged, behavior is unchanged
There was no verification of:
- commit integrity
- publisher identity
- code changes behind tags
2. Use of Mutable References
Using:
@v1
@latest
means:
👉 You are not controlling what code executes
👉 You are delegating that control externally
SHA-pinned references (immutable commits) would not have been affected.
3. Over-Privileged Execution Environments
CI/CD pipelines often run with:
- Broad environment variables
- Long-lived credentials
- Network access
- File system access
This creates an ideal environment for:
- credential theft
- lateral movement
4. What Signals Should Have Been Detected
5. Why Detection Is Hard
Despite these signals, most teams did not detect the attack in real time.
Why?
- CI/CD pipelines are not monitored like production systems
- Logs are often:
- incomplete
- not analyzed
- Behavior baselines are rarely defined
- Tool execution is treated as a black box
👉 We trust the tool, so we don’t question its behavior
6. Containment Strategies
Once detected, rapid containment is critical.
Key actions include:
Immediate Steps
- Stop affected pipelines
- Block outbound connections to known malicious endpoints
- Revoke all accessible tokens and credentials
Environment Cleanup
- Rebuild or reimage runners (especially self-hosted)
- Remove cached credentials
- Invalidate session tokens
Artifact Verification
- Identify artifacts built during exposure window
- Rebuild from trusted sources
7. Prevention: What Should Have Been Done
A. Enforce Immutable References
- Use commit SHAs instead of tags
- Use image digests instead of latest
B. Verify Provenance
- Validate source and publisher
- Use signed artifacts where possible
C. Restrict Pipeline Privileges
- Minimize exposed secrets
- Use short-lived credentials
- Apply least privilege
D. Monitor Runtime Behavior
- Track file access patterns
- Monitor network activity
- Detect abnormal behavior in CI/CD
8. The Bigger Lesson
This incident is not just about Trivy.
It reflects a broader shift:
The CI/CD pipeline is now a high-value attack surface.
And yet:
- It is highly automated
- It has privileged access
- It is lightly monitored
Final Thought
We have spent years improving how we detect vulnerabilities in code.
We now need to invest in understanding and controlling:
👉 what actually executes in our pipelines
Because in modern supply chain attacks,
execution is the vulnerability.
0 Comments