AWS credentials in files not detected in git scan #2620

Constantin07 · 2024-03-25T17:25:19Z

Please review the Community Note before submitting

TruffleHog Version

3.72.0

Trace Output

Expected Behaviour

It should detect AWS credentials saved in files.

Actual Behavior

It doesn't detect them at all.

Steps to Reproduce

In a repository, create the .aws/credentials file and add the AWS creds like:

[default]
aws_access_key_id=****
aws_secret_access_key=****

Run trufflehog filesystem .
Below logs output is produced:

🐷🔑🐷  TruffleHog. Unearth your secrets. 🐷🔑🐷

2024-03-25T17:16:40Z    info-0  trufflehog      running source  {"source_manager_worker_id": "YnfOv", "with_units": true}
2024-03-25T17:16:40Z    info-0  trufflehog      finished scanning       {"chunks": 696, "bytes": 660010, "verified_secrets": 0, "unverified_secrets": 0, "scan_duration": "158.160668ms"}

It doesn't detect anything.

Environment

OS: MacOS
Version [Sonoma 14.4]

Additional Context

References

#0000

The text was updated successfully, but these errors were encountered:

mikedidomizio · 2024-04-01T00:48:22Z

Not sure if I'm having the same issue as @Constantin07 but adding to this since I'm also scanning AWS secrets in the filesystem and I can't quite understand the inconsistency with this tool.

I created an example repository and it doesn't pick up the secrets in the file or in the Git objects (it does in another repo).

I'm running a different version through docker:

docker run --platform linux/arm64 -v $PWD:/pwd trufflesecurity/trufflehog:latest --version

trufflehog 3.68.4

The following is the command I run, both in the repo it detects the AWS secret properly and the repo included here

docker run --platform linux/arm64 -v $PWD:/pwd trufflesecurity/trufflehog:latest filesystem /pwd

I don't quite understand what gets trufflehog to finally detect properly.

This comment may also be related.

rgmz · 2024-04-01T21:25:50Z

Out of curiosity, does this occur if you run trufflehog filesystem . --allow-verification-overlap?

It may be a similar issue as #2515; if not, I could try to modify the script debugging script in #2515 (comment) to see if there's a particular commit responsible.

mikedidomizio · 2024-04-01T23:47:51Z

Hey @rgmz, same unexpected result with that command.

I tried to find a good commit to start from and I couldn't find one, even back 1-2 years ago. I took the same approach as your bisect script but without the run, building the Docker image and running it against my repo. I was hoping I'd find something that way I could share it with you, but nothing.

I also double checked that the current AWS regex matches the fake AWS creds in that repo and it does.

Maybe you'll have better luck.

rgmz · 2024-04-02T17:13:32Z

Ah, I took a look at your repo and noticed that your key ("AKIAIOSFODNN7EXAMPLE") is an obvious false-positive. This results in it being filtered by the false-positive check.

trufflehog/pkg/detectors/aws/aws.go

Lines 231 to 235 in becce51

 if !s1.Verified { 

 // Unverified results that contain common test words are probably not secrets 

 if detectors.IsKnownFalsePositive(resSecretMatch, detectors.DefaultFalsePositives, true) { 

 continue 

 }

trufflehog/pkg/detectors/words.txt

Line 22 in becce51

example

I'm not sure if @Constantin07's issue is similar, or if the credentials in their example are 'live'.

Constantin07 · 2024-04-02T17:16:09Z

@rgmz yes, I've used live credentials.

rgmz · 2024-04-02T18:11:41Z

Hmm, strange. What do you see if you run the following commands (based on this branch)?
(This will print the ID/Secret/account number, so don't run it somewhere that it can be exposed in the logs, and remove that information if you paste the output.)

For the example that Mike shared, we can see that it skips the result because of the false-positive check.

git clone [email protected]:rgmz/trufflehog.git
cd trufflehog
git switch -C debug/aws-logging
go build
./trufflehog filesystem /path/to/repo

# Output
🐷🔑🐷  TruffleHog. Unearth your secrets. 🐷🔑🐷

2024-04-02T14:05:04-04:00       info-0  trufflehog      possible duplicate detector configured  {"detector": "JiraToken.v2"}
2024-04-02T14:05:04-04:00       info-0  trufflehog      running source  {"source_manager_worker_id": "Hx87Z", "with_units": true}
2024-04-02T14:05:04-04:00       info-0  trufflehog.aws  Found ID        {"detector_worker_id": "usiWR", "timeout": 10, "id": ["AKIAIOSFODNN7EXAMPLE", "AKIAIOSFODNN7EXAMPLE", "AKIA"]}
2024-04-02T14:05:04-04:00       info-0  trufflehog.aws  Found ID        {"detector_worker_id": "4aQdJ", "timeout": 10, "id": ["AKIAIOSFODNN7EXAMPLE", "AKIAIOSFODNN7EXAMPLE", "AKIA"]}
2024-04-02T14:05:04-04:00       info-0  trufflehog.aws  Found Secret    {"detector_worker_id": "4aQdJ", "timeout": 10, "secret": ["\"wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY\"", "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"]}
2024-04-02T14:05:04-04:00       info-0  trufflehog.aws  Determined account number       {"detector_worker_id": "4aQdJ", "timeout": 10, "account": ""}
2024-04-02T14:05:04-04:00       info-0  trufflehog.aws  Attempting to verify match      {"detector_worker_id": "4aQdJ", "timeout": 10}
2024-04-02T14:05:04-04:00       info-0  trufflehog.aws  Skipping result, it is a false-positive {"detector_worker_id": "4aQdJ", "timeout": 10}
2024-04-02T14:05:04-04:00       info-0  trufflehog      finished scanning       {"chunks": 27, "bytes": 26566, "verified_secrets": 0, "unverified_secrets": 0, "scan_duration": "411.535354ms"}

mikedidomizio · 2024-04-02T22:41:57Z

(If at any point my issue is confirmed separate from Constantin07 I'll create a new issue, for now I'll just continue here)

I took an active AWS IAM cred, put it into the code, and modified the secret access key slightly to see if it would trigger. It did not. I even committed it to give it a bit more area to catch it, didn't catch it.

Here was the output on main branch

🐷🔑🐷  TruffleHog. Unearth your secrets. 🐷🔑🐷

2024-04-02T22:38:20Z	info-0	trufflehog	possible duplicate detector configured	{"detector": "JiraToken.v2"}
2024-04-02T22:38:20Z	info-0	trufflehog	running source	{"source_manager_worker_id": "rDI2H", "with_units": true}
2024-04-02T22:38:21Z	info-0	trufflehog	finished scanning	{"chunks": 19, "bytes": 24222, "verified_secrets": 0, "unverified_secrets": 0, "scan_duration": "629.643375ms"}

I checked out your branch, built it with Docker. Here is the output (with .git/objects removed) with "REDACTED" where the id/secret or account id would be:

🐷🔑🐷  TruffleHog. Unearth your secrets. 🐷🔑🐷

2024-04-02T22:35:01Z	info-0	trufflehog	possible duplicate detector configured	{"detector": "JiraToken.v2"}
2024-04-02T22:35:01Z	info-0	trufflehog	running source	{"source_manager_worker_id": "rxKlR", "with_units": true}
2024-04-02T22:35:01Z	info-0	trufflehog.aws	Found ID	{"detector_worker_id": "Rplh1", "timeout": 10, "id": ["REDACTED", "REDACTED", "AKIA"]}
2024-04-02T22:35:01Z	info-0	trufflehog.aws	Found Secret	{"detector_worker_id": "Rplh1", "timeout": 10, "secret": ["\"REDACTED\"", "REDACTED"]}
2024-04-02T22:35:01Z	info-0	trufflehog.aws	Determined account number	{"detector_worker_id": "Rplh1", "timeout": 10, "account": "REDACTED"}
2024-04-02T22:35:01Z	info-0	trufflehog.aws	Attempting to verify match	{"detector_worker_id": "Rplh1", "timeout": 10}
2024-04-02T22:35:01Z	info-0	trufflehog.aws	Found ID	{"detector_worker_id": "wg89M", "timeout": 10, "id": ["REDACTED", "REDACTED", "AKIA"]}
2024-04-02T22:35:02Z	info-0	trufflehog.aws	Skipping result, it is a false-positive	{"detector_worker_id": "Rplh1", "timeout": 10}
2024-04-02T22:35:02Z	info-0	trufflehog	finished scanning	{"chunks": 19, "bytes": 24222, "verified_secrets": 0, "unverified_secrets": 0, "scan_duration": "472.759083ms"}

rgmz · 2024-04-02T23:04:47Z

I took an active AWS IAM cred, put it into the code, and modified the secret access key slightly to see if it would trigger. It did not. I even committed it to give it a bit more area to catch it, didn't catch it.
...
2024-04-02T22:35:02Z info-0 trufflehog.aws Skipping result, it is a false-positive {"detector_worker_id": "Rplh1", "timeout": 10}

Interesting, that definitely shouldn't be the case.

Can you pull the latest changes and try again? I added some print statements which should show the specific reason it's getting flagged as a false-positive.

2024-04-02T19:03:13-04:00       info-0  trufflehog.aws  Attempting to verify match      {"detector_worker_id": "V5ZP0", "timeout": 10}
matched FalsePositive 'example'
2024-04-02T19:03:13-04:00       info-0  trufflehog.aws  Skipping result, it is a false-positive {"detector_worker_id": "V5ZP0", "timeout": 10}

mikedidomizio · 2024-04-02T23:48:24Z

Okay, I know why for me it didn't flag a real access key id. When I modified the access secret I left the EXAMPLE portion in the secret, when I changed it to EZAMPLE it flags as expected. I now see that that word is in the code base to ignore.

Anyways, sorry for hijacking this @Constantin07. It's possibly not the same issue as you. Thanks @rgmz for bearing with me.

Constantin07 · 2024-04-03T10:27:37Z

I have generated a new set of AWS credentials and now it does detect them in filesystem scan:

trufflehog filesystem .

🐷🔑🐷  TruffleHog. Unearth your secrets. 🐷🔑🐷

2024-04-03T11:07:21+01:00       info-0  trufflehog      running source  {"source_manager_worker_id": "0RByN", "with_units": true}
Found unverified result 🐷🔑❓
Detector Type: AWS
Decoder Type: PLAIN
Raw result: AKIAU6GDVF*XE1N*****
Resource_type: Access key
Account: *******************
File: .aws/credentials
Line: 1

Found unverified result 🐷🔑❓
Detector Type: AWS
Decoder Type: PLAIN
Raw result: AKIAU6GDVF*XE1N*****
Resource_type: Access key
Account: 339712749422
File: .git/objects/93/78557dddb3ad9b9492f2b3f71e8851e1858df3
Line: 1

But detection doesn't work when I scan the repository where the commit has been made with credentials (it's in commit log).

Here is the pre-commit config:

  - repo: local
    hooks:
      - id: trufflehog
        name: TruffleHog
        description: Detect secrets in your data with TruffleHog.
        entry: bash -c 'trufflehog git file://. --since-commit HEAD --only-verified --fail --no-update'
        language: system
        stages: [pre-commit, pre-push, manual]

(Following example from here https://github.com/trufflesecurity/trufflehog?tab=readme-ov-file#pre-commit-hook)

 pre-commit run -a
...
TruffleHog...............................................................Passed

Manually:

$ trufflehog git file://. --since-commit HEAD --only-verified --fail --no-update
🐷🔑🐷  TruffleHog. Unearth your secrets. 🐷🔑🐷

2024-04-03T11:17:17+01:00       info-0  trufflehog      running source  {"source_manager_worker_id": "EcnaA", "with_units": true}
2024-04-03T11:17:17+01:00       info-0  trufflehog      finished scanning       {"chunks": 0, "bytes": 0, "verified_secrets": 0, "unverified_secrets": 0, "scan_duration": "72.81015ms"}

If I rase a PR in GH with a branch containing AWS secrets, the below GH action check doesn't detect anything either:

GH action check:

jobs:
...
  secrets-scan:
    runs-on: ubuntu-22.04
    steps:
      - name: Checkout code
        uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - name: Secret Scanning
        uses: trufflesecurity/trufflehog@main
        with:
          extra_args: --only-verified

(Following example from here https://github.com/trufflesecurity/trufflehog?tab=readme-ov-file#general-usage)

Output:

I was expecting it to scan git history for diff between PR branch and the main one for leaked creds but it doesn't work for some reason despite PR contains valid AWS creds.

So filesystem scan works but git scan doesn't. What am I doing wrong ?

Constantin07 added the bug label Mar 25, 2024

rgmz mentioned this issue Apr 2, 2024

Move detectors.IsKnownFalsePositive from the detectors and into the engine #2643

Merged

2 tasks

Constantin07 changed the title ~~AWS credentials in files not detected~~ AWS credentials in files not detected in git scan Apr 3, 2024

domwhewell-sage mentioned this issue May 1, 2024

AWS Session Key not being matched #2771

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

AWS credentials in files not detected in git scan #2620

AWS credentials in files not detected in git scan #2620

Constantin07 commented Mar 25, 2024 •

edited

mikedidomizio commented Apr 1, 2024

rgmz commented Apr 1, 2024

mikedidomizio commented Apr 1, 2024

rgmz commented Apr 2, 2024

Constantin07 commented Apr 2, 2024

rgmz commented Apr 2, 2024 •

edited

mikedidomizio commented Apr 2, 2024

rgmz commented Apr 2, 2024

mikedidomizio commented Apr 2, 2024

Constantin07 commented Apr 3, 2024 •

edited

AWS credentials in files not detected in git scan #2620

AWS credentials in files not detected in git scan #2620

Comments

Constantin07 commented Mar 25, 2024 • edited

TruffleHog Version

Trace Output

Expected Behaviour

Actual Behavior

Steps to Reproduce

Environment

Additional Context

References

mikedidomizio commented Apr 1, 2024

rgmz commented Apr 1, 2024

mikedidomizio commented Apr 1, 2024

rgmz commented Apr 2, 2024

Constantin07 commented Apr 2, 2024

rgmz commented Apr 2, 2024 • edited

mikedidomizio commented Apr 2, 2024

rgmz commented Apr 2, 2024

mikedidomizio commented Apr 2, 2024

Constantin07 commented Apr 3, 2024 • edited

Constantin07 commented Mar 25, 2024 •

edited

rgmz commented Apr 2, 2024 •

edited

Constantin07 commented Apr 3, 2024 •

edited