Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(UI): New progress bar #3403

Closed
wants to merge 58 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
58 commits
Select commit Hold shift + click to select a range
856afb3
chore(update): rebase from master (#3067)
n4ch04 Nov 27, 2023
dedb03c
initial fix for issue #2516
JustinMoorcroft Dec 5, 2023
7bfdb8c
Update awslambda_service.py
Fennerr Dec 5, 2023
f71052b
Update awslambda_service.py
Fennerr Dec 6, 2023
5dfd846
Improved threading for EC2
Fennerr Dec 11, 2023
06d9ecc
Added threading
Fennerr Dec 12, 2023
19f38db
Modified logging statements
Fennerr Dec 12, 2023
e97e2e8
Merge branch 'master' of https://github.com/prowler-cloud/prowler
Fennerr Dec 12, 2023
5554e2b
Merge branch 'master' into issue-2516
Fennerr Dec 12, 2023
0481435
Made use of service thread_pool
Fennerr Dec 12, 2023
f7248df
feat(mute list): change allowlist to mute list (#3039)
sergargar Dec 12, 2023
6ee016e
chore(sts-endpoint): deprecate `--sts-endpoint-region` (#3046)
sergargar Dec 12, 2023
1fe7493
feat(CloudProvider): introduce global provider Azure&GCP (#3069)
n4ch04 Dec 12, 2023
1b3f830
test(lambda): fix tests
jfagoagas Dec 13, 2023
fb5d6cf
refactor(lambda): fetch code
jfagoagas Dec 13, 2023
85c44f0
Initial progress
Fennerr Dec 13, 2023
578b21f
Fixed error log message
Fennerr Dec 13, 2023
92e0d74
Keeping the code seperate from the function obj
Fennerr Dec 13, 2023
cb4de85
fix(lambda): Do not use function.code
jfagoagas Dec 13, 2023
2ca4656
fix(lambda): Do not use function.code
jfagoagas Dec 13, 2023
5b80082
merged issue-2516
Fennerr Dec 13, 2023
93a2431
feat(compliance): execute all compliance by default (#3003)
sergargar Dec 13, 2023
f324f27
updated ui redesign implementation
Fennerr Dec 13, 2023
126acc0
Added execution manager and live display
Fennerr Dec 15, 2023
0ef85b3
fix(gcp): fix error in generating compliance (#3201)
sergargar Dec 18, 2023
d8101ac
Moved all the check execution logic into execution manager
Fennerr Dec 18, 2023
ba70540
Moved all the check execution logic into execution manager
Fennerr Dec 18, 2023
01f3c86
WIP: improved layout
Fennerr Dec 18, 2023
49003fa
WIP: added results table
Fennerr Dec 19, 2023
c927408
WIP: centered results table
Fennerr Dec 19, 2023
e9c6b35
WIP: added verbose results and timer
Fennerr Dec 19, 2023
abaa785
Pull rebase from master
Fennerr Dec 11, 2023
040b780
WIP: improved layout
Fennerr Dec 19, 2023
57550e6
initial switch
Fennerr Dec 20, 2023
4845d60
added progress decorator
Fennerr Dec 20, 2023
132056f
some more progress
Fennerr Dec 20, 2023
606b4b5
merged threading progress
Fennerr Dec 20, 2023
f2f922d
fixed decorator to correctly handle args
Fennerr Dec 20, 2023
0d97780
cleaned up execution manager,live display. Added metaclass
Fennerr Dec 20, 2023
197a08a
Added --only-logs and some reordering
Fennerr Dec 21, 2023
b976cab
Added rich to poetry dependencies
Fennerr Dec 21, 2023
028d29b
Added rich to poetry dependencies
Fennerr Dec 21, 2023
c2f8980
feat(kubernetes): add Kubernetes provider (#3226)
sergargar Jan 9, 2024
c08e244
feat(status): add --status flag (#3238)
sergargar Jan 9, 2024
24efb34
chore(manual status): change INFO to MANUAL status (#3254)
sergargar Jan 9, 2024
36fc575
feat(AwsProvider): include new structure for AWS provider (#3252)
n4ch04 Jan 15, 2024
c3d129a
chore(update): rebase from master (#3067)
n4ch04 Nov 27, 2023
445821e
feat(mute list): change allowlist to mute list (#3039)
sergargar Dec 12, 2023
722fe0a
chore(sts-endpoint): deprecate `--sts-endpoint-region` (#3046)
sergargar Dec 12, 2023
0d96583
feat(CloudProvider): introduce global provider Azure&GCP (#3069)
n4ch04 Dec 12, 2023
09b4548
feat(compliance): execute all compliance by default (#3003)
sergargar Dec 13, 2023
217b8ad
fix(gcp): fix error in generating compliance (#3201)
sergargar Dec 18, 2023
e72831d
feat(kubernetes): add Kubernetes provider (#3226)
sergargar Jan 9, 2024
5419117
feat(status): add --status flag (#3238)
sergargar Jan 9, 2024
2eb774b
chore(manual status): change INFO to MANUAL status (#3254)
sergargar Jan 9, 2024
537081a
feat(AwsProvider): include new structure for AWS provider (#3252)
n4ch04 Jan 15, 2024
2f50aaa
resolve conflicts
sergargar Jan 16, 2024
bcfdcbd
Resolved some conflicts
Fennerr Feb 15, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
2 changes: 1 addition & 1 deletion docs/developer-guide/checks.md
Expand Up @@ -102,7 +102,7 @@ All the checks MUST fill the `report.status` and `report.status_extended` with t
- Status -- `report.status`
- `PASS` --> If the check is passing against the configured value.
- `FAIL` --> If the check is passing against the configured value.
- `INFO` --> This value cannot be used unless a manual operation is required in order to determine if the `report.status` is whether `PASS` or `FAIL`.
- `MANUAL` --> This value cannot be used unless a manual operation is required in order to determine if the `report.status` is whether `PASS` or `FAIL`.
- Status Extended -- `report.status_extended`
- MUST end in a dot `.`
- MUST include the service audited with the resource and a brief explanation of the result generated, e.g.: `EC2 AMI ami-0123456789 is not public.`
Expand Down
4 changes: 0 additions & 4 deletions docs/tutorials/aws/authentication.md
Expand Up @@ -37,7 +37,3 @@ If your IAM entity enforces MFA you can use `--mfa` and Prowler will ask you to

- ARN of your MFA device
- TOTP (Time-Based One-Time Password)

## STS Endpoint Region

If you are using Prowler in AWS regions that are not enabled by default you need to use the argument `--sts-endpoint-region` to point the AWS STS API calls `assume-role` and `get-caller-identity` to the non-default region, e.g.: `prowler aws --sts-endpoint-region eu-south-2`.
8 changes: 0 additions & 8 deletions docs/tutorials/aws/role-assumption.md
Expand Up @@ -23,14 +23,6 @@ prowler aws -R arn:aws:iam::<account_id>:role/<role_name>
prowler aws -T/--session-duration <seconds> -I/--external-id <external_id> -R arn:aws:iam::<account_id>:role/<role_name>
```

## STS Endpoint Region

If you are using Prowler in AWS regions that are not enabled by default you need to use the argument `--sts-endpoint-region` to point the AWS STS API calls `assume-role` and `get-caller-identity` to the non-default region, e.g.: `prowler aws --sts-endpoint-region eu-south-2`.

> Since v3.11.0, Prowler uses a regional token in STS sessions so it can scan all AWS regions without needing the `--sts-endpoint-region` argument.

> Make sure that you have enabled the AWS Region you want to scan in BOTH AWS Accounts (assumed role account and account from which you assume the role).

## Role MFA

If your IAM Role has MFA configured you can use `--mfa` along with `-R`/`--role <role_arn>` and Prowler will ask you to input the following values to get a new temporary session for the IAM Role provided:
Expand Down
31 changes: 20 additions & 11 deletions docs/tutorials/compliance.md
@@ -1,5 +1,18 @@
# Compliance
Prowler allows you to execute checks based on requirements defined in compliance frameworks.
Prowler allows you to execute checks based on requirements defined in compliance frameworks. By default, it will execute and give you an overview of the status of each compliance framework:

<img src="../img/compliance.png"/>

> You can find CSVs containing detailed compliance results inside the compliance folder within Prowler's output folder.

## Execute Prowler based on Compliance Frameworks
Prowler can analyze your environment based on a specific compliance framework and get more details, to do it, you can use option `--compliance`:
```sh
prowler <provider> --compliance <compliance_framework>
```
Standard results will be shown and additionally the framework information as the sample below for CIS AWS 1.5. For details a CSV file has been generated as well.

<img src="../img/compliance-cis-sample1.png"/>

## List Available Compliance Frameworks
In order to see which compliance frameworks are cover by Prowler, you can use option `--list-compliance`:
Expand All @@ -10,9 +23,12 @@ Currently, the available frameworks are:

- `cis_1.4_aws`
- `cis_1.5_aws`
- `cis_2.0_aws`
- `cisa_aws`
- `ens_rd2022_aws`
- `aws_audit_manager_control_tower_guardrails_aws`
- `aws_foundational_security_best_practices_aws`
- `aws_well_architected_framework_reliability_pillar_aws`
- `aws_well_architected_framework_security_pillar_aws`
- `cisa_aws`
- `fedramp_low_revision_4_aws`
Expand All @@ -22,6 +38,9 @@ Currently, the available frameworks are:
- `gxp_eu_annex_11_aws`
- `gxp_21_cfr_part_11_aws`
- `hipaa_aws`
- `iso27001_2013_aws`
- `iso27001_2013_aws`
- `mitre_attack_aws`
- `nist_800_53_revision_4_aws`
- `nist_800_53_revision_5_aws`
- `nist_800_171_revision_2_aws`
Expand All @@ -38,7 +57,6 @@ prowler <provider> --list-compliance-requirements <compliance_framework(s)>
```

Example for the first requirements of CIS 1.5 for AWS:

```
Listing CIS 1.5 AWS Compliance Requirements:

Expand Down Expand Up @@ -71,15 +89,6 @@ Requirement Id: 1.5

```

## Execute Prowler based on Compliance Frameworks
As we mentioned, Prowler can be execute to analyse you environment based on a specific compliance framework, to do it, you can use option `--compliance`:
```sh
prowler <provider> --compliance <compliance_framework>
```
Standard results will be shown and additionally the framework information as the sample below for CIS AWS 1.5. For details a CSV file has been generated as well.

<img src="../img/compliance-cis-sample1.png"/>

## Create and contribute adding other Security Frameworks

This information is part of the Developer Guide and can be found here: https://docs.prowler.cloud/en/latest/tutorials/developer-guide/.
12 changes: 6 additions & 6 deletions docs/tutorials/configuration_file.md
Expand Up @@ -29,10 +29,10 @@ The following list includes all the AWS checks with configurable variables that
| `organizations_delegated_administrators` | `organizations_trusted_delegated_administrators` | List of Strings |
| `ecr_repositories_scan_vulnerabilities_in_latest_image` | `ecr_repository_vulnerability_minimum_severity` | String |
| `trustedadvisor_premium_support_plan_subscribed` | `verify_premium_support_plans` | Boolean |
| `config_recorder_all_regions_enabled` | `allowlist_non_default_regions` | Boolean |
| `drs_job_exist` | `allowlist_non_default_regions` | Boolean |
| `guardduty_is_enabled` | `allowlist_non_default_regions` | Boolean |
| `securityhub_enabled` | `allowlist_non_default_regions` | Boolean |
| `config_recorder_all_regions_enabled` | `mute_non_default_regions` | Boolean |
| `drs_job_exist` | `mute_non_default_regions` | Boolean |
| `guardduty_is_enabled` | `mute_non_default_regions` | Boolean |
| `securityhub_enabled` | `mute_non_default_regions` | Boolean |

## Azure

Expand All @@ -50,8 +50,8 @@ The following list includes all the AWS checks with configurable variables that
aws:

# AWS Global Configuration
# aws.allowlist_non_default_regions --> Allowlist Failed Findings in non-default regions for GuardDuty, SecurityHub, DRS and Config
allowlist_non_default_regions: False
# aws.mute_non_default_regions --> Mute Failed Findings in non-default regions for GuardDuty, SecurityHub, DRS and Config
mute_non_default_regions: False

# AWS IAM Configuration
# aws.iam_user_accesskey_unused --> CIS recommends 45 days
Expand Down
Binary file added docs/tutorials/img/compliance.png
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
2 changes: 1 addition & 1 deletion docs/tutorials/logging.md
Expand Up @@ -8,7 +8,7 @@ There are different log levels depending on the logging information that is desi

- **DEBUG**: It will show low-level logs from Python.
- **INFO**: It will show all the API calls that are being invoked by the provider.
- **WARNING**: It will show all resources that are being **allowlisted**.
- **WARNING**: It will show all resources that are being **muted**.
- **ERROR**: It will show any errors, e.g., not authorized actions.
- **CRITICAL**: The default log level. If a critical log appears, it will **exit** Prowler’s execution.

Expand Down
6 changes: 3 additions & 3 deletions docs/tutorials/misc.md
Expand Up @@ -9,10 +9,10 @@ Execute Prowler in verbose mode (like in Version 2):
```console
prowler <provider> --verbose
```
## Show only Fails
Prowler can only display the failed findings:
## Filter findings by status
Prowler can filter the findings by their status:
```console
prowler <provider> -q/--quiet
prowler <provider> --status [PASS, FAIL, MANUAL]
```
## Disable Exit Code 3
Prowler does not trigger exit code 3 with failed checks:
Expand Down
68 changes: 34 additions & 34 deletions docs/tutorials/allowlist.md → docs/tutorials/mutelist.md
@@ -1,19 +1,19 @@
# Allowlisting
# Mute Listing
Sometimes you may find resources that are intentionally configured in a certain way that may be a bad practice but it is all right with it, for example an AWS S3 Bucket open to the internet hosting a web site, or an AWS Security Group with an open port needed in your use case.

Allowlist option works along with other options and adds a `WARNING` instead of `INFO`, `PASS` or `FAIL` to any output format.
Mute List option works along with other options and adds a `MUTED` instead of `MANUAL`, `PASS` or `FAIL` to any output format.

You can use `-w`/`--allowlist-file` with the path of your allowlist yaml file, but first, let's review the syntax.
You can use `-w`/`--mutelist-file` with the path of your mutelist yaml file, but first, let's review the syntax.

## Allowlist Yaml File Syntax
## Mute List Yaml File Syntax

### Account, Check and/or Region can be * to apply for all the cases.
### Resources and tags are lists that can have either Regex or Keywords.
### Tags is an optional list that matches on tuples of 'key=value' and are "ANDed" together.
### Use an alternation Regex to match one of multiple tags with "ORed" logic.
### For each check you can except Accounts, Regions, Resources and/or Tags.
########################### ALLOWLIST EXAMPLE ###########################
Allowlist:
########################### MUTE LIST EXAMPLE ###########################
Mute List:
Accounts:
"123456789012":
Checks:
Expand Down Expand Up @@ -79,10 +79,10 @@ You can use `-w`/`--allowlist-file` with the path of your allowlist yaml file, b
Tags:
- "environment=prod" # Will ignore every resource except in account 123456789012 except the ones containing the string "test" and tag environment=prod

## Allowlist specific regions
If you want to allowlist/mute failed findings only in specific regions, create a file with the following syntax and run it with `prowler aws -w allowlist.yaml`:
## Mute specific regions
If you want to mute failed findings only in specific regions, create a file with the following syntax and run it with `prowler aws -w mutelist.yaml`:

Allowlist:
Mute List:
Accounts:
"*":
Checks:
Expand All @@ -93,50 +93,50 @@ If you want to allowlist/mute failed findings only in specific regions, create a
Resources:
- "*"

## Default AWS Allowlist
Prowler provides you a Default AWS Allowlist with the AWS Resources that should be allowlisted such as all resources created by AWS Control Tower when setting up a landing zone.
You can execute Prowler with this allowlist using the following command:
## Default AWS Mute List
Prowler provides you a Default AWS Mute List with the AWS Resources that should be muted such as all resources created by AWS Control Tower when setting up a landing zone.
You can execute Prowler with this mutelist using the following command:
```sh
prowler aws --allowlist prowler/config/aws_allowlist.yaml
prowler aws --mutelist prowler/config/aws_mutelist.yaml
```
## Supported Allowlist Locations
## Supported Mute List Locations

The allowlisting flag supports the following locations:
The mutelisting flag supports the following locations:

### Local file
You will need to pass the local path where your Allowlist YAML file is located:
You will need to pass the local path where your Mute List YAML file is located:
```
prowler <provider> -w allowlist.yaml
prowler <provider> -w mutelist.yaml
```
### AWS S3 URI
You will need to pass the S3 URI where your Allowlist YAML file was uploaded to your bucket:
You will need to pass the S3 URI where your Mute List YAML file was uploaded to your bucket:
```
prowler aws -w s3://<bucket>/<prefix>/allowlist.yaml
prowler aws -w s3://<bucket>/<prefix>/mutelist.yaml
```
> Make sure that the used AWS credentials have s3:GetObject permissions in the S3 path where the allowlist file is located.
> Make sure that the used AWS credentials have s3:GetObject permissions in the S3 path where the mutelist file is located.

### AWS DynamoDB Table ARN

You will need to pass the DynamoDB Allowlist Table ARN:
You will need to pass the DynamoDB Mute List Table ARN:

```
prowler aws -w arn:aws:dynamodb:<region_name>:<account_id>:table/<table_name>
```

1. The DynamoDB Table must have the following String keys:
<img src="../img/allowlist-keys.png"/>
<img src="../img/mutelist-keys.png"/>

- The Allowlist Table must have the following columns:
- Accounts (String): This field can contain either an Account ID or an `*` (which applies to all the accounts that use this table as an allowlist).
- The Mute List Table must have the following columns:
- Accounts (String): This field can contain either an Account ID or an `*` (which applies to all the accounts that use this table as an mutelist).
- Checks (String): This field can contain either a Prowler Check Name or an `*` (which applies to all the scanned checks).
- Regions (List): This field contains a list of regions where this allowlist rule is applied (it can also contains an `*` to apply all scanned regions).
- Resources (List): This field contains a list of regex expressions that applies to the resources that are wanted to be allowlisted.
- Tags (List): -Optional- This field contains a list of tuples in the form of 'key=value' that applies to the resources tags that are wanted to be allowlisted.
- Exceptions (Map): -Optional- This field contains a map of lists of accounts/regions/resources/tags that are wanted to be excepted in the allowlist.
- Regions (List): This field contains a list of regions where this mutelist rule is applied (it can also contains an `*` to apply all scanned regions).
- Resources (List): This field contains a list of regex expressions that applies to the resources that are wanted to be muted.
- Tags (List): -Optional- This field contains a list of tuples in the form of 'key=value' that applies to the resources tags that are wanted to be muted.
- Exceptions (Map): -Optional- This field contains a map of lists of accounts/regions/resources/tags that are wanted to be excepted in the mutelist.

The following example will allowlist all resources in all accounts for the EC2 checks in the regions `eu-west-1` and `us-east-1` with the tags `environment=dev` and `environment=prod`, except the resources containing the string `test` in the account `012345678912` and region `eu-west-1` with the tag `environment=prod`:
The following example will mute all resources in all accounts for the EC2 checks in the regions `eu-west-1` and `us-east-1` with the tags `environment=dev` and `environment=prod`, except the resources containing the string `test` in the account `012345678912` and region `eu-west-1` with the tag `environment=prod`:

<img src="../img/allowlist-row.png"/>
<img src="../img/mutelist-row.png"/>

> Make sure that the used AWS credentials have `dynamodb:PartiQLSelect` permissions in the table.

Expand All @@ -151,7 +151,7 @@ prowler aws -w arn:aws:lambda:REGION:ACCOUNT_ID:function:FUNCTION_NAME
Make sure that the credentials that Prowler uses can invoke the Lambda Function:

```
- PolicyName: GetAllowList
- PolicyName: GetMuteList
PolicyDocument:
Version: '2012-10-17'
Statement:
Expand All @@ -160,14 +160,14 @@ Make sure that the credentials that Prowler uses can invoke the Lambda Function:
Resource: arn:aws:lambda:REGION:ACCOUNT_ID:function:FUNCTION_NAME
```

The Lambda Function can then generate an Allowlist dynamically. Here is the code an example Python Lambda Function that
generates an Allowlist:
The Lambda Function can then generate an Mute List dynamically. Here is the code an example Python Lambda Function that
generates an Mute List:

```
def handler(event, context):
checks = {}
checks["vpc_flow_logs_enabled"] = { "Regions": [ "*" ], "Resources": [ "" ], Optional("Tags"): [ "key:value" ] }

al = { "Allowlist": { "Accounts": { "*": { "Checks": checks } } } }
al = { "Mute List": { "Accounts": { "*": { "Checks": checks } } } }
return al
```
2 changes: 1 addition & 1 deletion mkdocs.yml
Expand Up @@ -36,7 +36,7 @@ nav:
- Slack Integration: tutorials/integrations.md
- Configuration File: tutorials/configuration_file.md
- Logging: tutorials/logging.md
- Allowlist: tutorials/allowlist.md
- Mute List: tutorials/mutelist.md
- Check Aliases: tutorials/check-aliases.md
- Custom Metadata: tutorials/custom-checks-metadata.md
- Ignore Unused Services: tutorials/ignore-unused-services.md
Expand Down
39 changes: 32 additions & 7 deletions poetry.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.