Modify AWS module to process logs from WAF v2 #22572

GGP1 · 2024-03-15T15:23:45Z

Epic
#23361

Description

Our current integration with AWS WAF is only extracting records through Kinesis.

In this issue, we should perform the modifications necessary to our AWS module to fetch the records being generated on AWS WAF v2, following the path

<bucket_name>/<prefix>/AWSLogs/<suffix>/<account_id>/WAFLogs/<region>/<waf_name>/<year>/<month>/<day>/<hh>/<mm>

Note

The Migrating your AWS WAF Classic resources to AWS WAF guide might be useful.

Checks

The following elements have been updated or reviewed (should also be checked if no modification is required):

Tests (unit tests, API integration tests).
Changelog.
Documentation.
Integration test mapping (using api/test/integration/mapping/_test_mapping.py).

The text was updated successfully, but these errors were encountered:

Selutario · 2024-03-18T13:24:58Z

https://github.com/wazuh/internal-devel-requests/issues/967#issuecomment-2003638945

javiersanchz · 2024-04-25T16:24:14Z

Update

I have been reading about the migration changes
I have been reviewing all the research that was done on it at Review and update AWS WAF documentation wazuh-documentation#5808
I have been looking at changes in the waf.py code and reviewing about it. I started with the reproduction about it.

javiersanchz · 2024-04-26T16:43:39Z

Update

I'm still working on the issue, I've been reproducing the case mentioned about the discrepancy in routes when the record comes from WAF classic or WAF v2 when logging in directly to an S3 instead of through Kinesis.
I've been making changes to the code to address this

javiersanchz · 2024-04-29T17:21:22Z

Update

I have been requesting all the necessary permissions to create a WAF v2, to be able to create rules, add an S3 bucket...
I have been testing generating the logs manually with the following output in the /var/ossec/logs/alerts/alerts.json:

{"timestamp":"2024-04-29T11:38:31.092+0000","rule":{"level":3,"description":"AWS WAF - Blocked request.","id":"80442","firedtimes":3,"mail":false,"groups":["amazon","aws","aws_waf","aws_waf_block"]},"agent":{"id":"000","name":"wazuh-master"},"manager":{"name":"wazuh-master"},"id":"xxx.822910","cluster":{"name":"wazuh","node":"master-node"},"decoder":{"name":"json"},"data":{"integration":"aws","aws":{"log_info":{"log_file":"2019/10/23/11/aws-waf-logs-delivery-stream-1-2019-10-23-11-32-48-7xxd1f-bfed-4b00-9f5e-88ce44718194","s3bucket":"wazuh-aws-wodle-waf"},"timestamp":"1576280412771.000000","formatVersion":"1","webaclId":"arn:aws:wafv2:ap-southeast-2:1xxx5:regional/webacl/test/111","terminatingRuleId":"STMTest_SQLi_XSS","terminatingRuleType":"REGULAR","action":"BLOCK","terminatingRuleMatchDetails":{"conditionType":"SQL_INJECTION","location":"UNKNOWN","matchedData":["10","AND","1"]},"httpSourceName":"ALB","httpSourceId":"alb","ruleGroupList":[],"rateBasedRuleList":[],"nonTerminatingMatchingRules":[],"httpRequest":{"clientIp":"1.1.1.1","country":"AU","httpVersion":"HTTP/1.1","httpMethod":"POST","requestId":"null"},"labels":{"name":"value"},"source":"waf"}},"location":"Wazuh-AWS"}

I've been creating an AWS WAF v2, associating a bucket, and creating rules to generate alerts:

I did some research on how to generate v2 logs.

javiersanchz · 2024-04-30T16:53:59Z

Update

I continue working on the migration to v2, I'm unable to generate the logs on an AWS WAF v2 (it might also be due to lack of permissions).
The S3 bucket associated with it still isn't receiving any kind of log:

root@wazuh-master:/# /var/ossec/wodles/aws/aws-s3 --bucket aws-waf-logs-wodle-v2  --aws_profile default --only_logs_after 2019-OCT-22 --regions us-east-1 --type waf --skip_on_error -d 2
DEBUG: +++ Debug mode on - Level: 2
DEBUG: No retries configuration found in profile config. Generating default configuration for retries: mode: standard - max_attempts: 10
DEBUG: Created Config object using profile: 'default' configuration
DEBUG: +++ Marker: 2019/10/22
DEBUG: +++ No logs to process in bucket: aws-waf-logs-wodle-v2
DEBUG: +++ DB Maintenance

javiersanchz · 2024-05-02T16:36:02Z

Update

I have been setting up a real WAF environment for log generation in the S3 bucket:

Created a new EC2 instance, adding it to a target group.
Created and configured an ACL (Access Control List).
Created and configured an ALB (AWS Load Balancer).
Associated the web ACL with the ALB through AWS WAF.
I have been configuring rules in the WAF so that when activated, they generate records that can be seen in the created S3 bucket and view the generated path.

javiersanchz · 2024-05-03T12:52:06Z

Update

After setting up the environment, our AWS WAF v2 generated the path in our S3 bucket with the following route:

Log format:

{"timestamp":1714735190413,"formatVersion":1,"webaclId":"arn:aws:wafv2:us-east-1:xxxxxx:regional/webacl/AWS-WAF-V2/xxxxxxx817d-f47add445857","terminatingRuleId":"Dont-allow-HTTP-GET-and-POST","terminatingRuleType":"REGULAR","action":"BLOCK","terminatingRuleMatchDetails":[],"httpSourceName":"ALB","httpSourceId":"xxxx422-app/ABL-WAF-V2/xxxxxc1b5f","ruleGroupList":[],"rateBasedRuleList":[],"nonTerminatingMatchingRules":[],"requestHeadersInserted":null,"responseCodeSent":402,"httpRequest":{"clientIp":"46.174.191.28","country":"UA","headers":[{"name":"User-Agent","value":"Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; AS; rv:11.0) like Gecko"},{"name":"Accept","value":"*/*"}],"uri":"/","args":"","httpVersion":"HTTP/1.0","httpMethod":"GET","requestId":"xxxxxxxx0713ede17e"}}

The path generated by WAF v2 would be the following:

<bucket_name>/<prefix>/AWSLogs/<suffix>/<account_id>/WAFLogs/<region>/<waf_name>/<year>/<month>/<day>/<hh>/<mm>

The route should be adapted on the AWSWAFBucket class to obtain the generated logs

javiersanchz · 2024-05-03T16:56:44Z

Update

We talked with the rest of the team about which path should be processed within waf.py:
- Path generated by WAF for storing records directly in S3:
  <bucket_name>/<prefix>/AWSLogs/<suffix>/<account_id>/WAFLogs/<region>/<waf_name>/<year>/<month>/<day>/<hh>/<mm>
- Path generated by WAF using Kinesis:
  <bucket_name>/<prefix>/<year>/<month>/<day>
These paths are almost the same as those from GuardDuty:
- Path generated by GuardDuty for storing records directly in S3:
  <bucket_name>/<prefix>/AWSLogs/<suffix>/<account_id>/WAFLogs/<region>/<waf_name>/<year>/<month>/<day>
- Path generated by GuardDuty using Kinesis:
  <bucket_name>/<prefix>/<year>/<month>/<day>
As you can see, the path using Kinesis is the same, and the path generated directly in the S3 bucket only differs in that WAF adds /<hh>/<mm>
Therefore, the decision was made to follow more or less the same procedure, processing both types of paths for WAF by issuing a message similar to what GuardDuty has been doing regarding the deprecation of Kinesis:

wazuh/wodles/aws/buckets_s3/guardduty.py

Lines 12 to 17 in 1e51c2d

 
 GUARDDUTY_URL = 'https://documentation.wazuh.com/current/amazon/services/supported-services/guardduty.html' 

 GUARDDUTY_DEPRECATED_MESSAGE = 'The functionality to process GuardDuty logs stored in S3 via Kinesis was deprecated ' \ 

 'in {release}. Consider configuring GuardDuty to store its findings directly in an S3 ' \ 

 'bucket instead. Check {url} for more information.'

javiersanchz · 2024-05-06T17:26:32Z

Update

It seems that when launching the WAF bucket, it's not processing the logs properly because the accessed path is only up to the day:

<bucket_name>/<prefix>/AWSLogs/<suffix>/<account_id>/WAFLogs/<region>/<waf_name>/<year>/<month>/<day>

As can be seen here:

root@wazuh-master:/# /var/ossec/wodles/aws/aws-s3 --bucket aws-waf-logs-wodle-v2  --aws_profile default --regions us-east-1 --type waf --skip_on_error -d 2
DEBUG: +++ Debug mode on - Level: 2
DEBUG: No retries configuration found in profile config. Generating default configuration for retries: mode: standard - max_attempts: 10
DEBUG: Created Config object using profile: 'default' configuration
DEBUG: +++ Working on 567970947422 - us-east-1
DEBUG: +++ Marker: AWSLogs/567970947422/WAFLogs/us-east-1/2024/05/06
DEBUG: +++ Unexpected error: 'bucket'
ERROR: Unexpected error querying/working with objects in S3: 'bucket'

I've been testing changes in aws_bucket, specifically in the function responsible for it, get_creation_date, but it still uses the same path only up to the <day>.

javiersanchz · 2024-05-09T16:52:27Z

Update

After discussing with the rest of the team about the WAF operation, it has been decided to create an epic Migrate Amazon WAF service to v2 #23361 to split the workload, as it is more than anticipated since the logging route differs from the rest of AWS integrations.
This issue will focus on obtaining the logs generated in the S3 bucket by WAF, either natively or through Kinesis, as well as unit testing

javiersanchz · 2024-05-13T13:07:01Z

Update

The necessary changes were made to obtain records of WAF v2 route, and the following tests were carried out:

WAF native :

root@wazuh-master:/# /var/ossec/wodles/aws/aws-s3 --bucket aws-waf-logs-wodle-v2  --aws_profile default --only_logs_after 2019-OCT-22 --regions us-east-1 --type waf --skip_on_error -d 2
DEBUG: +++ Debug mode on - Level: 2
DEBUG: No retries configuration found in profile config. Generating default configuration for retries: mode: standard - max_attempts: 10
DEBUG: Created Config object using profile: 'default' configuration
DEBUG: +++ Working on 567970947422 - us-east-1
DEBUG: +++ Marker: AWSLogs/567970947422/WAFLogs/us-east-1/AWS-WAF-V2/2019/10/22
DEBUG: ++ Found new log: AWSLogs/567970947422/WAFLogs/us-east-1/AWS-WAF-V2/2024/05/03/11/10/567970947422_waflogs_us-east-1_AWS-WAF-V2_20240503T1110Z_a195076a.log.gz
DEBUG: ++ Found new log: AWSLogs/567970947422/WAFLogs/us-east-1/AWS-WAF-V2/2024/05/03/11/15/567970947422_waflogs_us-east-1_AWS-WAF-V2_20240503T1115Z_7409afbb.log.gz
DEBUG: +++ DB Maintenance

As you can see, the logs belonging to WAF v2 routes were received. Next, the output of these logs has been verified at:

Alerts.log :

root@wazuh-master:/# grep '567970947422' /var/ossec/logs/alerts/alerts.log
{"integration": "aws", "aws": {"log_info": {"aws_account_alias": "", "log_file": "AWSLogs/567970947422/WAFLogs/us-east-1/AWS-WAF-V2/2024/05/03/11/10/567970947422_waflogs_us-east-1_AWS-WAF-V2_20240503T1110Z_a195076a.log.gz", "s3bucket": "aws-waf-logs-wodle-v2"}, "timestamp": 1714734687613, "formatVersion": 1, "webaclId": "arn:aws:wafv2:us-east-1:567970947422:regional/webacl/AWS-WAF-V2/98f8c0f9-6ec8-4c11-817d-xxxxxx", "terminatingRuleId": "Dont-allow-HTTP-GET-and-POST", "terminatingRuleType": "REGULAR", "action": "BLOCK", "terminatingRuleMatchDetails": [], "httpSourceName": "ALB", "httpSourceId": "567970947422-app/ABL-WAF-V2/xxxxx", "ruleGroupList": [], "rateBasedRuleList": [], "nonTerminatingMatchingRules": [], "responseCodeSent": 402, "httpRequest": {"clientIp": "87.236.176.171", "country": "GB", "headers": {"Host": "35.153.251.153", "User-Agent": "Mozilla/5.0 (compatible; InternetMeasurement/1.0; +https://internet-measurement.com/)", "Connection": "close", "Accept": "*/*", "Accept-Encoding": "gzip"}, "uri": "/", "args": "", "httpVersion": "HTTP/1.1", "httpMethod": "GET", "requestId": "1-6634c65f-xxxxxxxxx"}, "source": "waf"}}
aws.log_info.log_file: AWSLogs/567970947422/WAFLogs/us-east-1/AWS-WAF-V2/2024/05/03/11/10/567970947422_waflogs_us-east-1_AWS-WAF-V2_20240503T1110Z_a195076a.log.gz
{"integration": "aws", "aws": {"log_info": {"aws_account_alias": "", "log_file": "AWSLogs/567970947422/WAFLogs/us-east-1/AWS-WAF-V2/2024/05/03/11/15/567970947422_waflogs_us-east-1_AWS-WAF-V2_20240503T1115Z_7409afbb.log.gz", "s3bucket": "aws-waf-logs-wodle-v2"}, "timestamp": 1714735190413, "formatVersion": 1, "webaclId": "arn:aws:wafv2:us-east-1:567970947422:regional/webacl/AWS-WAF-V2/98f8c0f9-6ec8-4c11-817d-f47add445857", "terminatingRuleId": "Dont-allow-HTTP-GET-and-POST", "terminatingRuleType": "REGULAR", "action": "BLOCK", "terminatingRuleMatchDetails": [], "httpSourceName": "ALB", "httpSourceId": "567970947422-app/ABL-WAF-V2/xxxxxxx", "ruleGroupList": [], "rateBasedRuleList": [], "nonTerminatingMatchingRules": [], "responseCodeSent": 402, "httpRequest": {"clientIp": "46.174.191.28", "country": "UA", "headers": {"User-Agent": "Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; AS; rv:11.0) like Gecko", "Accept": "*/*"}, "uri": "/", "args": "", "httpVersion": "HTTP/1.0", "httpMethod": "GET", "requestId": "xxxxxxxxxxxxxxxx"}, "source": "waf"}}
aws.log_info.log_file: AWSLogs/567970947422/WAFLogs/us-east-1/AWS-WAF-V2/2024/05/03/11/15/567970947422_waflogs_us-east-1_AWS-WAF-V2_20240503T1115Z_7409afbb.log.gz

Alerts.json :

{"timestamp":"2024-05-13T11:22:26.600+0000","rule":{"level":3,"description":"AWS WAF - Blocked request.","id":"xxx","firedtimes":1,"mail":false,"groups":["amazon","aws","aws_waf","aws_waf_block"]},"agent":{"id":"000","name":"wazuh-master"},"manager":{"name":"wazuh-master"},"id":"xxxxxx","cluster":{"name":"wazuh","node":"master-node"},"decoder":{"name":"json"},"data":{"integration":"aws","aws":{"log_info":{"log_file":"AWSLogs/567970947422/WAFLogs/us-east-1/AWS-WAF-V2/2024/05/03/11/10/567970947422_waflogs_us-east-1_AWS-WAF-V2_20240503T1110Z_a195076a.log.gz","s3bucket":"aws-waf-logs-wodle-v2"},"timestamp":"1714734687613.000000","formatVersion":"1","webaclId":"arn:aws:wafv2:us-east-1:567970947422:regional/webacl/AWS-WAF-V2/98f8c0f9-6ec8-4c11-817d-f47add445857","terminatingRuleId":"Dont-allow-HTTP-GET-and-POST","terminatingRuleType":"REGULAR","action":"BLOCK","terminatingRuleMatchDetails":[],"httpSourceName":"ALB","httpSourceId":"567970947422-app/ABL-WAF-V2/27426a40f4ac1b5f","ruleGroupList":[],"rateBasedRuleList":[],"nonTerminatingMatchingRules":[],"responseCodeSent":"402","httpRequest":{"clientIp":"87.236.176.171","country":"GB","headers":{"Host":"35.153.251.153","User-Agent":"Mozilla/5.0 (compatible; InternetMeasurement/1.0; +https://internet-measurement.com/)","Connection":"close","Accept":"*/*","Accept-Encoding":"gzip"},"uri":"/","httpVersion":"HTTP/1.1","httpMethod":"GET","requestId":"1xxxxxxxxx"},"source":"waf"}},"location":"Wazuh-AWS"}
{"timestamp":"2024-05-13T11:22:26.885+0000","rule":{"level":3,"description":"AWS WAF - Blocked request.","id":"xxxx","firedtimes":2,"mail":false,"groups":["amazon","aws","aws_waf","aws_waf_block"]},"agent":{"id":"000","name":"wazuh-master"},"manager":{"name":"wazuh-master"},"id":"xxxxxx","cluster":{"name":"wazuh","node":"master-node"},"decoder":{"name":"json"},"data":{"integration":"aws","aws":{"log_info":{"log_file":"AWSLogs/567970947422/WAFLogs/us-east-1/AWS-WAF-V2/2024/05/03/11/15/567970947422_waflogs_us-east-1_AWS-WAF-V2_20240503T1115Z_7409afbb.log.gz","s3bucket":"aws-waf-logs-wodle-v2"},"timestamp":"1714735190413.000000","formatVersion":"1","webaclId":"arn:aws:wafv2:us-east-1:567970947422:regional/webacl/AWS-WAF-V2/98f8c0f9-6ec8-4c11-817d-f47add445857","terminatingRuleId":"Dont-allow-HTTP-GET-and-POST","terminatingRuleType":"REGULAR","action":"BLOCK","terminatingRuleMatchDetails":[],"httpSourceName":"ALB","httpSourceId":"567970947422-app/ABL-WAF-V2/27426a40f4ac1b5f","ruleGroupList":[],"rateBasedRuleList":[],"nonTerminatingMatchingRules":[],"responseCodeSent":"402","httpRequest":{"clientIp":"46.174.191.28","country":"UA","headers":{"User-Agent":"Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; AS; rv:11.0) like Gecko","Accept":"*/*"},"uri":"/","httpVersion":"HTTP/1.0","httpMethod":"GET","requestId":"xxxxxxxx"},"source":"waf"}},"location":"Wazuh-AWS"}

WAF kinesis :

The operation of WAF Kinesis was also verified, including the deprecation message starting from version 5.0:

root@wazuh-master:/# /var/ossec/wodles/aws/aws-s3 --bucket wazuh-aws-wodle-waf  --aws_profile default --only_logs_after 2019-OCT-22 --regions us-east-1 --type waf --skip_on_error -d 2
DEBUG: +++ Debug mode on - Level: 2
DEBUG: No retries configuration found in profile config. Generating default configuration for retries: mode: standard - max_attempts: 10
DEBUG: Created Config object using profile: 'default' configuration
The functionality to process WAF logs stored in S3 via Kinesis was deprecated in 5.0. Consider configuring WAF to store its logs directly in an S3 bucket instead. Check https://documentation.wazuh.com/current/amazon/services/supported-services/waf.html for more information.
DEBUG: +++ Marker: 2019/10/22
DEBUG: ++ Found new log: 2019/10/23/10/aws-waf-logs-delivery-stream-1-2019-10-23-10-32-48-792c6d1f-bfed-4b00-9f5e-88ce44718194
DEBUG: ++ Found new log: 2019/10/23/11/aws-waf-logs-delivery-stream-1-2019-10-23-11-32-48-792c6d1f-bfed-4b00-9f5e-88ce44718194
DEBUG: ++ Found new log: 2022/06/03/aws-waf-logs-delivery-stream-1-2022-06-03-10-32-48-792c6d1f-bfed-4b00-9f5e-88ce4471fake
DEBUG: ++ Found new log: 2022/11/30/11/aws-waf-logs-delivery-stream-1-2022-11-30-11-32-48-sample-1
DEBUG: ++ Found new log: 2022/11/30/11/aws-waf-logs-delivery-stream-1-2022-11-30-11-32-48-sample-2
DEBUG: ++ Found new log: 2022/11/30/11/aws-waf-logs-delivery-stream-1-2022-11-30-11-32-48-sample-3
DEBUG: ++ Found new log: 2023/12/05/17/aws-waf-logs-delivery-stream-1-2023-12-05-17-32-48-792c6d1f-bfed-4b00-9f5e-88ce44718194
DEBUG: ++ Found new log: 2023/12/05/17/aws-waf-logs-delivery-stream-1-2023-12-05-17-32-48-No-Modifications
DEBUG: ++ Found new log: 2023/12/05/17/aws-waf-logs-delivery-stream-1-2023-12-05-17-32-48-multiple-values-in-ruleGroupList
DEBUG: ++ Found new log: 2023/12/08/17/aws-waf-logs-delivery-stream-1-2023-12-08-17-32-48-No-Modifications
DEBUG: ++ Found new log: 2023/12/08/17/aws-waf-logs-delivery-stream-1-2023-12-08-17-32-48-multiple-values-in-ruleGroupList
DEBUG: +++ DB Maintenance

The new unit tests for AWS WAF are pending creation.

javiersanchz · 2024-05-14T09:52:10Z

Update

The tests related to WAF were updated and added:

(unittest-env) wazuh@javier:~/Git/wazuh$ PYTHONPATH=/home/wazuh/Git/wazuh/api:/home/wazuh/Git/wazuh/framework python3 -m pytest wodles/aws/tests/test_waf.py -v
=============================================================================================== test session starts ===============================================================================================
platform linux -- Python 3.10.12, pytest-7.3.1, pluggy-1.4.0 -- /home/wazuh/venv/unittest-env/bin/python3
cachedir: .pytest_cache
metadata: {'Python': '3.10.12', 'Platform': 'Linux-6.5.0-17-generic-x86_64-with-glibc2.35', 'Packages': {'pytest': '7.3.1', 'pluggy': '1.4.0'}, 'Plugins': {'anyio': '4.3.0', 'aiohttp': '1.0.4', 'trio': '0.8.0', 'html': '2.1.1', 'metadata': '3.1.0', 'asyncio': '0.18.1', 'tavern': '1.23.5'}}
rootdir: /home/wazuh/Git/wazuh/wodles/aws/tests
configfile: pytest.ini
plugins: anyio-4.3.0, aiohttp-1.0.4, trio-0.8.0, html-2.1.1, metadata-3.1.0, asyncio-0.18.1, tavern-1.23.5
asyncio: mode=auto
collected 15 items                                                                                                                                                                                                

wodles/aws/tests/test_waf.py::test_aws_waf_bucket_initializes_properly PASSED                                                                                                                               [  6%]
wodles/aws/tests/test_waf.py::test_aws_waf_bucket_load_information_from_file[/home/wazuh/Git/wazuh/wodles/aws/tests/data/log_files/WAF/aws-waf-False] PASSED                                                [ 13%]
wodles/aws/tests/test_waf.py::test_aws_waf_bucket_load_information_from_file[/home/wazuh/Git/wazuh/wodles/aws/tests/data/log_files/WAF/aws-waf-True] PASSED                                                 [ 20%]
wodles/aws/tests/test_waf.py::test_aws_waf_bucket_load_information_from_file[/home/wazuh/Git/wazuh/wodles/aws/tests/data/log_files/WAF/aws-waf-invalid-json-True] PASSED                                    [ 26%]
wodles/aws/tests/test_waf.py::test_aws_waf_bucket_load_information_from_file[/home/wazuh/Git/wazuh/wodles/aws/tests/data/log_files/WAF/aws-waf-wrong-structure-True] PASSED                                 [ 33%]
wodles/aws/tests/test_waf.py::test_aws_waf_bucket_load_information_from_file_handles_exception_on_invalid_argument[/home/wazuh/Git/wazuh/wodles/aws/tests/data/log_files/WAF/aws-waf-invalid-json-False-SystemExit] PASSED [ 40%]
wodles/aws/tests/test_waf.py::test_aws_waf_bucket_load_information_from_file_handles_exception_on_invalid_argument[/home/wazuh/Git/wazuh/wodles/aws/tests/data/log_files/WAF/aws-waf-wrong-structure-False-SystemExit] PASSED [ 46%]
wodles/aws/tests/test_waf.py::test_aws_waf_bucket_check_waf_type[object_list0-True] PASSED                                                                                                                  [ 53%]
wodles/aws/tests/test_waf.py::test_aws_waf_bucket_check_waf_type[object_list1-False] PASSED                                                                                                                 [ 60%]
wodles/aws/tests/test_waf.py::test_aws_waf_bucket_check_waf_type_handles_exceptions PASSED                                                                                                                  [ 66%]
wodles/aws/tests/test_waf.py::test_aws_waf_bucket_get_service_prefix PASSED                                                                                                                                 [ 73%]
wodles/aws/tests/test_waf.py::test_aws_waf_bucket_get_base_prefix[True] PASSED                                                                                                                              [ 80%]
wodles/aws/tests/test_waf.py::test_aws_waf_bucket_get_base_prefix[False] PASSED                                                                                                                             [ 86%]
wodles/aws/tests/test_waf.py::test_aws_waf_bucket_iter_regions_and_accounts[True] PASSED                                                                                                                    [ 93%]
wodles/aws/tests/test_waf.py::test_aws_waf_bucket_iter_regions_and_accounts[False] PASSED                                                                                                                   [100%]

=============================================================================================== 15 passed in 0.27s ================================================================================================

(unittest-env) wazuh@javier:~/Git/wazuh$ PYTHONPATH=/home/wazuh/Git/wazuh/api:/home/wazuh/Git/wazuh/framework python3 -m pytest wodles/aws/tests
=============================================================================================== test session starts ===============================================================================================
platform linux -- Python 3.10.12, pytest-7.3.1, pluggy-1.4.0
rootdir: /home/wazuh/Git/wazuh/wodles/aws/tests
configfile: pytest.ini
plugins: anyio-4.3.0, aiohttp-1.0.4, trio-0.8.0, html-2.1.1, metadata-3.1.0, asyncio-0.18.1, tavern-1.23.5
asyncio: mode=auto
collected 598 items                                                                                                                                                                                               

wodles/aws/tests/test_aws_bucket.py ....................................................................................................................................................................... [ 27%]
....................................                                                                                                                                                                        [ 33%]
wodles/aws/tests/test_aws_s3.py ....................                                                                                                                                                        [ 37%]
wodles/aws/tests/test_aws_service.py ....                                                                                                                                                                   [ 37%]
wodles/aws/tests/test_cloudtrail.py ..                                                                                                                                                                      [ 38%]
wodles/aws/tests/test_cloudwatchlogs.py .....................................................                                                                                                               [ 47%]
wodles/aws/tests/test_config.py ..............................................................................                                                                                              [ 60%]
wodles/aws/tests/test_guardduty.py .................                                                                                                                                                        [ 63%]
wodles/aws/tests/test_inspector.py ......                                                                                                                                                                   [ 64%]
wodles/aws/tests/test_load_balancers.py ............                                                                                                                                                        [ 66%]
wodles/aws/tests/test_s3_log_handler.py ................                                                                                                                                                    [ 68%]
wodles/aws/tests/test_server_access.py .................................                                                                                                                                    [ 74%]
wodles/aws/tests/test_sqs_message_processor.py ........                                                                                                                                                     [ 75%]
wodles/aws/tests/test_sqs_queue.py .......                                                                                                                                                                  [ 76%]
wodles/aws/tests/test_tools.py ..................................                                                                                                                                           [ 82%]
wodles/aws/tests/test_umbrella.py ......                                                                                                                                                                    [ 83%]
wodles/aws/tests/test_vpcflow.py .....................                                                                                                                                                      [ 86%]
wodles/aws/tests/test_waf.py ...............                                                                                                                                                                [ 89%]
wodles/aws/tests/test_wazuh_integration.py ...............................................................                                                                                                  [100%]

=============================================================================================== 598 passed in 2.94s ===============================================================================================

javiersanchz · 2024-05-23T15:25:35Z

Update

The requested changes were checked and the tests for WAF v2 were run again.
As the changes were, errors were being generated when running the bucket with the following output:

root@wazuh-master:/# /var/ossec/wodles/aws/aws-s3 --bucket aws-waf-logs-wodle-v2  --aws_profile default --only_logs_after 2019-OCT-22 --regions us-east-1 --type waf --skip_on_error -d 2
DEBUG: +++ Debug mode on - Level: 2
DEBUG: No retries configuration found in profile config. Generating default configuration for retries: mode: standard - max_attempts: 10
DEBUG: Created Config object using profile: 'default' configuration
DEBUG: +++ Marker: AWSLogs/567970947422/WAFLogs/None/AWS-WAF-V2/2019/10/22
DEBUG: +++ No logs to process in bucket: aws-waf-logs-wodle-v2
DEBUG: +++ DB Maintenanc

As can be seen in the bucket, it is marking the region being iterated over as None. This issue arose due to the change introduced in the review #23397 (comment), without realizing that depending on the type of waf_type, it uses the method from one class or another.

This was corrected back to how it was before, the other requested changes were added, and the corresponding tests were updated:

Tests performed:

Tests performed

I run the WAF v2 bucket:

root@wazuh-master:/# /var/ossec/wodles/aws/aws-s3 --bucket aws-waf-logs-wodle-v2  --aws_profile default --only_logs_after 2019-OCT-22 --regions us-east-1 --type waf --skip_on_error -d 2
DEBUG: +++ Debug mode on - Level: 2
DEBUG: No retries configuration found in profile config. Generating default configuration for retries: mode: standard - max_attempts: 10
DEBUG: Created Config object using profile: 'default' configuration
DEBUG: +++ Table does not exist; create
DEBUG: +++ Working on 567970947422 - us-east-1
DEBUG: +++ Marker: AWSLogs/567970947422/WAFLogs/us-east-1/AWS-WAF-V2/2019/10/22
DEBUG: ++ Found new log: AWSLogs/567970947422/WAFLogs/us-east-1/AWS-WAF-V2/2024/05/03/11/10/567970947422_waflogs_us-east-1_AWS-WAF-V2_20240503T1110Z_a195076a.log.gz
DEBUG: ++ Found new log: AWSLogs/567970947422/WAFLogs/us-east-1/AWS-WAF-V2/2024/05/03/11/15/567970947422_waflogs_us-east-1_AWS-WAF-V2_20240503T1115Z_7409afbb.log.gz
DEBUG: +++ DB Maintenance

I run the bucket again after processing the logs:

root@wazuh-master:/# /var/ossec/wodles/aws/aws-s3 --bucket aws-waf-logs-wodle-v2  --aws_profile default --only_logs_after 2019-OCT-22 --regions us-east-1 --type waf --skip_on_error -d 2
DEBUG: +++ Debug mode on - Level: 2
DEBUG: No retries configuration found in profile config. Generating default configuration for retries: mode: standard - max_attempts: 10
DEBUG: Created Config object using profile: 'default' configuration
DEBUG: +++ Working on 567970947422 - us-east-1
DEBUG: +++ Marker: AWSLogs/567970947422/WAFLogs/us-east-1/AWS-WAF-V2/2024/05/03/11/15/567970947422_waflogs_us-east-1_AWS-WAF-V2_20240503T1115Z_7409afbb.log.gz
DEBUG: +++ No logs to process in bucket: 567970947422/us-east-1
DEBUG: +++ DB Maintenance

I run the Kinesis bucket:

root@wazuh-master:/# /var/ossec/wodles/aws/aws-s3 --bucket wazuh-aws-wodle-waf  --aws_profile default --only_logs_after 2019-OCT-22 --regions us-east-1 --type waf --skip_on_error -d 2
DEBUG: +++ Debug mode on - Level: 2
DEBUG: No retries configuration found in profile config. Generating default configuration for retries: mode: standard - max_attempts: 10
DEBUG: Created Config object using profile: 'default' configuration
The functionality to process WAF logs stored in S3 via Kinesis was deprecated in 5.0. Consider configuring WAF to store its logs directly in an S3 bucket instead. Check https://documentation.wazuh.com/current/amazon/services/supported-services/waf.html for more information.
DEBUG: +++ Marker: 2019/10/22
DEBUG: ++ Found new log: 2019/10/23/10/aws-waf-logs-delivery-stream-1-2019-10-23-10-32-48-792c6d1f-bfed-4b00-9f5e-88ce44718194
DEBUG: ++ Found new log: 2019/10/23/11/aws-waf-logs-delivery-stream-1-2019-10-23-11-32-48-792c6d1f-bfed-4b00-9f5e-88ce44718194
DEBUG: ++ Found new log: 2022/06/03/aws-waf-logs-delivery-stream-1-2022-06-03-10-32-48-792c6d1f-bfed-4b00-9f5e-88ce4471fake
ERROR: the 2022/06/03/aws-waf-logs-delivery-stream-1-2022-06-03-10-32-48-792c6d1f-bfed-4b00-9f5e-88ce4471fake file doesn't have the expected structure.
DEBUG: ++ Found new log: 2022/11/30/11/aws-waf-logs-delivery-stream-1-2022-11-30-11-32-48-sample-1
DEBUG: ++ Found new log: 2022/11/30/11/aws-waf-logs-delivery-stream-1-2022-11-30-11-32-48-sample-2
DEBUG: ++ Found new log: 2022/11/30/11/aws-waf-logs-delivery-stream-1-2022-11-30-11-32-48-sample-3
DEBUG: ++ Found new log: 2023/12/05/17/aws-waf-logs-delivery-stream-1-2023-12-05-17-32-48-792c6d1f-bfed-4b00-9f5e-88ce44718194
DEBUG: ++ Found new log: 2023/12/05/17/aws-waf-logs-delivery-stream-1-2023-12-05-17-32-48-No-Modifications
DEBUG: ++ Found new log: 2023/12/05/17/aws-waf-logs-delivery-stream-1-2023-12-05-17-32-48-multiple-values-in-ruleGroupList
DEBUG: ++ Found new log: 2023/12/08/17/aws-waf-logs-delivery-stream-1-2023-12-08-17-32-48-No-Modifications
DEBUG: ++ Found new log: 2023/12/08/17/aws-waf-logs-delivery-stream-1-2023-12-08-17-32-48-multiple-values-in-ruleGroupList
DEBUG: +++ DB Maintenance

The alerts generated from the logs were also verified.

The related tests were checked:

Tests

(unittest-env) wazuh@javier:~/Git/wazuh$ PYTHONPATH=/home/wazuh/Git/wazuh/api:/home/wazuh/Git/wazuh/framework python3 -m pytest wodles/aws/tests/test_waf.py
=============================================================================================== test session starts ===============================================================================================
platform linux -- Python 3.10.12, pytest-7.3.1, pluggy-1.4.0
rootdir: /home/wazuh/Git/wazuh/wodles/aws/tests
configfile: pytest.ini
plugins: anyio-4.3.0, aiohttp-1.0.4, trio-0.8.0, html-2.1.1, metadata-3.1.0, asyncio-0.18.1, tavern-1.23.5
asyncio: mode=auto
collected 15 items                                                                                                                                                                                                

wodles/aws/tests/test_waf.py ...............                                                                                                                                                                [100%]

=============================================================================================== 15 passed in 0.26s ================================================================================================

(unittest-env) wazuh@javier:~/Git/wazuh$ PYTHONPATH=/home/wazuh/Git/wazuh/api:/home/wazuh/Git/wazuh/framework python3 -m pytest wodles/aws/tests
=============================================================================================== test session starts ===============================================================================================
platform linux -- Python 3.10.12, pytest-7.3.1, pluggy-1.4.0
rootdir: /home/wazuh/Git/wazuh/wodles/aws/tests
configfile: pytest.ini
plugins: anyio-4.3.0, aiohttp-1.0.4, trio-0.8.0, html-2.1.1, metadata-3.1.0, asyncio-0.18.1, tavern-1.23.5
asyncio: mode=auto
collected 598 items                                                                                                                                                                                               

wodles/aws/tests/test_aws_bucket.py ....................................................................................................................................................................... [ 27%]
....................................                                                                                                                                                                        [ 33%]
wodles/aws/tests/test_aws_s3.py ....................                                                                                                                                                        [ 37%]
wodles/aws/tests/test_aws_service.py ....                                                                                                                                                                   [ 37%]
wodles/aws/tests/test_cloudtrail.py ..                                                                                                                                                                      [ 38%]
wodles/aws/tests/test_cloudwatchlogs.py .....................................................                                                                                                               [ 47%]
wodles/aws/tests/test_config.py ..............................................................................                                                                                              [ 60%]
wodles/aws/tests/test_guardduty.py .................                                                                                                                                                        [ 63%]
wodles/aws/tests/test_inspector.py ......                                                                                                                                                                   [ 64%]
wodles/aws/tests/test_load_balancers.py ............                                                                                                                                                        [ 66%]
wodles/aws/tests/test_s3_log_handler.py ................                                                                                                                                                    [ 68%]
wodles/aws/tests/test_server_access.py .................................                                                                                                                                    [ 74%]
wodles/aws/tests/test_sqs_message_processor.py ........                                                                                                                                                     [ 75%]
wodles/aws/tests/test_sqs_queue.py .......                                                                                                                                                                  [ 76%]
wodles/aws/tests/test_tools.py ..................................                                                                                                                                           [ 82%]
wodles/aws/tests/test_umbrella.py ......                                                                                                                                                                    [ 83%]
wodles/aws/tests/test_vpcflow.py .....................                                                                                                                                                      [ 86%]
wodles/aws/tests/test_waf.py ...............                                                                                                                                                                [ 89%]
wodles/aws/tests/test_wazuh_integration.py ...............................................................                                                                                                  [100%]

=============================================================================================== 598 passed in 2.59s ===============================================================================================

javiersanchz · 2024-05-27T16:46:21Z

Update

About the last requested changes:

Fixed syntax errors in the code.
Modified the get_full_prefix method to access the last key of the Contents directly and get the acl_name
The functionality of AWS WAF was tested again.
The tests were rerun, and I verified that they all passed.

GGP1 added type/enhancement New feature or request level/task labels Mar 15, 2024

javiersanchz self-assigned this Mar 21, 2024

javiersanchz changed the title ~~Migrate Amazon WAF service to v2~~ Modify AWS module to process logs from WAF v2 May 9, 2024

javiersanchz mentioned this issue May 9, 2024

Migrate Amazon WAF service to v2 #23361

Open

4 tasks

javiersanchz added level/subtask and removed level/task labels May 9, 2024

javiersanchz mentioned this issue May 9, 2024

Add integration tests for AWS WAF v2 #23365

Open

4 tasks

javiersanchz linked a pull request May 14, 2024 that will close this issue

Add waf v2 logs #23397

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Modify AWS module to process logs from WAF v2 #22572

Modify AWS module to process logs from WAF v2 #22572

GGP1 commented Mar 15, 2024 •

edited

Selutario commented Mar 18, 2024

javiersanchz commented Apr 25, 2024

javiersanchz commented Apr 26, 2024

javiersanchz commented Apr 29, 2024

javiersanchz commented Apr 30, 2024

javiersanchz commented May 2, 2024

javiersanchz commented May 3, 2024

javiersanchz commented May 3, 2024 •

edited

We talked with the rest of the team about which path should be processed within `waf.py`:

These paths are almost the same as those from `GuardDuty`:

javiersanchz commented May 6, 2024

javiersanchz commented May 9, 2024

javiersanchz commented May 13, 2024 •

edited

javiersanchz commented May 14, 2024

javiersanchz commented May 23, 2024 •

edited

javiersanchz commented May 27, 2024

Modify AWS module to process logs from WAF v2 #22572

Modify AWS module to process logs from WAF v2 #22572

Comments

GGP1 commented Mar 15, 2024 • edited

Description

Checks

Selutario commented Mar 18, 2024

javiersanchz commented Apr 25, 2024

Update

javiersanchz commented Apr 26, 2024

Update

javiersanchz commented Apr 29, 2024

Update

javiersanchz commented Apr 30, 2024

Update

javiersanchz commented May 2, 2024

Update

javiersanchz commented May 3, 2024

Update

javiersanchz commented May 3, 2024 • edited

Update

We talked with the rest of the team about which path should be processed within waf.py:

These paths are almost the same as those from GuardDuty:

javiersanchz commented May 6, 2024

Update

javiersanchz commented May 9, 2024

Update

javiersanchz commented May 13, 2024 • edited

Update

WAF native :

WAF kinesis :

javiersanchz commented May 14, 2024

Update

javiersanchz commented May 23, 2024 • edited

Update

Tests performed:

javiersanchz commented May 27, 2024

Update

GGP1 commented Mar 15, 2024 •

edited

javiersanchz commented May 3, 2024 •

edited

We talked with the rest of the team about which path should be processed within `waf.py`:

These paths are almost the same as those from `GuardDuty`:

javiersanchz commented May 13, 2024 •

edited

javiersanchz commented May 23, 2024 •

edited