Complete isolated generation of resources in AWS IT #22514
Assignees
Labels
Comments
8 tasks
Issue Update
|
Issue UpdateMade the necessary modifications to run the test_discard_regex.pyroot@vagrant:/wazuh/tests/integration/test_aws# pytest -x test_discard_regex.py
=============================================================================================== test session starts ===============================================================================================
platform linux -- Python 3.10.12, pytest-7.1.2, pluggy-1.4.0
rootdir: /wazuh/tests/integration, configfile: pytest.ini
plugins: metadata-3.1.1, html-3.1.1
collected 17 items
test_discard_regex.py .F
==================================================================================================== FAILURES =====================================================================================================
__________________________________________________________________________________ test_bucket_discard_regex[vpc_discard_regex] ___________________________________________________________________________________
configuration = {'metadata': {'bucket_name': 'wazuh-vpcflow-integration-tests-99a3fa46-todelete', 'bucket_type': 'vpcflow', 'descripti...elements': [{'disabled': {'value': 'no'}}, {'bucket': {'attributes': [...], 'elements': [...]}}], 'section': 'wodle'}]}
metadata = {'bucket_name': 'wazuh-vpcflow-integration-tests-99a3fa46-todelete', 'bucket_type': 'vpcflow', 'description': 'VPC discard regex configurations', 'discard_field': 'srcport', ...}
create_test_bucket = None, manage_bucket_files = None, load_wazuh_basic_configuration = None, set_wazuh_configuration = None, clean_s3_cloudtrail_db = None, configure_local_internal_options_function = None
truncate_monitored_files = None, restart_wazuh_function = None, file_monitoring = None
@pytest.mark.tier(level=0)
@pytest.mark.parametrize('configuration, metadata',
zip(configurator.test_configuration_template, configurator.metadata),
ids=configurator.cases_ids)
def test_bucket_discard_regex(
configuration, metadata, create_test_bucket, manage_bucket_files, load_wazuh_basic_configuration,
set_wazuh_configuration, clean_s3_cloudtrail_db, configure_local_internal_options_function,
truncate_monitored_files, restart_wazuh_function, file_monitoring,
):
"""
description: Check that some bucket logs are excluded when the regex and field defined in <discard_regex>
match an event.
test_phases:
- setup:
- Load Wazuh light configuration.
- Apply ossec.conf configuration changes according to the configuration template and use case.
- Apply custom settings in local_internal_options.conf.
- Truncate wazuh logs.
- Restart wazuh-manager service to apply configuration changes.
- test:
- Check in the ossec.log that a line has appeared calling the module with correct parameters.
- Check the expected number of events were forwarded to analysisd, only logs stored in the bucket and skips
the ones that match with regex.
- Check the database was created and updated accordingly.
- teardown:
- Truncate wazuh logs.
- Restore initial configuration, both ossec.conf and local_internal_options.conf.
wazuh_min_version: 4.6.0
parameters:
- configuration:
type: dict
brief: Get configurations from the module.
- metadata:
type: dict
brief: Get metadata from the module.
- load_wazuh_basic_configuration:
type: fixture
brief: Load basic wazuh configuration.
- set_wazuh_configuration:
type: fixture
brief: Apply changes to the ossec.conf configuration.
- clean_s3_cloudtrail_db:
type: fixture
brief: Delete the DB file before and after the test execution.
- configure_local_internal_options_function:
type: fixture
brief: Apply changes to the local_internal_options.conf configuration.
- truncate_monitored_files:
type: fixture
brief: Truncate wazuh logs.
- restart_wazuh_daemon_function:
type: fixture
brief: Restart the wazuh service.
- file_monitoring:
type: fixture
brief: Handle the monitoring of a specified file.
assertions:
- Check in the log that the module was called with correct parameters.
- Check the expected number of events were forwarded to analysisd.
- Check the database was created and updated accordingly.
input_description:
- The `configuration_bucket_discard_regex` file provides the module configuration for this test.
- The `cases_bucket_discard_regex` file provides the test cases.
"""
bucket_name = metadata['bucket_name']
bucket_type = metadata['bucket_type']
only_logs_after = metadata['only_logs_after']
discard_field = metadata['discard_field']
discard_regex = metadata['discard_regex']
found_logs = metadata['found_logs']
skipped_logs = metadata['skipped_logs']
path = metadata['path'] if 'path' in metadata else None
pattern = fr'.*The "{discard_regex}" regex found a match in the "{discard_field}" field.' \
' The event will be skipped.'
parameters = [
'wodles/aws/aws-s3',
'--bucket', bucket_name,
'--only_logs_after', only_logs_after,
'--discard-field', discard_field,
'--discard-regex', discard_regex,
'--type', bucket_type,
'--debug', '2'
]
if path is not None:
parameters.insert(5, path)
parameters.insert(5, '--trail_prefix')
# Check AWS module started
log_monitor.start(
timeout=session_parameters.default_timeout,
callback=event_monitor.callback_detect_aws_module_start
)
assert log_monitor.callback_result is not None, ERROR_MESSAGE['failed_start']
# Check command was called correctly
log_monitor.start(
timeout=session_parameters.default_timeout,
callback=event_monitor.callback_detect_aws_module_called(parameters)
)
assert log_monitor.callback_result is not None, ERROR_MESSAGE['incorrect_parameters']
log_monitor.start(
timeout=TIMEOUT[20],
callback=event_monitor.callback_detect_event_processed_or_skipped(pattern),
accumulations=found_logs + skipped_logs
)
> assert log_monitor.callback_result is not None, ERROR_MESSAGE['incorrect_discard_regex_message']
E AssertionError: The AWS module did not show the correct message about discard regex or, did not process the expected amount of logs
E assert None is not None
E + where None = <wazuh_testing.tools.monitors.file_monitor.FileMonitor object at 0x7fca5f6f69b0>.callback_result
test_discard_regex.py:149: AssertionError
----------------------------------------------------------------------------------------------- Captured log setup ------------------------------------------------------------------------------------------------
DEBUG wazuh_testing:conftest.py:182 Created new bucket: type wazuh-vpcflow-integration-tests-99a3fa46-todelete
DEBUG wazuh_testing:conftest.py:231 Uploaded file: AWSLogs/819751203818/vpcflowlogs/us-east-1/2024/03/25/819751203818_vpcflowlogs_us-east-1_fl-0754d951c16f517fa_20240325T2118Z_1376354592399921175.log to bucket "wazuh-vpcflow-integration-tests-99a3fa46-todelete"
DEBUG wazuh_testing:conftest.py:183 Set local_internal_option to {'wazuh_modules.debug': '2', 'monitord.rotate_log': '0'}
DEBUG wazuh_testing:conftest.py:206 Restarting all daemon
DEBUG wazuh_testing:conftest.py:242 Initializing file to monitor to /var/ossec/logs/ossec.log
---------------------------------------------------------------------------------------------- Captured log teardown ----------------------------------------------------------------------------------------------
DEBUG wazuh_testing:conftest.py:250 Trucanted /var/ossec/logs/ossec.log
DEBUG wazuh_testing:conftest.py:218 Stopping all daemons
DEBUG wazuh_testing:conftest.py:188 Restore local_internal_option to {}
============================================================================================= short test summary info =============================================================================================
FAILED test_discard_regex.py::test_bucket_discard_regex[vpc_discard_regex] - AssertionError: The AWS module did not show the correct message about discard regex or, did not process the expected amount of logs
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
===================================================================================== 1 failed, 1 passed in 86.75s (0:01:26) ====================================================================================== |
Issue UpdateThe following errors were found during the Analysis of each errorVPCThis error is related to how the AWS module implements the feature to fetch VPC Flow logs. To achieve this it is necessary to have an active Flow Log inside an EC2 Network interface. In the current tests, these two resources are static and have been created to run the tests in a determined environment, achieving the goal of correctly fetching the logs. The implementation of the functions for the VPC resources will impact every related test, not only the one for Cisco UmbrellaThis error was related to the following reasons:
These two causes were already fixed and tested. Nevertheless, it was found that modifying the Services errorsThis is related to the task mentioned in the issue's description: |
Issue UpdateThe fixtures for the creation and deletion of the Which can be verified in the The methods will be extended and enhanced for the simple text case. Also, it remains to be checked how the |
Issue UpdateCloudWatch Logs simple text discard regex feature test succesfully modified: root@vagrant:/# pytest wazuh/tests/integration/test_aws/test_discard_regex.py::test_cloudwatch_discard_regex_simple_text[cloudwatch_discard_regex_simple_text]
====================================================================== test session starts ======================================================================
platform linux -- Python 3.10.12, pytest-7.1.2, pluggy-1.4.0
rootdir: /wazuh/tests/integration, configfile: pytest.ini
plugins: metadata-3.1.1, html-3.1.1
collected 1 item
wazuh/tests/integration/test_aws/test_discard_regex.py . [100%]
====================================================================== 1 passed in 22.43s =======================================================================The next task was modifying and adding the necessary functions to generate the Inspector Classic service findings. Reading the AWS docs, a finding is a detailed report about a vulnerability that affects one of the user's AWS resources. Therefore, a resource is needed to run the report on it to generate the findings. In wazuh/wazuh-qa#3345, a template was generated with that objective and then the report was run against an EC2 instance with a determined tag (one that matched the The alternative is to follow the first approach regarding the Inspector service, having previously obtained findings in the service. These do not generate the race condition that was the cause of wazuh/wazuh-qa#4714 since new resources were never inserted into the service. |
Issue UpdateAfter discussing with the team, it was decided that trying to generate new Inspector findings would be an overkill for the current development since it would require to set up an EC2 instance, an Inspector template and running the assesment to obtain new findings in each run. This would make the test slower and would consume more resources that do not have the risk of the race condition present in the bucket files. Therefore, new findings were generated in the # pytest wazuh/tests/integration/test_aws/test_discard_regex.py::test_inspector_discard_regex[inspector_discard_regex]
====================================================================== test session starts ======================================================================
platform linux -- Python 3.10.12, pytest-7.1.2, pluggy-1.4.0
rootdir: /wazuh/tests/integration, configfile: pytest.ini
plugins: metadata-3.1.1, html-3.1.1
collected 1 item
wazuh/tests/integration/test_aws/test_discard_regex.py . [100%]
====================================================================== 1 passed in 31.65s =======================================================================VPC resources generationboto3 methods and resources management in AWSThe following EC2 methods are the ones to be used to generate and delete the Flow logs: The flow logs can be generated for Network interfaces, as suggested in our documentation, but these would require to create a VPC, a Subnet, and a Network interface before proceding with the flow log. Therefore, since the flow logs can also be created for VPC instances, we could create these for each run and an associated flow log id that would match the file to upload to the S3 bucket. This approach has already been tested:
Inclusion of the required methods in the current codeCurrently, the An analysis on where the creation and deletion of the necessary resources is on going, since we should not break the consistency of the code already developed or give sections of the tests or the framework responsibilities that do not correspond. |
Issue Update
|
Issue Update
root@vagrant:/# pytest /wazuh/tests/integration/test_aws/test_discard_regex.py
====================================================================== test session starts ======================================================================
platform linux -- Python 3.10.12, pytest-7.1.2, pluggy-1.4.0
rootdir: /wazuh/tests/integration, configfile: pytest.ini
plugins: metadata-3.1.1, html-3.1.1
collected 17 items
wazuh/tests/integration/test_aws/test_discard_regex.py ................. [100%]
================================================================ 17 passed in 601.95s (0:10:01) =================================================================
|
Issue Update
root@vagrant:/# pytest -x /wazuh/tests/integration/test_aws/test_discard_regex.py
====================================================================== test session starts ======================================================================
platform linux -- Python 3.10.12, pytest-7.1.2, pluggy-1.4.0
rootdir: /wazuh/tests/integration, configfile: pytest.ini
plugins: metadata-3.1.1, html-3.1.1
collected 17 items
wazuh/tests/integration/test_aws/test_discard_regex.py ................. [100%]
================================================================ 17 passed in 392.74s (0:06:32) =================================================================Note Changes already pushed to the
test_log_groups.pyroot@vagrant:/# pytest -x /wazuh/tests/integration/test_aws/test_log_groups.py
====================================================================== test session starts ======================================================================
platform linux -- Python 3.10.12, pytest-7.1.2, pluggy-1.4.0
rootdir: /wazuh/tests/integration, configfile: pytest.ini
plugins: metadata-3.1.1, html-3.1.1
collected 2 items
wazuh/tests/integration/test_aws/test_log_groups.py F
=========================================================================== FAILURES ============================================================================
_____________________________________________________ test_log_groups[cloudwatchlogs_log_groups_with_data] ______________________________________________________
configuration = {'metadata': {'description': 'CloudWatch log groups configurations', 'expected_results': 3, 'log_group_name': 'wazuh-c...lements': [{'disabled': {'value': 'no'}}, {'service': {'attributes': [...], 'elements': [...]}}], 'section': 'wodle'}]}
metadata = {'description': 'CloudWatch log groups configurations', 'expected_results': 3, 'log_group_name': 'wazuh-cloudwatchlogs...og-group-b18a8eeb-todelete', 'log_stream_name': 'wazuh-cloudwatchlogs-integration-tests-stream-b18a8eeb-todelete', ...}
create_test_log_group = None, create_test_log_stream = None, manage_log_group_events = None, load_wazuh_basic_configuration = None
set_wazuh_configuration = None, clean_aws_services_db = None, configure_local_internal_options_function = None, truncate_monitored_files = None
restart_wazuh_function = None, file_monitoring = None
@pytest.mark.tier(level=0)
@pytest.mark.parametrize('configuration, metadata',
zip(configurator.test_configuration_template, configurator.metadata),
ids=configurator.cases_ids)
def test_log_groups(
configuration, metadata, create_test_log_group, create_test_log_stream, manage_log_group_events,
load_wazuh_basic_configuration, set_wazuh_configuration, clean_aws_services_db,
configure_local_internal_options_function, truncate_monitored_files, restart_wazuh_function, file_monitoring,
):
"""
description: Only the events for the specified log_group are processed.
test_phases:
- setup:
- Load Wazuh light configuration.
- Apply ossec.conf configuration changes according to the configuration template and use case.
- Apply custom settings in local_internal_options.conf.
- Truncate wazuh logs.
- Restart wazuh-manager service to apply configuration changes.
- test:
- Check in the ossec.log that a line has appeared calling the module with correct parameters.
- If a region that does not exist was specified, make sure that a message is displayed in the ossec.log
warning the user.
- Check the expected number of events were forwarded to analysisd, only logs stored in the bucket
for the specified region.
- Check the database was created and updated accordingly.
- teardown:
- Truncate wazuh logs.
- Restore initial configuration, both ossec.conf and local_internal_options.conf.
- Delete the uploaded file.
wazuh_min_version: 4.6.0
parameters:
- configuration:
type: dict
brief: Get configurations from the module.
- metadata:
type: dict
brief: Get metadata from the module.
- create_log_stream:
type: fixture
brief: Create a log stream with events for the day of execution.
- load_wazuh_basic_configuration:
type: fixture
brief: Load basic wazuh configuration.
- set_wazuh_configuration:
type: fixture
brief: Apply changes to the ossec.conf configuration.
- clean_aws_services_db:
type: fixture
brief: Delete the DB file before and after the test execution.
- configure_local_internal_options_function:
type: fixture
brief: Apply changes to the local_internal_options.conf configuration.
- truncate_monitored_files:
type: fixture
brief: Truncate wazuh logs.
- restart_wazuh_daemon_function:
type: fixture
brief: Restart the wazuh service.
- file_monitoring:
type: fixture
brief: Handle the monitoring of a specified file.
assertions:
- Check in the log that the module was called with correct parameters.
- Check the expected number of events were forwarded to analysisd.
- Check the database was created and updated accordingly, using the correct path for each entry.
input_description:
- The `configuration_regions` file provides the module configuration for this test.
- The `cases_regions` file provides the test cases.
"""
service_type = metadata['service_type']
log_group_names = metadata['log_group_name']
expected_results = metadata['expected_results']
parameters = [
'wodles/aws/aws-s3',
'--service', service_type,
'--only_logs_after', '2023-JAN-12',
'--regions', 'us-east-1',
'--aws_log_groups', log_group_names,
'--debug', '2'
]
# Check AWS module started
log_monitor.start(
timeout=session_parameters.default_timeout,
callback=event_monitor.callback_detect_aws_module_start
)
assert log_monitor.callback_result is not None, ERROR_MESSAGE['failed_start']
# Check command was called correctly
log_monitor.start(
timeout=session_parameters.default_timeout,
callback=event_monitor.callback_detect_aws_module_called(parameters)
)
if expected_results:
log_monitor.start(
timeout=TIMEOUT[20],
callback=event_monitor.callback_detect_service_event_processed(expected_results, service_type),
accumulations=len(log_group_names.split(','))
)
else:
log_monitor.start(
timeout=TIMEOUT[10],
callback=event_monitor.make_aws_callback(pattern=fr"{NON_EXISTENT_SPECIFIED_LOG_GROUPS}")
)
assert log_monitor.callback_result is not None, ERROR_MESSAGE['incorrect_no_existent_log_group']
assert path_exist(path=AWS_SERVICES_DB_PATH)
if expected_results:
log_group_list = log_group_names.split(",")
for row in get_multiple_service_db_row(table_name='cloudwatch_logs'):
> assert row.aws_log_group in log_group_list
E AssertionError: assert 'wazuh-cloudwatchlogs-integration-tests' in ['wazuh-cloudwatchlogs-integration-tests-b18a8eeb-todelete', 'temporary-log-group-b18a8eeb-todelete']
E + where 'wazuh-cloudwatchlogs-integration-tests' = ServiceCloudWatchRow(aws_region='us-east-1', aws_log_group='wazuh-cloudwatchlogs-integration-tests', aws_log_stream='w...token='f/38198203508651813109938003761325147006176055714992062463/s', start_time=1673481600000, end_time=1673481600000).aws_log_group
wazuh/tests/integration/test_aws/test_log_groups.py:151: AssertionError
---------------------------------------------------------------------- Captured log setup -----------------------------------------------------------------------
DEBUG wazuh_testing:conftest.py:315 Created log group: wazuh-cloudwatchlogs-integration-tests-b18a8eeb-todelete
DEBUG wazuh_testing:conftest.py:315 Created log group: temporary-log-group-b18a8eeb-todelete
DEBUG wazuh_testing:conftest.py:360 Created log stream wazuh-cloudwatchlogs-integration-tests-stream-b18a8eeb-todelete within log group wazuh-cloudwatchlogs-integration-tests-b18a8eeb-todelete
DEBUG wazuh_testing:conftest.py:360 Created log stream wazuh-cloudwatchlogs-integration-tests-stream-b18a8eeb-todelete within log group temporary-log-group-b18a8eeb-todelete
DEBUG wazuh_testing:conftest.py:183 Set local_internal_option to {'wazuh_modules.debug': '2', 'monitord.rotate_log': '0'}
DEBUG wazuh_testing:conftest.py:206 Restarting all daemon
DEBUG wazuh_testing:conftest.py:242 Initializing file to monitor to /var/ossec/logs/ossec.log
----------------------------------------------------------------------- Captured log call -----------------------------------------------------------------------
INFO wazuh_testing:db_administrator.py:19 Connection established with /var/ossec/wodles/aws/aws_services.db
--------------------------------------------------------------------- Captured log teardown ---------------------------------------------------------------------
DEBUG wazuh_testing:conftest.py:250 Trucanted /var/ossec/logs/ossec.log
DEBUG wazuh_testing:conftest.py:218 Stopping all daemons
DEBUG wazuh_testing:conftest.py:188 Restore local_internal_option to {'wazuh_modules.debug': '2\n', 'monitord.rotate_log': '0\n'}
==================================================================== short test summary info ====================================================================
FAILED wazuh/tests/integration/test_aws/test_log_groups.py::test_log_groups[cloudwatchlogs_log_groups_with_data] - AssertionError: assert 'wazuh-cloudwatchlog...
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
====================================================================== 1 failed in 44.72s =======================================================================The failure is related to hardcoded values in the services database, which contains log group names values defined in the A solution is being studied. |
Issue Update
root@vagrant:/# pytest -x /wazuh/tests/integration/test_aws/test_log_groups.py
====================================================================== test session starts ======================================================================
platform linux -- Python 3.10.12, pytest-7.1.2, pluggy-1.4.0
rootdir: /wazuh/tests/integration, configfile: pytest.ini
plugins: metadata-3.1.1, html-3.1.1
collected 2 items
wazuh/tests/integration/test_aws/test_log_groups.py .. [100%]
====================================================================== 2 passed in 55.00s =======================================================================
Given how the parameter works, it is needed to be able to upload files from different dates for the module to retrieve them accordingly. Added to this is the need for variety in the logs detected when the modifications for the discard_regex tests were made. Therefore, the development will continue with the necessary modifications to the |
This was referenced Apr 15, 2024
Issue Update
|
Issue Update
|
Issue Update
|
Issue Update
|
Issue Update
|
6 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Description
We need to complete the resources isolation that @EduLeon12 started in wazuh/wazuh-qa#4714. These are the pending tasks according to what he described in the last update:
Note
For this development it will be necessary to use 4714-generate-isolated-resources-for-aws-its as the base branch (link). This is because the PR of the previous issue has not yet been merged, but the delivery date requires that we parallelize both issues. In theory it shouldn't be a problem.
The text was updated successfully, but these errors were encountered: