Support custom fields in DefectDojo findings to fill specific custom fields in Jira

[FallenAtticus]

Description

In this PR I want to enhance the generic report importer of DefectDojo to extract some custom fields/columns from report files in order to send this custom fields to Jira to fill the mapped custom fields with the values from the report.

Let's assume you have a report with all of the required columns but also have some more columns you also want to have in your findings.

If you supply the following custom field mapping during the import / reimport:
{"customfield_12345": "CustomColumnA", "customfield_12346": "CustomColumnB"}

The importer would map the columns into a new json map with the custom field ids as key and the value of each report row as value. For example:
{"customfield_12345": "USD123.00", "customfield_12346": "96%"}

This json is stored in a new field in the finding table called "custom_fields".

Jira will merge the finding custom fields with existing product custom fields and send it to Jira.

Test results

I have added a bunch of unittests for this change.

Documentation

I am not sure if the documentation needs to be updated. Please let me know.

Checklist

This checklist is for your information.

• Make sure to rebase your PR against the very latest dev.
• Features/Changes should be submitted against the dev.
• Bugfixes should be submitted against the bugfix branch.
• Give a meaningful name to your PR, as it may end up being used in the release notes.
• Your code is flake8 compliant.
• Your code is python 3.11 compliant.
• If this is a new feature and not a bug fix, you've included the proper documentation in the docs at https://github.com/DefectDojo/django-DefectDojo/tree/dev/docs as part of this PR.
• Model changes must include the necessary migrations in the dojo/db_migrations folder.
• Add applicable tests to the unit tests.
• Add the proper label to categorize your PR.

[dryrunsecurity]

Hi there 👋, @DryRunSecurity here, below is a summary of our analysis and findings.

DryRun Security	Status	Findings
Configured Codepaths Analyzer	❌	49 findings
Sensitive Files Analyzer	✅	0 findings
AppSec Analyzer	✅	0 findings
Authn/Authz Analyzer	❕	1 finding
Secrets Analyzer	✅	0 findings

Note

🔴 Risk threshold exceeded. Adding a reviewer if one is configured in .dryrunsecurity.yaml.

notification list: @mtesauro @grendel513

Change Summary (click to expand)

The following is a summary of changes in this pull request made by me, your security buddy 🤖. Note that this summary is auto-generated and not meant to be a definitive list of security issues but rather a helpful summary from a security perspective.

Summary:

The code changes in this pull request focus on improving the functionality and security of the Defect Dojo application, particularly in the areas of Jira integration, custom field handling, and parsing of security scan reports.

The key changes include:

1. Jira Integration Enhancements: The changes in the dojo/jira_link/helper.py file improve the integration between Defect Dojo and Jira, allowing for better synchronization of finding statuses, custom fields, labels, and screenshots between the two systems.

2. Custom Fields Support: Several files, such as dojo/engagement/views.py, dojo/forms.py, dojo/models.py, and dojo/tools/generic/csv_parser.py, have been updated to provide better support for handling custom fields associated with findings and other entities. This includes the ability to map custom fields from security scan reports to the corresponding Defect Dojo fields.

3. Parsing Improvements: The changes in the dojo/tools/generic/parser.py, dojo/tools/generic/json_parser.py, and unittests/tools/test_generic_parser.py files focus on enhancing the parsing capabilities of the application, particularly for handling custom columns and different JSON structures in security scan reports.

4. Input Validation and Sanitization: While the changes generally do not introduce any obvious security vulnerabilities, the application security engineer should review the code to ensure that all user-supplied data is properly validated and sanitized to prevent potential issues such as injection attacks or unintended behavior.

Files Changed:

1. dojo/engagement/views.py: Changes related to the handling of Jira integration and custom fields.
2. dojo/db_migrations/0212_finding_custom_fields.py: Adds a new custom_fields field to the Finding model.
3. dojo/api_v2/serializers.py: Adds a custom_fields_mapping field to the ImportScanSerializer and ReImportScanSerializer classes.
4. dojo/filters.py: Defines various filter classes for the Defect Dojo application.
5. dojo/importers/base_importer.py: Improvements to the handling of get_findings and get_tests methods in the parser class.
6. dojo/forms.py: Adds a custom_fields_mapping field to the JIRAImportScanForm class.
7. dojo/jira_link/helper.py: Enhancements to the Jira integration functionality.
8. dojo/models.py: Adds a custom_fields field to the Finding model.
9. dojo/templates/dojo/view_finding.html: Adds a new column to display Jira custom fields.
10. dojo/templatetags/as_key_value_list.py: Introduces a new template filter for displaying dictionary data.
11. dojo/tools/generic/csv_parser.py: Adds support for custom fields mapping in the GenericCSVParser class.
12. dojo/tools/generic/parser.py: Updates the GenericParser class to handle both CSV and JSON file formats.
13. dojo/tools/generic/json_parser.py: Enhances the GenericJSONParser class to support custom fields mapping and input validation.
14. unittests/scans/generic/generic_report5.json: Adds new findings to a JSON-based security report.
15. unittests/scans/generic/generic_report5.csv: Adds new findings to a CSV-based security report.
16. unittests/tools/test_generic_parser.py: Adds new test cases for the GenericParser class, including custom columns and JSON parsing.

Powered by DryRun Security

[Maffooch]

Hi @FallenAtticus thank you for producing this feature! I will discuss it with other moderators of the project to determine how we will proceed with it

[github-actions]

This pull request has conflicts, please resolve those before we can evaluate the pull request.

[github-actions]

Conflicts have been resolved. A maintainer will review the pull request shortly.