Fixes for report generation #74
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This a a checkpoint before we break things
Made it so that the API gets called on startup to ensure the connection is viable.
Created the skeleton of the POC. Adding in the initial elements for API and database fetching. I also left a couple of pseudo code block so Sahim can see where things are planned to go
Got the email check done. The system now detects newly breached emails and adds them to the database. Checks all breached emails for new breaches, adding them to a list if they need updating. I also re-formatted the API requesting so that we can better account for unhappy response codes, and to add in recursion for when the API needs a cool down time.
Finished up the email update section. Now all that's left is to generate the report and finalise the POC
Finally. After a long day of non-stop programming it is complete. The system can fully check the domain with HIBP. detect the differences with the local database, correctly make updates, and report all the changes in a text file.
Added in the functionality for checking with the HIBP API for new breaches. This ensures that we aren't doing domains scans when there's no new information available.
Changed the way time_to_sleep was collected so that different messages with code:429 didn't crash the system.
I forgot to add in the thing that extracted the dict of breached emails, when receiving a successful API return. Oops
* Working New Identity assessment container * Before we break things This a a checkpoint before we break things * API check on startup Made it so that the API gets called on startup to ensure the connection is viable. * Created the Skeleton of the POC Created the skeleton of the POC. Adding in the initial elements for API and database fetching. I also left a couple of pseudo code block so Sahim can see where things are planned to go * 2/3 functional Got the email check done. The system now detects newly breached emails and adds them to the database. Checks all breached emails for new breaches, adding them to a list if they need updating. I also re-formatted the API requesting so that we can better account for unhappy response codes, and to add in recursion for when the API needs a cool down time. * Finished the email update Finished up the email update section. Now all that's left is to generate the report and finalise the POC * Finished the POC Finally. After a long day of non-stop programming it is complete. The system can fully check the domain with HIBP. detect the differences with the local database, correctly make updates, and report all the changes in a text file. * New Breaches Functionality Added in the functionality for checking with the HIBP API for new breaches. This ensures that we aren't doing domains scans when there's no new information available. * Retry-After header Changed the way time_to_sleep was collected so that different messages with code:429 didn't crash the system. * Forgot to add in domain functionality I forgot to add in the thing that extracted the dict of breached emails, when receiving a successful API return. Oops * Docker push * removed test
…domain files when a domain is deleted from the domain table.
* updates * fix demo user * sleep entrypoint * add ips as inactive * chore(deps): update docker/setup-buildx-action action to v3.3.0 * pg version fix * Updates * delete obsolete verison in docker * fix ipv6 issue * fix ip status issue * update project * python break-system-packages * add tags --------- Co-authored-by: faustro <[email protected]> Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: Faustin Roman <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Resolved issues with the report generation with the scan_email module so that reports correctly recoded the newly discovered breaches.
Changed the 'emails_domain_id_foreign' constraint to cascade the delete. This meant that emails that were imported with a domain scan would be removed with them too. (This wouldn't affect manually entered emails)
Finally made an attempt at creating a trigger for the domains table, so that when a domain is deleted the associated folder in /results would be removed. However we were unable to test or prove it's functianlity. (Warning: current 'delete_domain_files' script will also remove files from manually entered email under the same domain.)