Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(page): recent scans add page jumper #2348

Merged
merged 9 commits into from May 24, 2024
Merged

Conversation

miaoyc666
Copy link
Contributor

@miaoyc666 miaoyc666 commented Mar 1, 2024

Describe the Pull Request

When MobSF has a large amount of sample data in the system, the recent scans api speed has been very slow because it will retrieve full table data from the database.

To fix this, recent scans api add page jump, default display 10 items per page

Renderings
image

Checklist for PR

  • Run MobSF unit tests and lint tox -e lint,test
  • Tested Working on Linux, Mac, Windows, and Docker
  • Add unit test for any new Web API (Refer: StaticAnalyzer/tests.py)
  • Make sure tests are passing on your PR MobSF tests

Additional Comments (if any)

DESCRIBE HERE

@ajinabraham
Copy link
Member

Can you also include the changes in UI as well?

@miaoyc666
Copy link
Contributor Author

Can you also include the changes in UI as well?

Do you mean adding paging parameters in the fe as well? Of course, it's just that it takes time. 😂

@miaoyc666
Copy link
Contributor Author

Can you also include the changes in UI as well?

@ajinabraham ui done!

@ohyeah521
Copy link
Contributor

@miaoyc666 Do you have any suggestions for optimizing the database? I found that when the amount of data exceeds 30,000, data query will become extremely slow, even if the page display is still very slow

@miaoyc666 miaoyc666 changed the title feat(page): recent scans add default page and page_size, list 100 items feat(page): recent scans add page jumper Apr 5, 2024
@miaoyc666
Copy link
Contributor Author

@miaoyc666 Do you have any suggestions for optimizing the database? I found that when the amount of data exceeds 30,000, data query will become extremely slow, even if the page display is still very slow

@ohyeah521

When you query a database, the first thing to control is the number of rows you query. You need to add offset and limit parameters to your SQL statement for pagination queries, limiting the number of rows returned.
Secondly, you need to select the fields that need to be displayed for select queries. Do not search for useless fields and cannot use select * because it will query all fields in a record, which will consume additional network bandwidth and CPU computing resources.
You can take a look at the code I commited, perform pagination queries in django, and only query the fields displayed on the page in the recent list of samples.

@ohyeah521
Copy link
Contributor

@miaoyc666 thanks

@miaoyc666 miaoyc666 marked this pull request as draft April 7, 2024 11:21
@miaoyc666 miaoyc666 marked this pull request as ready for review April 8, 2024 02:05
@miaoyc666 miaoyc666 marked this pull request as draft May 15, 2024 03:39
@miaoyc666 miaoyc666 closed this May 15, 2024
@miaoyc666 miaoyc666 reopened this May 15, 2024
@miaoyc666 miaoyc666 marked this pull request as ready for review May 15, 2024 03:41
@ajinabraham
Copy link
Member

@miaoyc666 Thanks for taking your time and working on the UI bits. I will review this soon and get back to you.

@ajinabraham ajinabraham self-requested a review May 20, 2024 16:56
@ajinabraham
Copy link
Member

Initial testing looks good. I will see if I can improve up on the UI based on https://realpython.com/django-pagination/

@ajinabraham ajinabraham merged commit cf390e2 into MobSF:master May 24, 2024
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants