Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[dataviz] perf: load reco with several HTTP requests #1969

Open
wants to merge 5 commits into
base: 1867-delete_video_api
Choose a base branch
from

Conversation

GresilleSiffle
Copy link
Collaborator

@GresilleSiffle GresilleSiffle commented May 13, 2024

related issues #1867


Description

The page "Videos and channels" now retrieves the recommendations from the API /polls/ instead of the legacy API /video/.

Instead of retrieving up to 99999 entities in one single HTTP request, the new method uses a thread pool and several requests to fetch batches of 2000 entities. The new method takes ~ 28 seconds vs ~ 50 seconds for the previous one.

We can tweak several parameters to adapt the server workload:

  • lowering the query parameter limit will reduce the execution time per request but will increase the total number of requests (and vice versa)
  • lowering the number of worker threads will reduce the number of concurrent HTTP requests and increase the loading time of the Streamlit page

Let me know if you think 2000 is too low or too high.

Checklist

  • I added the related issue(s) id in the related issues section (if any)
    • if not, delete the related issues section
  • I described my changes and my decisions in the PR description
  • I read the development guidelines of the CONTRIBUTING.md
  • The tests pass and have been updated if relevant
  • The code quality check pass

@GresilleSiffle GresilleSiffle changed the base branch from main to 1867-delete_video_api May 13, 2024 12:34
@GresilleSiffle GresilleSiffle changed the title [data-viz] refactor: load reco with several HTTP requests [dataviz] refactor: load reco with several HTTP requests May 13, 2024
@GresilleSiffle GresilleSiffle changed the title [dataviz] refactor: load reco with several HTTP requests [dataviz] perf: load reco with several HTTP requests May 13, 2024
@GresilleSiffle GresilleSiffle self-assigned this May 13, 2024
@GresilleSiffle GresilleSiffle marked this pull request as ready for review May 13, 2024 14:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant