Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use pubsub and blocking for improved response times #404

Open
m-kuhn opened this issue May 31, 2023 · 6 comments
Open

Use pubsub and blocking for improved response times #404

m-kuhn opened this issue May 31, 2023 · 6 comments

Comments

@m-kuhn
Copy link

m-kuhn commented May 31, 2023

arq currently uses a polling mode (default 0.5 seconds) to fetch jobs on worker side and also to fetch results.
I think it should be possible to use blocking requests (on worker side) and pubsub (possibly with keyspace notifications, but not necessarily) for fetching the result.

This should be quite reliable in most scenarios, however it can occur that a job gets lost (there is no guarantee for pubsub delivery after all), so one could still poll from time to time to be on the safe side.

Are there any opinions on development in this direction?

From #374 (comment)

@m-kuhn m-kuhn changed the title Use pubsub and blocking for a quicker roundtrip Use pubsub and blocking for improved response times May 31, 2023
@m-kuhn
Copy link
Author

m-kuhn commented Jun 17, 2023

CC @arturzx

@gg718
Copy link

gg718 commented Jun 27, 2023

See also how rq recently implemented this: rq/rq#1939

@JonasKs
Copy link
Sponsor Collaborator

JonasKs commented Jun 29, 2023

A draft PR welcome.😊

@samuelcolvin
Copy link
Owner

My plan is to move to redis streams which are by far the best solution for tasks like those performed by arq. Stay tuned for a big update.

@samuelcolvin
Copy link
Owner

See #437.

@Wh1isper
Copy link

Wh1isper commented May 19, 2024

I agree that redis stream is the best option right now. Can't wait for the arq update, I developed a redis stream based library, aka brq. If you're in a hurry, you may try it now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants