Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WIP: Implement Redis queue system #4316

Closed
2 tasks
btylerburton opened this issue May 11, 2023 · 1 comment
Closed
2 tasks

WIP: Implement Redis queue system #4316

btylerburton opened this issue May 11, 2023 · 1 comment
Labels
H2.0/General General Harvesting 2.0 Issues

Comments

@btylerburton
Copy link
Contributor

User Story

In order to achieve stability in the Harvesting 2.0 pipeline, data.gov wants to implement a Redis queue system.

Acceptance Criteria

[ACs should be clearly demoable/verifiable whenever possible. Try specifying them using BDD.]

  • GIVEN a user/process wishes to initiate a harvesting job
    THEN the job processing will be managed by a Redis queue

Background

[Any helpful contextual notes or links to artifacts/evidence, if needed]

The Redis queue was defined here: #4314

Security Considerations (required)

[Any security concerns that might be implicated in the change. "None" is OK, just be explicit here!]

None

Sketch

  • Implement pre-defined Redis queue in ETL pipeline
@btylerburton btylerburton added H2.0/General General Harvesting 2.0 Issues H2.0/controller labels May 11, 2023
@btylerburton btylerburton changed the title Implement redis queue system Implement Redis queue system May 11, 2023
@hkdctol
Copy link
Contributor

hkdctol commented May 11, 2023

TO DO: this might be two tickets, one for job queue and one for the record queue.

@nickumia-reisys nickumia-reisys changed the title Implement Redis queue system WIP: Implement Redis queue system May 26, 2023
@btylerburton btylerburton removed the H2.0/General General Harvesting 2.0 Issues label Dec 13, 2023
@btylerburton btylerburton added H2.0/General General Harvesting 2.0 Issues and removed dep_H2.0/Airflow labels May 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
H2.0/General General Harvesting 2.0 Issues
Projects
Status: 🗄 Closed
Development

No branches or pull requests

2 participants