Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

953 reduce notification frequency to weekly #957

Merged
merged 9 commits into from
Jun 10, 2024

Conversation

gabrielwol
Copy link
Collaborator

@gabrielwol gabrielwol commented May 21, 2024

What this pull request accomplishes:

  • changes two daily "_check" DAGs to weekly.
    • adjust the schedule interval, external task sensors, and the sql params
  • removed a non critical check_gaps check for miovision gaps > 4 hours, < 1 day

Issue(s) this solves:

What, in particular, needs to reviewed:

  • My method of linking weekly dag to daily: set task sensor to look 1 week ahead of schedule interval.

What needs to be done by a sysadmin after this PR is merged

  • nothing

@gabrielwol gabrielwol self-assigned this May 21, 2024
@gabrielwol gabrielwol linked an issue May 21, 2024 that may be closed by this pull request
@gabrielwol gabrielwol requested a review from chmnata May 21, 2024 19:08
@gabrielwol gabrielwol marked this pull request as ready for review May 21, 2024 19:08
@chmnata
Copy link
Collaborator

chmnata commented May 22, 2024

wanted to make sure I understand the scheduling date logic,
ecocounter-check : run once a week on Monday, checks for past 60 days data
ecocounter-pull: runs daily, pulls daily data

ecocounter-check should be able to trigger after the same date's ecocounter pull (ecocounter pull's execution date + 1 hour), since we are checking for the past 60 days. Why do we need to look ahead a week for ecocounter pull DAG?

@gabrielwol
Copy link
Collaborator Author

Since the dag runs at the end of the schedule interval (in this case a week later), I want it to run for the week ahead of it's execution date.
eg. Run 2024-05-10 runs on 2024-05-17 and should look back 60 days from 2024-05-17. It should also only run after the daily DAG scheduled for 2024-05-17 completes.
It is a lot more frustrating to test weekly DAGs than daily 😠

Copy link
Collaborator

@chmnata chmnata left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Its working! Approved :gabe_approved:

@gabrielwol gabrielwol merged commit 45ce7f7 into master Jun 10, 2024
5 checks passed
@gabrielwol gabrielwol deleted the 953-reduce-notification-frequency-to-weekly branch June 10, 2024 15:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Reduce notification frequency to weekly
2 participants