Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feature: scrub logs #1802

Merged
merged 33 commits into from
May 8, 2024
Merged

feature: scrub logs #1802

merged 33 commits into from
May 8, 2024

Commits on Apr 16, 2024

  1. feat: add delete to s3 wrapper

    @mSounak @saikonen
    mSounak authored and saikonen committed Apr 16, 2024
    Configuration menu
    Copy the full SHA
    afca3e2 View commit details
    Browse the repository at this point in the history

  2. datastore changes to support deleting files

    @saikonen
    saikonen committed Apr 16, 2024
    Configuration menu
    Copy the full SHA
    5de92ef View commit details
    Browse the repository at this point in the history

  3. implement delete logs

    @saikonen
    saikonen committed Apr 16, 2024
    Configuration menu
    Copy the full SHA
    0888d35 View commit details
    Browse the repository at this point in the history

  4. add delete mode to datastore which inits _attempt

    @saikonen
    saikonen committed Apr 16, 2024
    Configuration menu
    Copy the full SHA
    a3497fc View commit details
    Browse the repository at this point in the history

Commits on Apr 19, 2024

  1. cleanup debug print

    @saikonen
    saikonen committed Apr 19, 2024
    Configuration menu
    Copy the full SHA
    2543d07 View commit details
    Browse the repository at this point in the history

  2. change logs CLI command to be a cmd group, with a default cmd for bac… 

    …kwards comliance.
    @saikonen
    saikonen committed Apr 19, 2024
    Configuration menu
    Copy the full SHA
    9c8a3e8 View commit details
    Browse the repository at this point in the history

Commits on Apr 22, 2024

  1. fix logger handling --quiet

    @saikonen
    saikonen committed Apr 22, 2024
    Configuration menu
    Copy the full SHA
    c08e6ee View commit details
    Browse the repository at this point in the history

Commits on Apr 23, 2024

  1. fix args parsing for custom click group

    @saikonen
    saikonen committed Apr 23, 2024
    Configuration menu
    Copy the full SHA
    54c6a7d View commit details
    Browse the repository at this point in the history

  2. another args fix

    @saikonen
    saikonen committed Apr 23, 2024
    Configuration menu
    Copy the full SHA
    8dbdb32 View commit details
    Browse the repository at this point in the history

Commits on Apr 24, 2024

  1. reword delete to scrub. Only delete log files that exist, so as to no… 

    …t create non-existent log files.
    @saikonen
    saikonen committed Apr 24, 2024
    Configuration menu
    Copy the full SHA
    9b20c01 View commit details
    Browse the repository at this point in the history

  2. add completion msg to log scrubbing

    @saikonen
    saikonen committed Apr 24, 2024
    Configuration menu
    Copy the full SHA
    7301df8 View commit details
    Browse the repository at this point in the history

Commits on Apr 25, 2024

  1. fix help msg wording

    @saikonen
    saikonen committed Apr 25, 2024
    Configuration menu
    Copy the full SHA
    e02135d View commit details
    Browse the repository at this point in the history

  2. overwrite logs instead of deleting them

    @saikonen
    saikonen committed Apr 25, 2024
    Configuration menu
    Copy the full SHA
    11c761f View commit details
    Browse the repository at this point in the history

  3. cleanup unnecessary delete functionality from datastore.

    @saikonen
    saikonen committed Apr 25, 2024
    Configuration menu
    Copy the full SHA
    94a8220 View commit details
    Browse the repository at this point in the history

  4. make it possible to specify a task attempt number to scrub logs for.

    @saikonen
    saikonen committed Apr 25, 2024
    Configuration menu
    Copy the full SHA
    a51f222 View commit details
    Browse the repository at this point in the history

  5. allow specifying --attempt for show logs as well

    @saikonen
    saikonen committed Apr 25, 2024
    Configuration menu
    Copy the full SHA
    77fe766 View commit details
    Browse the repository at this point in the history

  6. enable scrubbing of logs without task_id in order to target multiple … 

    …tasks at once.
    @saikonen
    saikonen committed Apr 25, 2024
    Configuration menu
    Copy the full SHA
    40dd2c7 View commit details
    Browse the repository at this point in the history

  7. handle attempts in scrubbing better.

    @saikonen
    saikonen committed Apr 25, 2024
    Configuration menu
    Copy the full SHA
    c21f2a7 View commit details
    Browse the repository at this point in the history

Commits on Apr 29, 2024

  1. Merge branch 'master' into feature/delete-logs

    @saikonen
    saikonen committed Apr 29, 2024
    Configuration menu
    Copy the full SHA
    1ae1262 View commit details
    Browse the repository at this point in the history

Commits on Apr 30, 2024

  1. address logs_cli nits.

    @saikonen
    saikonen committed Apr 30, 2024
    Configuration menu
    Copy the full SHA
    5dccca0 View commit details
    Browse the repository at this point in the history

  2. make log scrubbing more robust, do not halt on individual errors, and… 

    … output better scrubbing failure messages
    @saikonen
    saikonen committed Apr 30, 2024
    Configuration menu
    Copy the full SHA
    7da124b View commit details
    Browse the repository at this point in the history

  3. fix type hint in flowdatastore

    @saikonen
    saikonen committed Apr 30, 2024
    Configuration menu
    Copy the full SHA
    f6dbc1f View commit details
    Browse the repository at this point in the history

  4. record metadata after successful scrubbing

    @saikonen
    saikonen committed Apr 30, 2024
    Configuration menu
    Copy the full SHA
    b541e4f View commit details
    Browse the repository at this point in the history

  5. Revert "record metadata after successful scrubbing" 

    This reverts commit b541e4f.
    @saikonen
    saikonen committed Apr 30, 2024
    Configuration menu
    Copy the full SHA
    9828ab2 View commit details
    Browse the repository at this point in the history

Commits on May 1, 2024

  1. add flow datastore method for fetching taskdatastores for all attempt… 

    …s of the tasks
    @saikonen
    saikonen committed May 1, 2024
    Configuration menu
    Copy the full SHA
    913088e View commit details
    Browse the repository at this point in the history

  2. add support for --all flag when scrubbing logs of all task attempts

    @saikonen
    saikonen committed May 1, 2024
    Configuration menu
    Copy the full SHA
    cddd188 View commit details
    Browse the repository at this point in the history

  3. add support for specifying to include not-done task attempts in scrub… 

    …bing.
    @saikonen
    saikonen committed May 1, 2024
    Configuration menu
    Copy the full SHA
    a05d4cd View commit details
    Browse the repository at this point in the history

Commits on May 3, 2024

  1. refactor: remove new flow_datastore method and combine functionality … 

    …with existing one.
    @saikonen
    saikonen committed May 3, 2024
    Configuration menu
    Copy the full SHA
    e9c988b View commit details
    Browse the repository at this point in the history

Commits on May 7, 2024

  1. reword get_latest_task_datastores docstring

    @saikonen
    saikonen committed May 7, 2024
    Configuration menu
    Copy the full SHA
    fe5bc32 View commit details
    Browse the repository at this point in the history

  2. rename to get_task_datastores

    @saikonen
    saikonen committed May 7, 2024
    Configuration menu
    Copy the full SHA
    5e6245a View commit details
    Browse the repository at this point in the history

  3. limit attempt checks by MAX_ATTEMPTS

    @saikonen
    saikonen committed May 7, 2024
    Configuration menu
    Copy the full SHA
    d1edc5b View commit details
    Browse the repository at this point in the history

  4. fix attempt iteration

    @saikonen
    saikonen committed May 7, 2024
    Configuration menu
    Copy the full SHA
    470fbed View commit details
    Browse the repository at this point in the history

  5. fix attempt_range logic

    @saikonen
    saikonen committed May 7, 2024
    Configuration menu
    Copy the full SHA
    6e5dcd0 View commit details
    Browse the repository at this point in the history