Training ML Model and pushing it back to repository

[amorenocb]

Hey! I’m currently facing an issue I can’t seem to get around. I have a Github action which install the associated repository and the required dependencies, pull some training data from other sources and then train the repositorys ML model. After training, the model files are overwritten and my intention is to push the new model back to the repo.

If I train the model for less than 100 epochs (less than 1 hours) then the action works fine, it trains the model and then pushes the newly trained model data back to the repository. If I train for more than 100 epochs then when I reach the push step it says :

Push to branch automatic-retraining
remote: Invalid username or password.
fatal: Authentication failed for 'https://github.com/thenotcompany/formula-generator-model.git/'

I find it really weird, same code, same model, just different amount of epochs (and thus runtime) and I get this error. Any ideas?

Thanks in advance!

[logankilpatrick]

Hey @amorenocb, you are not hitting the job timeout but the VM you are running on must forget the git credentials after an hour. Can you provide the yml file you are using for the workflow (making sure to omit any sensitive info)?

[amorenocb]

Hey @logankilpatrick! Thanks for you reply, I have the same feeling but haven’t been able to find any reference to “Github runner forgeting credentinals”. Below is my yaml file. 

name: Model re-training
on:
schedule:
# Train once a week on Mondays at 0:00
- cron: '0 00 * * 1'
[push]
jobs:

re-train:

name: Re-training model

runs-on: ubuntu-latest
steps:

# Setup SSH to clone private dependencies.
- uses: webfactory/[email protected]
  with:
    ssh-private-key: ${{ secrets.A_PRIVATE_KEY }}

- name: Checkout
  uses: actions/checkout@v2

- name: Checkout a database repository
  uses: actions/checkout@v2
  with:
    repository: ---
    token: ${{ secrets.A_PAT }}
    path: ---

- name: Checkout another database repository
  uses: actions/checkout@v2
  with:
    repository: ---
    token: ${{ secrets.A_PAT }}
    path: ---

- name: Setup Python
  uses: actions/setup-python@v1
  with:
    python-version: '3.6' # Version range or exact version of a Python version to use, using SemVer's version range syntax

- name: Install DVC
  run: |
    pip install 'dvc[gs]==0.86.5'

- name: Setup DVC credentials
  id: dvc_setup
  uses: timheuer/[email protected]
  with:
    fileName: '---'
    encodedString: ${{ secrets.SOME_CREDENTIALS }}

- name: Get training data
  env:
    [...]
  run: |
    [...]

# Install formula generator and run re-training command line utility.
- name: Re-train
  run: |
    pip install --upgrade setuptools
    pip install -e .
    train-model [...]

- name: Commit newly trained model
  run: |
    git config --local user.email "[email protected]"
    git config --local user.name "Github Action"
    git commit -m "Model retraining" -a
- name: Push newly trained model
  uses: ad-m/[email protected]
  with:
    branch: automatic-retraining
    github_token: ${{ secrets.A_PAT }}

[amorenocb]

Hey @logankilpatrick thanks for you reply! I have the same feeling, somehow the GH runner is forgetting the credentials after a while. Ill attach a the yml file below.

name: Model re-training
on:

[push]
jobs:

re-train:

name: Re-training model

runs-on: ubuntu-latest
steps:

# Setup SSH to clone private dependencies.
- uses: webfactory/[email protected]
[...]
- name: Checkout
  uses: actions/checkout@v2

- name: Checkout db repository
  uses: actions/checkout@v2
  with:
    repository: [...]
    token: [...]
    path: [...]

- name: Checkout another db
  uses: actions/checkout@v2
  with:
    repository: [...]
    token: [...]
    path: [...]

- name: Setup Python
  uses: actions/setup-python@v1
  with:
    python-version: '3.6'

- name: Install DVC
  run: |
    pip install 'dvc[gs]==0.86.5'

- name: Setup DVC credentials
  id: dvc_setup
  uses: timheuer/[email protected]
  with:
    fileName: [...]
    encodedString: [...]

- name: Get training data
  env:
    CREDENTIALS: [...]
  run: |
    [...]

# Install repository model and run re-training command line utility.
- name: Re-train
  run: |
    pip install --upgrade setuptools
    pip install -e .
    train-model --epochs 120
- name: Commit newly trained model
  run: |
    git config --local user.email "[email protected]"
    git config --local user.name "Github Action"
    git commit -m "Model retraining" -a
- name: Push newly trained model
  uses: ad-m/[email protected]
  with:
    branch: automatic-retraining
    github_token: [...]

Thanks again!

[weide-zhou]

Hi @amorenocb ,

This is by designed, the access token expires after 60 minutes. GitHub fetches a token for each job, before the job begins. When a workflow run or its jobs are queued for more than one hour, the token may expire before the job starts. You can find more details here and here. 

Thanks.

[amorenocb]

Hi @weide-zhou! Is there any way to get around this? Maybe using a personal access token to pull and then push?

[logankilpatrick]

If there is a way to get around this, it’s using your PAT! I would give it a shot. 

[weide-zhou]

Hi @amorenocb ,

Check on my side, i used PAT , and add parameter ‘persist-credentials: false’ to actions/checkout@v2 to successfully complete the push. Code sample as below:

name: tokenexpire
on: [push]
jobs:

job1:

runs-on: [ubuntu-latest]

steps:

- name: checkout 2nd

uses: actions/checkout@v2

with:

repository: weide-zhou/ticket19

token: ${{ secrets.PAT }}

path: ticket19

persist-credentials: false

- name: checkout 1st

uses: actions/checkout@v2

with:

persist-credentials: false

clean: false

- name: touch file

run: |

touch filetest1.txt

- name: sleep 1hour

run: |

sleep 1h

- name: Commit newly trained model

run: |

git config --local user.email "[email protected]"

git config --local user.name "Github Action"

git add .

git commit -m "Model retraining" -a

- name: Push newly trained model

uses: ad-m/[email protected]

with:

branch: B66

github_token: ${{ secrets.PAT }}

[amorenocb]

Hey! I managed to get it working by manually doing the git pull/push instructions using the PAT. I’ll post it here if its of any use for someone in the future :slight_smile:

name: Model re-training
on:
schedule:
# Train once a week on Mondays at 0:00
- cron: '0 00 * * 5'
push
jobs:

re-train:

name: Re-training model

runs-on: ubuntu-latest
steps:

# Setup SSH to clone private dependencies.
- uses: webfactory/[email protected]
  with:
    ssh-private-key: ${{ secrets.SSH_PRIVATE_KEY }}

# Get some parsed info of the repository
- name: Get repository metadata
  id: r_m
  uses: amorenocb/[email protected]

- name: Checkout repository
  env:
    GIT_CLONE: ${{ format('{0} --depth 1 https://{1}@github.com/{2}.git', steps.r_m.outputs.branch, secrets.PAT, github.repository) }}
  run: |
    git clone --single-branch --branch $GIT_CLONE

# Install model and run re-training command line utility.
- name: Re-train
  env:
    REPO_NAME: ${{ format('{0}',steps.r_m.outputs.name) }}
  run: |
    pip install --upgrade setuptools
    cd $GITHUB_WORKSPACE/$REPO_NAME
    pip install -e .
    train-model

- name: Commit newly trained model
  env:
    REPO_NAME: ${{ format('{0}',steps.r_m.outputs.name) }}
  run: |
    cd $GITHUB_WORKSPACE/$REPO_NAME
    git config --local user.email "[email protected]"
    git config --local user.name "Trainer"
    git commit -m "Model retraining" -a
    git push --force https://${{ secrets.PAT }}@github.com/${{ github.repository }}.git

[amorenocb]

Hey @weide-zhou it works! Although I noticed that now that I don’t use the github token after I push the trained model the action triggers itself again and again. Is there a way to specify a parameter with the push command so it does not trigger the action again? 

It’s not going to be a problem anyway, because the trigger of this action will be a schedule, not a push. But anyways it would be nice to know. 

Thank you for your response!

[weide-zhou]

Hi @amorenocb ,

Glad to know it works for you!

Is there a way to specify a parameter with the push command so it does not trigger the action again? 

>> Sorry, No, only Github token can aviod infinite loop for push event, you can find the answer in this link. using PAT to commit will always trigger the event, using schedule event instead is a good option.

Could you please help to mark the answer? It will help other guys who encounter the same problem.

Thanks.

[roel1975]

I think so, I read about it, there are a lot of possibilities
@roel1975