Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Get docker image running in github action #655

Closed
stevie- opened this issue Apr 9, 2024 · 9 comments · Fixed by #656
Closed

Get docker image running in github action #655

stevie- opened this issue Apr 9, 2024 · 9 comments · Fixed by #656
Labels
area/docker documentation Improvements or additions to documentation feature New feature or request

Comments

@stevie-
Copy link
Contributor

stevie- commented Apr 9, 2024

Describe the bug

We are trying to run pre-commit/action within github action.
To make sure dependencies are installed we use the managed docker image in the github action.
Despite other pre-commit hooks, pre-commit-terraform doesn't take care of tool installation (probably due to multiple tool approach)

But we get errors runing pre-commit inside this image.

Snippet from the github action logs

Run pre-commit run --show-diff-on-failure --color=always --all-files
An error has occurred: FatalError: git failed. Is it installed, and are you in a Git repository directory?
Check the log at /github/home/.cache/pre-commit/pre-commit.log
Error: Process completed with exit code 1.

Why don't you use the docker image within your gha workflow? https://github.com/antonbabenko/pre-commit-terraform/blob/master/.github/workflows/pre-commit.yaml

How can we reproduce it?

our workflow looks like this.
We added the uid:gid to make sure permissions are aligned with github actions, but this makes no difference.

name: pre-commit

on:
  pull_request:
  push:
    branches: [main]

jobs:
  get-gha-parameters:
    runs-on: ubuntu-latest
    outputs:
      uid: ${{ steps.get.outputs.uid }}
      gid: ${{ steps.get.outputs.gid }}
    steps:
    - name: Get UID and GID
      id: get
      run: |
        echo "uid=$(id -u) gid=$(id -g)"
        echo "uid=$(id -u)" >> $GITHUB_OUTPUT
        echo "gid=$(id -g)" >> $GITHUB_OUTPUT
  pre-commit:
    runs-on: ubuntu-latest
    needs: get-gha-parameters
    container:
      image: ghcr.io/antonbabenko/pre-commit-terraform:latest
      env:
        USERID: ${{ needs.get-gha-parameters.outputs.uid }}:${{ needs.get-gha-parameters.outputs.gid }}
    steps:
    - uses: actions/checkout@v4
    - uses: actions/setup-python@v5
    - uses: pre-commit/[email protected]

Environment information

  • OS:

Current runner version: '2.315.0'

  • docker info:
command output
 /usr/bin/docker version --format '{{.Server.APIVersion}}'
  '1.43'
  Docker daemon API version: '1.43'
  /usr/bin/docker version --format '{{.Client.APIVersion}}'
  '1.43'
  Docker client API version: '1.43'
  • Docker image tag/git commit:

    Digest: sha256:802440d81ee1409184dbc91425319814c42495e04f41e95f9f47c18724533284
    ghcr.io/antonbabenko/pre-commit-terraform:latest

  • Tools versions. Don't forget to specify right tag in command -
    TAG=latest && docker run --entrypoint cat pre-commit:$TAG /usr/bin/tools_versions_info

check managed image contents
  • .pre-commit-config.yaml:
file content
repos:
  - repo: https://github.com/antonbabenko/pre-commit-terraform
    rev: v1.77.3
    hooks:
      - id: terragrunt_fmt
      - id: terraform_fmt
      - id: terraform_tflint
        args:
          - --args=--config=__GIT_WORKING_DIR__/.tflint.hcl
  - repo: https://github.com/pre-commit/pre-commit-hooks
    rev: v4.4.0
    hooks:
      - id: end-of-file-fixer
      - id: trailing-whitespace
  - repo: https://github.com/commitizen-tools/commitizen
    rev: v3.1.0
    hooks:
      - id: commitizen
        stages:
          - commit-msg
@stevie- stevie- added area/docker bug Something isn't working labels Apr 9, 2024
@yermulnik
Copy link
Collaborator

Are you sure that it's the pre-commit-terraform container what produces the error?
From what I see, the git gets installed into pre-commit-terraform container image: https://github.com/antonbabenko/pre-commit-terraform/blob/master/Dockerfile#L219-L221 (the pre-commit is also installed into the image)
And the Run pre-commit run --show-diff-on-failure --color=always --all-files is from pre-commit itself: https://github.com/pre-commit/action/blob/main/action.yml#L19

Would you please try and run pre-commit-terraform without pre-commit/action step and see whether the error persists?

Probably related: actions/runner#3000

@MaxymVlasov
Copy link
Collaborator

Why don't you use the docker image within your gha workflow? https://github.com/antonbabenko/pre-commit-terraform/blob/master/.github/workflows/pre-commit.yaml

Because to check bash hooks you need different hooks and deps than we provide in this repo.
Also, pre-commit GHA has existed for much longer than the Docker image was introduced.

@MaxymVlasov
Copy link
Collaborator

btw, if that will help you, that's what I use as GHA in my infra
name: Common issues check

on: [pull_request]

env:
  # Prevent GH API rate-limit issue
  GITHUB_TOKEN: ${{secrets.GITHUB_TOKEN}}

jobs:
  pre-commit:
    runs-on: [self-hosted]
    container: python:3.12-slim@sha256:36d57d7f9948fefe7b6092cfe8567da368033e71ba281b11bb9eeffce3d45bc6
    steps:
    - name: Install container pre-requirements
      run: |
        apt update
        apt install -y \
            git \
            curl \
            unzip \
            jq \
            shellcheck \
            nodejs # Needed for Terraform installation
        curl -L https://github.com/mikefarah/yq/releases/latest/download/yq_linux_amd64 > /usr/bin/yq &&\
          chmod +x /usr/bin/yq
    - name: Checkout
      uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4.1.1
      with:
        ref: ${{ github.base_ref }}

    - uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4.1.1
    - run: |
        git config --global --add safe.directory /__w/infrastructure/infrastructure
        git fetch --no-tags --prune --depth=1 origin +refs/heads/*:refs/remotes/origin/*

    - name: Get changed files
      id: file_changes
      run: |
        export DIFF=$(git diff --name-only origin/${{ github.base_ref }} ${{ github.sha }})
        echo "Diff between ${{ github.base_ref }} and ${{ github.sha }}"
        echo "files=$( echo "$DIFF" | xargs echo )" >> $GITHUB_OUTPUT

    - name: TFLint cache plugin dir
      uses: actions/cache@ab5e6d0c87105b4c9c2047343972218f562e4319 # v4.0.1
      with:
        path: ~/.tflint.d/plugins
        key: ubuntu-latest-tflint-${{ hashFiles('.tflint.hcl') }}

    - name: Setup TFLint
      uses: terraform-linters/setup-tflint@19a52fbac37dacb22a09518e4ef6ee234f2d4987 # v4.0.0
      with:
        github_token: "${{ secrets.GITHUB_TOKEN }}"

    - name: Init TFLint
      run: tflint --init

    - name: Setup Terraform docs
      env:
        # renovate: datasource=github-releases depName=terraform-docs lookupName=terraform-docs/terraform-docs
        TERRAFORM_DOCS_VERSION: '0.17.0'
      run: |
        curl -L https://github.com/terraform-docs/terraform-docs/releases/download/v${TERRAFORM_DOCS_VERSION}/terraform-docs-v${TERRAFORM_DOCS_VERSION}-linux-amd64.tar.gz > terraform-docs.tgz \
          && tar -xzf terraform-docs.tgz terraform-docs && rm terraform-docs.tgz \
          && chmod +x terraform-docs && mv terraform-docs /usr/bin/

    - name: Setup tfupdate
      env:
        # renovate: datasource=github-releases depName=tfupdate lookupName=minamijoyo/tfupdate
        TFUPDATE_VERSION: '0.8.1'
      run: |
        curl -L https://github.com/minamijoyo/tfupdate/releases/download/v${TFUPDATE_VERSION}/tfupdate_${TFUPDATE_VERSION}_linux_amd64.tar.gz > tfupdate.tgz \
          && tar -xzf tfupdate.tgz tfupdate && rm tfupdate.tgz \
          && chmod +x tfupdate && mv tfupdate /usr/bin/

    - name: Install shfmt
      env:
        # renovate: datasource=github-releases depName=shfmt lookupName=mvdan/sh
        SHFMT_VERSION: '3.8.0'
      run: |
        curl -L https://github.com/mvdan/sh/releases/download/v${SHFMT_VERSION}/shfmt_v${SHFMT_VERSION}_linux_amd64 > shfmt \
        && chmod +x shfmt && mv shfmt /usr/bin/

    - uses: hashicorp/setup-terraform@a1502cd9e758c50496cc9ac5308c4843bcd56d36 # v3.0.0
      with:
        terraform_version: 1.5.4

    - name: Setup OPA
      uses: open-policy-agent/setup-opa@34a30e8a924d1b03ce2cf7abe97250bbb1f332b5 # v2.2.0
      with:
        version: latest

    # Need to success pre-commit fix push
    - uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4.1.1
      with:
        fetch-depth: 0
        ref: ${{ github.event.pull_request.head.ref }}
        # Need to trigger pre-commit workflow on autofix commit
        # Guide: https://web.archive.org/web/20210731173012/https://github.community/t/required-check-is-expected-after-automated-push/187545/
        ssh-key: "${{ secrets.GHA_AUTOFIX_COMMIT_KEY }}"

    - name: Execute pre-commit
      uses: pre-commit/action@2c7b3805fd2a0fd8c1884dcaebf91fc102a13ecd # v3.0.1
      timeout-minutes: 30
      env:
        SKIP: no-commit-to-branch,manual-apply-warning
      with:
        extra_args: --color=always --show-diff-on-failure --files ${{ steps.file_changes.outputs.files }}

      # Need to trigger pre-commit workflow on autofix commit.
    - name: Push fixes
      if: failure()
      uses: EndBug/add-and-commit@a94899bca583c204427a224a7af87c02f9b325d5 # v9.1.4
      with:
        # Determines the way the action fills missing author name and email. Three options are available:
        # - github_actor -> UserName <[email protected]>
        # - user_info -> Your Display Name <[email protected]>
        # - github_actions -> github-actions <email associated with the github logo>
        # Default: github_actor
        default_author: github_actor
        # The message for the commit.
        # Default: 'Commit from GitHub Actions (name of the workflow)'
        message: '[pre-commit] Autofix violations'

@stevie-
Copy link
Contributor Author

stevie- commented Apr 10, 2024

btw, if that will help you, that's what I use as GHA in my infra

I've seen your approach, but these bloats up the gha action for every tool, which is not self-installed via pre-commit. (Which should be solved via the docker image)

@stevie-
Copy link
Contributor Author

stevie- commented Apr 10, 2024

Are you sure that it's the pre-commit-terraform container what produces the error? From what I see, the git gets installed into pre-commit-terraform container image: https://github.com/antonbabenko/pre-commit-terraform/blob/master/Dockerfile#L219-L221 (the pre-commit is also installed into the image) And the Run pre-commit run --show-diff-on-failure --color=always --all-files is from pre-commit itself: https://github.com/pre-commit/action/blob/main/action.yml#L19

Would you please try and run pre-commit-terraform without pre-commit/action step and see whether the error persists?

Probably related: actions/runner#3000

I've changed the workflow to call pre-commit after checkout, but still get this error. Which maybe is related to the error in the github action runner image and git binary.

# snippet
 pre-commit:
    runs-on: ubuntu-latest
    needs: get-gha-parameters
    container:
      image: ghcr.io/antonbabenko/pre-commit-terraform:latest
      env:
        USERID: ${{ needs.get-gha-parameters.outputs.uid }}:${{ needs.get-gha-parameters.outputs.gid }}
    steps:
    - uses: actions/checkout@v4
    - run: |
        pre-commit --version
        pre-commit run -a

Error:

pre-commit 3.7.0
An error has occurred: FatalError: git failed. Is it installed, and are you in a Git repository directory?
Check the log at /github/home/.cache/pre-commit/pre-commit.log
Error: Process completed with exit code 1.

Will try to check if git binary is available.

@stevie-
Copy link
Contributor Author

stevie- commented Apr 10, 2024

Workflow step added with these debug steps:

  pre-commit --version
  which git
  git --version
  echo $PATH
  pre-commit run -a || true
  cat $HOME/.cache/pre-commit/pre-commit.log

result:

pre-commit 3.7.0
/usr/bin/git
git version 2.38.5
/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
An error has occurred: FatalError: git failed. Is it installed, and are you in a Git repository directory?
Check the log at /github/home/.cache/pre-commit/pre-commit.log
### version information
``
pre-commit version: 3.7.0
git --version: git version 2.38.5
sys.version:
    3.12.0 (main, Dec  1 2023, 01:43:25) [GCC 12.2.1 20220924]
sys.executable: /usr/local/bin/python3
os.name: posix
sys.platform: linux
``
### error information
``
An error has occurred: FatalError: git failed. Is it installed, and are you in a Git repository directory?
``
``
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/pre_commit/git.py", line 58, in get_root
    cmd_output('git', 'rev-parse', '--show-cdup')[1].strip(),
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/pre_commit/util.py", line 117, in cmd_output
    returncode, stdout_b, stderr_b = cmd_output_b(*cmd, **kwargs)
                                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/pre_commit/util.py", line 111, in cmd_output_b
    raise CalledProcessError(returncode, cmd, stdout_b, stderr_b)
pre_commit.util.CalledProcessError: command: ('/usr/bin/git', 'rev-parse', '--show-cdup')
return code: 128
stdout: (none)
stderr:
    fatal: detected dubious ownership in repository at '/__w/dp-cloud-infra/dp-cloud-infra'
    To add an exception for this directory, call:
    
    	git config --global --add safe.directory /__w/dp-cloud-infra/dp-cloud-infra
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/pre_commit/error_handler.py", line 73, in error_handler
    yield
  File "/usr/local/lib/python3.12/site-packages/pre_commit/main.py", line 373, in main
    _adjust_args_and_chdir(args)
  File "/usr/local/lib/python3.12/site-packages/pre_commit/main.py", line 183, in _adjust_args_and_chdir
    toplevel = git.get_root()
               ^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/pre_commit/git.py", line 64, in get_root
    raise FatalError(
pre_commit.errors.FatalError: git failed. Is it installed, and are you in a Git repository directory?
``

@stevie-
Copy link
Contributor Author

stevie- commented Apr 10, 2024

the fix is to run this, before running pre-commit

git config --global --add safe.directory $GITHUB_WORKSPACE

final workflow ( basically rebuilding pre-commit action without input parameter )

name: pre-commit

on:
  pull_request:
  push:
    branches: [main]

jobs:
  pre-commit:
    runs-on: ubuntu-latest
    container:
      image: ghcr.io/antonbabenko/pre-commit-terraform:v1.88.4
    steps:
    - uses: actions/checkout@v4
    - name: fix tar dependency
      shell: bash
      run: |
        apk --no-cache add tar
        # check python modules installed versions
        python -m pip freeze --local
    - uses: actions/cache@v4
      with:
        path: ~/.cache/pre-commit
        key: pre-commit-3|${{ hashFiles('.pre-commit-config.yaml') }}
    - shell: bash
      run: |
        pre-commit --version
        git config --global --add safe.directory $GITHUB_WORKSPACE
        pre-commit run --show-diff-on-failure --color=always || cat $HOME/.cache/pre-commit/pre-commit.log

Suggestion:
To add an example for other and increase your test coverage you could also add a workflow like this in this repo?

@yermulnik
Copy link
Collaborator

Suggestion:
To add an example for other and increase your test coverage you could also add a workflow like this in this repo?

Good point. Would you be up to contribute via PR please?

@MaxymVlasov
Copy link
Collaborator

MaxymVlasov commented Apr 12, 2024

Yeah, safe.directory was introduced by git a few years ago, and caused some distractions across all CIs, including GHA.

Also, pre-commit action in 2.0.3 (which is used in this repo to avoid providing secrets as in #655 (comment)) can commit and push fixes to branch.

It's not very useful just to get information that something is wrong, especially if it can be fixed automatically.

@MaxymVlasov MaxymVlasov added question 3rd party Issues not related to `pre-commit-terraform` and removed bug Something isn't working area/docker labels Apr 12, 2024
@MaxymVlasov MaxymVlasov changed the title Get docker image running in github action [git safe.directory check in CI] Get docker image running in github action Apr 12, 2024
@MaxymVlasov MaxymVlasov changed the title [git safe.directory check in CI] Get docker image running in github action Get docker image running in github action Apr 12, 2024
@MaxymVlasov MaxymVlasov added feature New feature or request documentation Improvements or additions to documentation area/docker and removed question 3rd party Issues not related to `pre-commit-terraform` labels Apr 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/docker documentation Improvements or additions to documentation feature New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants