Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GHA Cache Not Caching for Pre-Stages (multi-stage builds) #829

Open
sbaynham-gridwise opened this issue Mar 3, 2023 · 11 comments
Open

GHA Cache Not Caching for Pre-Stages (multi-stage builds) #829

sbaynham-gridwise opened this issue Mar 3, 2023 · 11 comments

Comments

@sbaynham-gridwise
Copy link

sbaynham-gridwise commented Mar 3, 2023

Troubleshooting

Before submitting a bug report please read the Troubleshooting doc.

Behaviour

Steps to reproduce this issue

  1. Use following Dockerfile:
FROM golang:1.19-buster as build-env

ARG SECRETS_PATH="build/package/secrets"

COPY ${SECRETS_PATH} /secrets

WORKDIR /build

## Snip... set up private go repo

COPY go.* ./
RUN go mod download

COPY . .
RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -o /tmp/insights cmd/insights/main.go

FROM scratch

COPY --from=build-env /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/ca-certificates.crt
COPY --from=build-env /tmp/insights ./insights

EXPOSE 8080
ENTRYPOINT ["/insights"]

  1. Build in GHA once to write the cache
  2. Build again to read the cache

Expected behaviour

All operations should be cached, since nothing in the build has changed since the previous run

Actual behaviour

The final image's operations are cached, but not the pre-stage where most of the work is

Configuration

  • Repository URL (if public): not public
  • Build URL (if public): not public
name: Docker
on:
  push:
    branches:
      - master
    tags:
      - v*
  pull_request:

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
        name: Checkout code
      - name: Generate Docker Metadata
        id: meta
        uses: docker/metadata-action@v4
        with:
          images: us.gcr.io/${{ secrets.GCP_PROJECT_ID }}/insights-service
      - name: Setup Secrets
        run: |
          # ... snip ...
      - id: auth
        name: Authenticate to Google Cloud
        uses: google-github-actions/auth@v1
        with:
          project_id: ${{ secrets.GCP_PROJECT_ID }}
          credentials_json: ${{ secrets.GCP_SA_KEY }}
          export_environment_variables: true
          token_format: access_token
      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v2
      - name: Login to Registry
        uses: docker/login-action@v2
        with:
          registry: us.gcr.io
          username: oauth2accesstoken
          password: ${{ steps.auth.outputs.access_token }}
      - name: Build And Push Docker Image
        id: build
        uses: docker/build-push-action@v4
        with:
          push: true
          file: ./build/package/Dockerfile
          context: .
          cache-from: type=gha,scope=insights-service
          cache-to: type=gha,mode=max,scope=insights-service
          tags: ${{ steps.meta.outputs.tags }}
          labels: ${{ steps.meta.outputs.labels }}

Logs

Download the log file of your build and attach it to this issue.

@sbaynham-gridwise
Copy link
Author

log_write.zip
log_read.zip

@tonistiigi
Copy link
Member

I don't know if this is the issue but git does not create deterministic .git directory on clone(at least there is no guarantee). As you are doing COPY . that copies to .git directory as well. A simple way to avoid that issue is to not define context: . but let the action build directly from the repo files (without .git directory by default).

@sbaynham-gridwise
Copy link
Author

sbaynham-gridwise commented Mar 4, 2023

That's important and useful info & I'll put together some fixes on my end for that. However, there are several steps prior to COPY . . that are also not cached so I don't think it's the root cause

EDIT: I just reread my own logs and saw that stages 1-5 are, in fact, cached, so I think you're right and this is an issue with my own setup. Thank you!

@sbaynham-gridwise
Copy link
Author

I'm sorry, but after digging into this more, I woudl like to re-open, because the logs I posted reveal an issue (if I should open a different task, let me know). In particular, check this out, from the log_read.zip build log:

2023-03-03T23:11:33.6389149Z 
2023-03-03T23:11:33.6389320Z #12 [build-env 6/8] RUN go mod download
2023-03-03T23:11:33.9025245Z #12 CACHED
2023-03-03T23:11:34.0529333Z 
2023-03-03T23:11:34.0531019Z #12 [build-env 6/8] RUN go mod download
2023-03-03T23:11:34.1923278Z #12 sha256:955bce42cb0bc6ba89d9cb7120c1816158f37ce494e1f61810b9d8dbef64aba6 0B / 130.46MB 0.2s
2023-03-03T23:11:34.1924610Z #12 sha256:5e6550d30dade172d76055b9bdeb052f877a3609ad9ce34571a5582a2c3b6884 0B / 33.68kB 0.2s
2023-03-03T23:11:34.1925620Z #12 sha256:f0ee7d822262943f0c8d7739222b554f1c1b1dbeae5545615a50adb931df121c 250B / 250B 0.2s done
2023-03-03T23:11:34.1926063Z #12 sha256:8aad0f0be38edf49e431a7da225e78749d79d4238b5e0888062af2a9bde73d80 101B / 101B 0.2s done
2023-03-03T23:11:34.1926585Z #12 sha256:f2931c2c4bf895d4385799e5199ca517e791b0120f32c78f5f838ace8f735ab1 197B / 197B 0.1s done
2023-03-03T23:11:34.3422597Z #12 sha256:5e6550d30dade172d76055b9bdeb052f877a3609ad9ce34571a5582a2c3b6884 33.68kB / 33.68kB 0.3s done
2023-03-03T23:11:34.3423510Z #12 sha256:e4ac58638d50338f5e89948ce856b91753c7063a48a322f7369d5fd3e1c2ebb3 155B / 155B 0.1s done
2023-03-03T23:11:34.4929450Z #12 sha256:c7c452053b608f32eb40cfee6ec0068b9e660f9e1ddfe4c732960e38c5ec5887 0B / 149.01MB 0.2s
2023-03-03T23:11:34.4929950Z #12 sha256:b5253fb917d27d45dda13ba51813713fac4266088086cd5721f622543cd4a676 0B / 68.85MB 0.2s
2023-03-03T23:11:34.4930360Z #12 sha256:47452733a7c0712961d7ef8b48e3d823328f056af1c90c715966748b137b3d1c 0B / 51.87MB 0.2s
2023-03-03T23:11:35.0933697Z #12 sha256:47452733a7c0712961d7ef8b48e3d823328f056af1c90c715966748b137b3d1c 4.19MB / 51.87MB 0.8s
2023-03-03T23:11:35.2427125Z #12 sha256:955bce42cb0bc6ba89d9cb7120c1816158f37ce494e1f61810b9d8dbef64aba6 8.39MB / 130.46MB 1.2s
2023-03-03T23:11:35.2429156Z #12 sha256:c7c452053b608f32eb40cfee6ec0068b9e660f9e1ddfe4c732960e38c5ec5887 11.53MB / 149.01MB 1.1s
2023-03-03T23:11:35.2431520Z #12 sha256:b5253fb917d27d45dda13ba51813713fac4266088086cd5721f622543cd4a676 6.29MB / 68.85MB 0.9s
2023-03-03T23:11:35.3933182Z #12 sha256:b5253fb917d27d45dda13ba51813713fac4266088086cd5721f622543cd4a676 10.49MB / 68.85MB 1.1s
2023-03-03T23:11:35.3937921Z #12 sha256:47452733a7c0712961d7ef8b48e3d823328f056af1c90c715966748b137b3d1c 8.39MB / 51.87MB 1.1s
2023-03-03T23:11:35.5428429Z #12 sha256:c7c452053b608f32eb40cfee6ec0068b9e660f9e1ddfe4c732960e38c5ec5887 20.97MB / 149.01MB 1.4s
2023-03-03T23:11:35.6939600Z #12 sha256:955bce42cb0bc6ba89d9cb7120c1816158f37ce494e1f61810b9d8dbef64aba6 16.78MB / 130.46MB 1.7s
2023-03-03T23:11:35.6940401Z #12 sha256:b5253fb917d27d45dda13ba51813713fac4266088086cd5721f622543cd4a676 17.83MB / 68.85MB 1.4s
2023-03-03T23:11:35.6941246Z #12 sha256:47452733a7c0712961d7ef8b48e3d823328f056af1c90c715966748b137b3d1c 12.58MB / 51.87MB 1.4s
2023-03-03T23:11:35.8449070Z #12 sha256:b5253fb917d27d45dda13ba51813713fac4266088086cd5721f622543cd4a676 22.02MB / 68.85MB 1.5s
2023-03-03T23:11:35.9927454Z #12 sha256:c7c452053b608f32eb40cfee6ec0068b9e660f9e1ddfe4c732960e38c5ec5887 37.75MB / 149.01MB 1.8s
2023-03-03T23:11:35.9928276Z #12 sha256:b5253fb917d27d45dda13ba51813713fac4266088086cd5721f622543cd4a676 26.21MB / 68.85MB 1.7s
2023-03-03T23:11:35.9929553Z #12 sha256:47452733a7c0712961d7ef8b48e3d823328f056af1c90c715966748b137b3d1c 16.78MB / 51.87MB 1.7s
2023-03-03T23:11:36.1429205Z #12 sha256:955bce42cb0bc6ba89d9cb7120c1816158f37ce494e1f61810b9d8dbef64aba6 24.12MB / 130.46MB 2.1s
2023-03-03T23:11:36.1429720Z #12 sha256:b5253fb917d27d45dda13ba51813713fac4266088086cd5721f622543cd4a676 31.46MB / 68.85MB 1.8s
2023-03-03T23:11:36.2926130Z #12 sha256:c7c452053b608f32eb40cfee6ec0068b9e660f9e1ddfe4c732960e38c5ec5887 48.23MB / 149.01MB 2.1s
2023-03-03T23:11:36.2927237Z #12 sha256:b5253fb917d27d45dda13ba51813713fac4266088086cd5721f622543cd4a676 36.70MB / 68.85MB 2.0s
2023-03-03T23:11:36.2929868Z #12 sha256:47452733a7c0712961d7ef8b48e3d823328f056af1c90c715966748b137b3d1c 20.97MB / 51.87MB 2.0s
2023-03-03T23:11:36.4434141Z #12 sha256:b5253fb917d27d45dda13ba51813713fac4266088086cd5721f622543cd4a676 40.89MB / 68.85MB 2.1s
2023-03-03T23:11:36.5929176Z #12 sha256:955bce42cb0bc6ba89d9cb7120c1816158f37ce494e1f61810b9d8dbef64aba6 32.51MB / 130.46MB 2.6s
2023-03-03T23:11:36.5929894Z #12 sha256:c7c452053b608f32eb40cfee6ec0068b9e660f9e1ddfe4c732960e38c5ec5887 58.72MB / 149.01MB 2.4s
2023-03-03T23:11:36.5930336Z #12 sha256:b5253fb917d27d45dda13ba51813713fac4266088086cd5721f622543cd4a676 47.19MB / 68.85MB 2.3s
2023-03-03T23:11:36.5930744Z #12 sha256:47452733a7c0712961d7ef8b48e3d823328f056af1c90c715966748b137b3d1c 27.26MB / 51.87MB 2.3s
2023-03-03T23:11:36.7431825Z #12 sha256:b5253fb917d27d45dda13ba51813713fac4266088086cd5721f622543cd4a676 51.38MB / 68.85MB 2.4s
2023-03-03T23:11:36.8934704Z #12 sha256:b5253fb917d27d45dda13ba51813713fac4266088086cd5721f622543cd4a676 57.67MB / 68.85MB 2.6s
2023-03-03T23:11:36.8940474Z #12 sha256:47452733a7c0712961d7ef8b48e3d823328f056af1c90c715966748b137b3d1c 32.51MB / 51.87MB 2.6s
2023-03-03T23:11:37.0450823Z #12 sha256:955bce42cb0bc6ba89d9cb7120c1816158f37ce494e1f61810b9d8dbef64aba6 40.89MB / 130.46MB 3.0s
2023-03-03T23:11:37.0455688Z #12 sha256:c7c452053b608f32eb40cfee6ec0068b9e660f9e1ddfe4c732960e38c5ec5887 71.30MB / 149.01MB 2.7s
2023-03-03T23:11:37.0456549Z #12 sha256:b5253fb917d27d45dda13ba51813713fac4266088086cd5721f622543cd4a676 61.87MB / 68.85MB 2.7s
2023-03-03T23:11:37.1970854Z #12 sha256:b5253fb917d27d45dda13ba51813713fac4266088086cd5721f622543cd4a676 68.85MB / 68.85MB 2.9s done
2023-03-03T23:11:37.1971338Z #12 sha256:47452733a7c0712961d7ef8b48e3d823328f056af1c90c715966748b137b3d1c 37.75MB / 51.87MB 2.9s
2023-03-03T23:11:37.3435938Z #12 sha256:c7c452053b608f32eb40cfee6ec0068b9e660f9e1ddfe4c732960e38c5ec5887 85.98MB / 149.01MB 3.2s
2023-03-03T23:11:37.3436417Z #12 sha256:47452733a7c0712961d7ef8b48e3d823328f056af1c90c715966748b137b3d1c 40.89MB / 51.87MB 3.0s
2023-03-03T23:11:37.3436879Z #12 sha256:4efc3ea86e67ce8cdce1cf3619f311735cbbb8744a1c69536f8a390d014c0a99 3.15MB / 10.00MB 0.2s
2023-03-03T23:11:37.4932885Z #12 sha256:955bce42cb0bc6ba89d9cb7120c1816158f37ce494e1f61810b9d8dbef64aba6 49.28MB / 130.46MB 3.5s
2023-03-03T23:11:37.4933904Z #12 sha256:4efc3ea86e67ce8cdce1cf3619f311735cbbb8744a1c69536f8a390d014c0a99 8.39MB / 10.00MB 0.3s
2023-03-03T23:11:37.6438518Z #12 sha256:47452733a7c0712961d7ef8b48e3d823328f056af1c90c715966748b137b3d1c 46.14MB / 51.87MB 3.3s
2023-03-03T23:11:37.6439028Z #12 sha256:4efc3ea86e67ce8cdce1cf3619f311735cbbb8744a1c69536f8a390d014c0a99 10.00MB / 10.00MB 0.4s done
2023-03-03T23:11:37.7931549Z #12 sha256:c7c452053b608f32eb40cfee6ec0068b9e660f9e1ddfe4c732960e38c5ec5887 102.76MB / 149.01MB 3.6s
2023-03-03T23:11:37.7932045Z #12 sha256:47452733a7c0712961d7ef8b48e3d823328f056af1c90c715966748b137b3d1c 49.28MB / 51.87MB 3.5s
2023-03-03T23:11:37.7932485Z #12 sha256:b4fc05781c9a4a3dee77723abf708dd674020a00d7d57dd6cc1f9bc72da96143 2.10MB / 7.86MB 0.2s
2023-03-03T23:11:37.9435462Z #12 sha256:955bce42cb0bc6ba89d9cb7120c1816158f37ce494e1f61810b9d8dbef64aba6 57.67MB / 130.46MB 3.9s
2023-03-03T23:11:37.9435971Z #12 sha256:47452733a7c0712961d7ef8b48e3d823328f056af1c90c715966748b137b3d1c 51.87MB / 51.87MB 3.6s done
2023-03-03T23:11:37.9436410Z #12 sha256:b4fc05781c9a4a3dee77723abf708dd674020a00d7d57dd6cc1f9bc72da96143 7.86MB / 7.86MB 0.4s done
2023-03-03T23:11:38.0940143Z #12 sha256:d8a6bce2a40cb0c3e3cc6646bfeccfb2ac5b303c39ea70d67e30406d270f2009 2.10MB / 50.45MB 0.2s
2023-03-03T23:11:38.2427083Z #12 sha256:c7c452053b608f32eb40cfee6ec0068b9e660f9e1ddfe4c732960e38c5ec5887 119.54MB / 149.01MB 4.1s
2023-03-03T23:11:38.3933878Z #12 sha256:955bce42cb0bc6ba89d9cb7120c1816158f37ce494e1f61810b9d8dbef64aba6 66.06MB / 130.46MB 4.4s
2023-03-03T23:11:38.3934397Z #12 sha256:d8a6bce2a40cb0c3e3cc6646bfeccfb2ac5b303c39ea70d67e30406d270f2009 7.34MB / 50.45MB 0.5s
2023-03-03T23:11:38.5444087Z #12 sha256:d8a6bce2a40cb0c3e3cc6646bfeccfb2ac5b303c39ea70d67e30406d270f2009 10.49MB / 50.45MB 0.6s
2023-03-03T23:11:38.6919789Z #12 sha256:c7c452053b608f32eb40cfee6ec0068b9e660f9e1ddfe4c732960e38c5ec5887 136.31MB / 149.01MB 4.5s
2023-03-03T23:11:38.8424478Z #12 sha256:955bce42cb0bc6ba89d9cb7120c1816158f37ce494e1f61810b9d8dbef64aba6 75.50MB / 130.46MB 4.8s
2023-03-03T23:11:38.8425012Z #12 sha256:d8a6bce2a40cb0c3e3cc6646bfeccfb2ac5b303c39ea70d67e30406d270f2009 15.73MB / 50.45MB 0.9s
2023-03-03T23:11:38.9929569Z #12 sha256:c7c452053b608f32eb40cfee6ec0068b9e660f9e1ddfe4c732960e38c5ec5887 146.80MB / 149.01MB 4.8s
2023-03-03T23:11:38.9930216Z #12 sha256:d8a6bce2a40cb0c3e3cc6646bfeccfb2ac5b303c39ea70d67e30406d270f2009 19.92MB / 50.45MB 1.1s
2023-03-03T23:11:39.1430794Z #12 sha256:d8a6bce2a40cb0c3e3cc6646bfeccfb2ac5b303c39ea70d67e30406d270f2009 23.07MB / 50.45MB 1.2s
2023-03-03T23:11:39.2932521Z #12 sha256:955bce42cb0bc6ba89d9cb7120c1816158f37ce494e1f61810b9d8dbef64aba6 83.89MB / 130.46MB 5.3s
2023-03-03T23:11:39.2933099Z #12 sha256:d8a6bce2a40cb0c3e3cc6646bfeccfb2ac5b303c39ea70d67e30406d270f2009 27.26MB / 50.45MB 1.4s
2023-03-03T23:11:39.5945311Z #12 sha256:c7c452053b608f32eb40cfee6ec0068b9e660f9e1ddfe4c732960e38c5ec5887 149.01MB / 149.01MB 5.3s done
2023-03-03T23:11:39.5945957Z #12 sha256:d8a6bce2a40cb0c3e3cc6646bfeccfb2ac5b303c39ea70d67e30406d270f2009 33.55MB / 50.45MB 1.7s
2023-03-03T23:11:39.7518431Z #12 sha256:955bce42cb0bc6ba89d9cb7120c1816158f37ce494e1f61810b9d8dbef64aba6 92.27MB / 130.46MB 5.7s
2023-03-03T23:11:39.7519013Z #12 sha256:d8a6bce2a40cb0c3e3cc6646bfeccfb2ac5b303c39ea70d67e30406d270f2009 37.75MB / 50.45MB 1.8s
2023-03-03T23:11:39.8951517Z #12 sha256:d8a6bce2a40cb0c3e3cc6646bfeccfb2ac5b303c39ea70d67e30406d270f2009 40.89MB / 50.45MB 2.0s
2023-03-03T23:11:40.0456261Z #12 sha256:d8a6bce2a40cb0c3e3cc6646bfeccfb2ac5b303c39ea70d67e30406d270f2009 45.09MB / 50.45MB 2.1s
2023-03-03T23:11:40.1890914Z #12 sha256:955bce42cb0bc6ba89d9cb7120c1816158f37ce494e1f61810b9d8dbef64aba6 100.66MB / 130.46MB 6.2s
2023-03-03T23:11:40.1891494Z #12 sha256:d8a6bce2a40cb0c3e3cc6646bfeccfb2ac5b303c39ea70d67e30406d270f2009 50.45MB / 50.45MB 2.4s done
2023-03-03T23:11:40.3401871Z #12 extracting sha256:d8a6bce2a40cb0c3e3cc6646bfeccfb2ac5b303c39ea70d67e30406d270f2009
2023-03-03T23:11:40.6415277Z #12 sha256:955bce42cb0bc6ba89d9cb7120c1816158f37ce494e1f61810b9d8dbef64aba6 109.05MB / 130.46MB 6.6s
2023-03-03T23:11:41.0934341Z #12 sha256:955bce42cb0bc6ba89d9cb7120c1816158f37ce494e1f61810b9d8dbef64aba6 117.44MB / 130.46MB 7.1s
2023-03-03T23:11:41.5453835Z #12 sha256:955bce42cb0bc6ba89d9cb7120c1816158f37ce494e1f61810b9d8dbef64aba6 125.83MB / 130.46MB 7.5s
2023-03-03T23:11:41.6482428Z #12 extracting sha256:d8a6bce2a40cb0c3e3cc6646bfeccfb2ac5b303c39ea70d67e30406d270f2009 1.5s done
2023-03-03T23:11:41.7986545Z #12 extracting sha256:b4fc05781c9a4a3dee77723abf708dd674020a00d7d57dd6cc1f9bc72da96143
2023-03-03T23:11:42.0994747Z #12 sha256:955bce42cb0bc6ba89d9cb7120c1816158f37ce494e1f61810b9d8dbef64aba6 130.46MB / 130.46MB 8.1s done
2023-03-03T23:11:42.0995313Z #12 extracting sha256:b4fc05781c9a4a3dee77723abf708dd674020a00d7d57dd6cc1f9bc72da96143 0.4s done
2023-03-03T23:11:42.0995797Z #12 extracting sha256:4efc3ea86e67ce8cdce1cf3619f311735cbbb8744a1c69536f8a390d014c0a99
2023-03-03T23:11:42.2223209Z #12 extracting sha256:4efc3ea86e67ce8cdce1cf3619f311735cbbb8744a1c69536f8a390d014c0a99 0.2s done
2023-03-03T23:11:42.2223613Z #12 DONE 8.7s
2023-03-03T23:11:42.3724075Z 
2023-03-03T23:11:42.3725121Z #12 [build-env 6/8] RUN go mod download
2023-03-03T23:11:42.3725807Z #12 extracting sha256:47452733a7c0712961d7ef8b48e3d823328f056af1c90c715966748b137b3d1c
2023-03-03T23:11:43.8761018Z #12 extracting sha256:47452733a7c0712961d7ef8b48e3d823328f056af1c90c715966748b137b3d1c 1.5s done
2023-03-03T23:11:43.8761537Z #12 extracting sha256:b5253fb917d27d45dda13ba51813713fac4266088086cd5721f622543cd4a676
2023-03-03T23:11:45.5287464Z #12 extracting sha256:b5253fb917d27d45dda13ba51813713fac4266088086cd5721f622543cd4a676 1.7s done
2023-03-03T23:11:45.5287991Z #12 extracting sha256:c7c452053b608f32eb40cfee6ec0068b9e660f9e1ddfe4c732960e38c5ec5887
2023-03-03T23:11:50.0188804Z #12 extracting sha256:c7c452053b608f32eb40cfee6ec0068b9e660f9e1ddfe4c732960e38c5ec5887 4.6s done
2023-03-03T23:11:50.0189218Z #12 DONE 16.5s
2023-03-03T23:11:50.1687999Z 
2023-03-03T23:11:50.1689044Z #12 [build-env 6/8] RUN go mod download
2023-03-03T23:11:50.1689827Z #12 extracting sha256:e4ac58638d50338f5e89948ce856b91753c7063a48a322f7369d5fd3e1c2ebb3 done
2023-03-03T23:11:50.1691794Z #12 extracting sha256:f2931c2c4bf895d4385799e5199ca517e791b0120f32c78f5f838ace8f735ab1 done
2023-03-03T23:11:50.1692300Z #12 extracting sha256:8aad0f0be38edf49e431a7da225e78749d79d4238b5e0888062af2a9bde73d80 done
2023-03-03T23:11:50.1692743Z #12 extracting sha256:f0ee7d822262943f0c8d7739222b554f1c1b1dbeae5545615a50adb931df121c done
2023-03-03T23:11:50.1693174Z #12 extracting sha256:5e6550d30dade172d76055b9bdeb052f877a3609ad9ce34571a5582a2c3b6884 done
2023-03-03T23:11:50.1693606Z #12 extracting sha256:955bce42cb0bc6ba89d9cb7120c1816158f37ce494e1f61810b9d8dbef64aba6
2023-03-03T23:11:56.4796144Z #12 extracting sha256:955bce42cb0bc6ba89d9cb7120c1816158f37ce494e1f61810b9d8dbef64aba6 6.3s done
2023-03-03T23:11:56.4799977Z #12 DONE 22.8s

See how it says CACHED but then it runs it anyway? And the run takes LONGER than when it's uncached- here's the same step from log_write.zip:

2023-03-03T23:07:36.9842492Z #12 [build-env 6/8] RUN go mod download
2023-03-03T23:07:49.0067500Z #12 DONE 12.1s

This seems to happen to arbitrary steps- while working on this, I had it happen with a step that literally runs a single command and normally takes less than a second:

#10 [build-env 4/9] RUN export REPO_ACCESS_TOKEN=`cat /secrets/REPO_ACCESS_TOKEN` &&     git config --global url."https://${REPO_ACCESS_TOKEN}:[email protected]/gridwise".insteadOf "https://github.com/gridwise"
#10 CACHED

#10 [build-env 4/9] RUN export REPO_ACCESS_TOKEN=`cat /secrets/REPO_ACCESS_TOKEN` &&     git config --global url."https://${REPO_ACCESS_TOKEN}:[email protected]/gridwise".insteadOf "https://github.com/gridwise"
#10 sha256:f0ee7d822262943f0c8d7739222b554f1c1b1dbeae5545615a50adb931df121c 250B / 250B 0.0s done
#10 sha256:8aad0f0be38edf49e431a7da225e78749d79d4238b5e0888062af2a9bde73d80 101B / 101B 0.0s done
#10 sha256:f2931c2c4bf895d4385799e5199ca517e791b0120f32c78f5f838ace8f735ab1 197B / 197B 0.0s done
#10 sha256:e4ac58638d50338f5e89948ce856b91753c7063a48a322f7369d5fd3e1c2ebb3 155B / 155B 0.0s done
#10 sha256:c7c452053b608f32eb40cfee6ec0068b9e660f9e1ddfe4c732960e38c5ec5887 5.24MB / 149.01MB 0.2s
#10 sha256:47452733a7c0712961d7ef8b48e3d823328f056af1c90c715966748b137b3d1c 7.34MB / 51.87MB 0.2s
#10 sha256:4efc3ea86e67ce8cdce1cf3619f311735cbbb8744a1c69536f8a390d014c0a99 2.10MB / 10.00MB 0.2s
#10 sha256:b5253fb917d27d45dda13ba5[181](https://github.com/gridwise/insights-service/actions/runs/4329194547/jobs/7559576409#step:9:185)3713fac4266088086cd5721f622543cd4a676 9.44MB / 68.85MB 0.2s
#10 sha256:c7c452053b608f32eb40cfee6ec0068b9e660f9e1ddfe4c732960e38c5ec5887 19.09MB / 149.01MB 0.3s
#10 sha256:47452733a7c0712961d7ef8b48e3d823328f056af1c90c715966748b137b3d1c 15.73MB / 51.87MB 0.3s
#10 sha256:4efc3ea86e67ce8cdce1cf3619f311735cbbb8744a1c69536f8a390d014c0a99 10.00MB / 10.00MB 0.3s done
#10 sha256:b5253fb917d27d45dda13ba51813713fac4266088086cd5721f622543cd4a676 23.07MB / 68.85MB 0.3s
#10 sha256:c7c452053b608f32eb40cfee6ec0068b9e660f9e1ddfe4c732960e38c5ec5887 35.65MB / 149.01MB 0.5s
#10 sha256:47452733a7c0712961d7ef8b48e3d823328f056af1c90c715966748b137b3d1c 27.26MB / 51.87MB 0.5s
#10 sha256:b5253fb917d27d45dda13ba51813713fac4266088086cd5721f622543cd4a676 39.85MB / 68.85MB 0.5s
#10 sha256:b4fc05781c9a4a3dee77723abf708dd674020a00d7d57dd6cc1f9bc72da96143 2.08MB / 7.86MB 0.2s
#10 sha256:c7c452053b608f32eb40cfee6ec0068b9e660f9e1ddfe4c732960e38c5ec5887 54.53MB / 149.01MB 0.6s
#10 sha256:47452733a7c0712961d7ef8b48e3d823328f056af1c90c715966748b137b3d1c 37.75MB / 51.87MB 0.6s
#10 sha256:b5253fb917d27d45dda13ba51813713fac4266088086cd5721f622543cd4a676 59.77MB / 68.85MB 0.6s
#10 sha256:b4fc05781c9a4a3dee77723abf708dd674020a00d7d57dd6cc1f9bc72da96143 3.15MB / 7.86MB 0.3s
#10 sha256:c7c452053b608f32eb40cfee6ec0068b9e660f9e1ddfe4c732960e38c5ec5887 72.35MB / 149.01MB 0.8s
#10 sha256:47452733a7c0712961d7ef8b48e3d823328f056af1c90c715966748b137b3d1c 49.28MB / 51.87MB 0.8s
#10 sha256:b5253fb917d27d45dda13ba51813713fac4266088086cd5721f622543cd4a676 68.85MB / 68.85MB 0.8s done
#10 sha256:b4fc05781c9a4a3dee77723abf708dd674020a00d7d57dd6cc1f9bc72da96143 7.86MB / 7.86MB 0.5s done
#10 sha256:c7c452053b608f32eb40cfee6ec0068b9e660f9e1ddfe4c732960e38c5ec5887 82.84MB / 149.01MB 0.9s
#10 sha256:d8a6bce2a40cb0c3e3cc6646bfeccfb2ac5b303c39ea70d67e30406d270f2009 13.63MB / 50.45MB 0.2s
#10 sha256:c7c452053b608f32eb40cfee6ec0068b9e660f9e1ddfe4c732960e38c5ec5887 96.47MB / 149.01MB 1.1s
#10 sha256:47452733a7c0712961d7ef8b48e3d823328f056af1c90c715966748b137b3d1c 51.87MB / 51.87MB 1.0s done
#10 sha256:d8a6bce2a40cb0c3e3cc6646bfeccfb2ac5b303c39ea70d67e30406d270f2009 32.73MB / 50.45MB 0.3s
#10 sha256:c7c452053b608f32eb40cfee6ec0068b9e660f9e1ddfe4c732960e38c5ec5887 113.25MB / 149.01MB 1.2s
#10 sha256:d8a6bce2a40cb0c3e3cc6646bfeccfb2ac5b303c39ea70d67e30406d270f2009 50.45MB / 50.45MB 0.5s
#10 sha256:c7c452053b608f32eb40cfee6ec0068b9e660f9e1ddfe4c732960e38c5ec5887 128.97MB / 149.01MB 1.4s
#10 sha256:d8a6bce2a40cb0c3e3cc6646bfeccfb2ac5b303c39ea70d67e30406d270f2009 50.45MB / 50.45MB 0.5s done
#10 extracting sha256:d8a6bce2a40cb0c3e3cc6646bfeccfb2ac5b303c39ea70d67e30406d270f2009
#10 sha256:c7c452053b608f32eb40cfee6ec0068b9e660f9e1ddfe4c732960e38c5ec5887 148.58MB / 149.01MB 1.5s
#10 sha256:c7c452053b608f32eb40cfee6ec0068b9e660f9e1ddfe4c732960e38c5ec5887 149.01MB / 149.01MB 2.0s done
#10 extracting sha256:d8a6bce2a40cb0c3e3cc6646bfeccfb2ac5b303c39ea70d67e30406d270f2009 1.5s done
#10 extracting sha256:b4fc05781c9a4a3dee77723abf708dd674020a00d7d57dd6cc1f9bc72da96143
#10 extracting sha256:b4fc05781c9a4a3dee77723abf708dd674020a00d7d57dd6cc1f9bc72da96143 0.2s done
#10 extracting sha256:4efc3ea86e67ce8cdce1cf3619f311735cbbb8744a1c69536f8a390d014c0a99
#10 extracting sha256:4efc3ea86e67ce8cdce1cf3619f311735cbbb8744a1c69536f8a390d014c0a99 0.2s done
#10 DONE 3.4s

#10 [build-env 4/9] RUN export REPO_ACCESS_TOKEN=`cat /secrets/REPO_ACCESS_TOKEN` &&     git config --global url."https://${REPO_ACCESS_TOKEN}:[email protected]/gridwise".insteadOf "https://github.com/gridwise"
#10 extracting sha256:47452733a7c0712961d7ef8b48e3d823328f056af1c90c715966748b137b3d1c
#10 extracting sha256:47452733a7c0712961d7ef8b48e3d823328f056af1c90c715966748b137b3d1c 1.4s done
#10 extracting sha256:b5253fb917d27d45dda13ba51813713fac4266088086cd5721f622543cd4a676
#10 extracting sha256:b5253fb917d27d45dda13ba51813713fac4266088086cd5721f622543cd4a676 1.8s done
#10 extracting sha256:c7c452053b608f32eb40cfee6ec0068b9e660f9e1ddfe4c732960e38c5ec5887
#10 extracting sha256:c7c452053b608f32eb40cfee6ec0068b9e660f9e1ddfe4c732960e38c5ec5887 5.1s done
#10 extracting sha256:e4ac58638d50338f5e89948ce856b91753c7063a48a322f7369d5fd3e1c2ebb3 done
#10 extracting sha256:f2931c2c4bf895d4385799e5[199](https://github.com/gridwise/insights-service/actions/runs/4329194547/jobs/7559576409#step:9:203)ca517e791b0120f32c78f5f838ace8f735ab1 done
#10 extracting sha256:8aad0f0be38edf49e431a7da225e78749d79d4238b5e0888062af2a9bde73d80 done
#10 extracting sha256:f0ee7d8[222](https://github.com/gridwise/insights-service/actions/runs/4329194547/jobs/7559576409#step:9:226)62943f0c8d7739222b554f1c1b1dbeae5545615a50adb931df121c done
#10 DONE 11.7s

What does this mean? Why is it happening?

@tonistiigi
Copy link
Member

It doesn't pull in the cache only for that command. It needs to pull in all the previous layers as well. Where do you see that step is cached but still executes as well? For the step that runs go mod download that command would pull in tarballs and extract them locally what is pretty similar to what happens on cache import. It isn't surprising that times are similar for such cases.

@aaronleopold
Copy link

aaronleopold commented Mar 10, 2023

Hello, I apologize if this isn't exactly the right place but wanted to piggyback off of this issue for a question I feel might be relevant and potentially a similar issue, and discussions aren't available for this repo.

I've successfully gotten a working cache locally (outside of actions context) but it breaks in CI. Effectively, I have a procedural workaround to get docker to cache the compiled Rust dependencies so changes to the codebase don't invalidate the docker cache and trigger all the dependencies to get recompiled from scratch (Dockerfile). Locally, like I said, this works great. I can load a build for one platform, make a small change somewhere, run another build and the caches make it a breeze.

In CI, the only time the cache actually gets loaded for majority of the layers is on consecutive runs without any changes. If I make a tiny change to the code, most layers weren't recovered and all the deps get recompiled again. Here is the workflow.yml.

If you feel this warrants a separate issue let me know and I'll make one

@mzealey
Copy link

mzealey commented Sep 8, 2023

Following on from #829 (comment) , we are also seeing this same issue with Python. We have a multi-stage build which does setup, copies requirements.txt, runs install etc. Using a standard registry cache it caches all the layer just fine. However if we use gha cache it seems to not cache some of the stages correctly.

ie if we do the following diff, our builds take a lot longer because 1. some of the layers arn't cached, and 2. those that are suddenly have a lot more cache layers seemingly downloaded to them:

-          cache-from: type=registry,ref=ghcr.io/org/${{ matrix.distribution.image }}:buildcache
-          cache-to: type=registry,ref=ghcr.io/org/${{ matrix.distribution.image }}:buildcache,mode=max
+          cache-from: type=gha
+          cache-to: type=gha,mode=max

@krhubert
Copy link

I see the same behavior, Are there any updates on this issue?

@oerp-odoo
Copy link

oerp-odoo commented Nov 1, 2023

I am also struggling with caching for multi stage builds or multi image builds. I tried both build-push-action and bake-action. With various caching modes (registry, gha, local), none work correctly. I have base image and then final image based on that base.

I tried caching both, but for some reason only base image is cached. I see cache being exported for both, but when I re-run build command, even if nothing changed, it will always rebuild final image, where base image will be cached normally.

The only way I was able to fully cache it, was using local docker instead of build-push-action or bake-action. Then it fully utilizes cache. But then you get into other problems (like sharing cache between different runners and disk space management issues..)

So if direct use of docker manages cache properly, I guess something is wrong with build-push-action and bake-action or the way it uses docker buildx.

You can see my open issue about that: docker/bake-action#154

P.S. If I change it to build as single image without any base image, then it caches properly..

@moshest
Copy link

moshest commented Feb 11, 2024

I also have the same issue when building the Next.js & Turborepo docker file example.

It worked great locally but when using in Github actions it always rebuild the images in the first COPY command, even if there is no file changes.

@BastienFaure
Copy link

Interesting issue. I have a similar behavior but as opposed as the author's problem, the pre-stage is properly cached for me but the second part if systematically rebuilt even if there no file change.

@moshest I have a COPY even in the pre-stage and it doesn't get rebuilt, however the second part gets rebuilt entirely:

FROM public.ecr.aws/amazonlinux/amazonlinux:2 as builder
LABEL maintainer="Bastien Faure <[email protected]>"

#ENV LC_ALL fr_FR.UTF-8

# install packages
RUN yum groupinstall -y "Development Tools"
RUN amazon-linux-extras enable python3.8
RUN amazon-linux-extras enable mariadb10.5
RUN yum install -y python38 python38-devel mariadb pkgconfig
RUN yum install -y libcurl-devel openssl-devel

# update python build tools
RUN pip3.8 --no-cache-dir --disable-pip-version-check install --upgrade pip setuptools wheel

COPY src/requirements.txt /tmp/requirements.txt
RUN mkdir /build
RUN pip3.8 install --no-cache-dir --disable-pip-version-check -r /tmp/requirements.txt --target /build


# install serverless_wsgi bridge
# RUN pip3.8 install serverless_wsgi

FROM public.ecr.aws/lambda/python:3.8 as afelp_app

ENV PYTHONUNBUFFERED 1
ENV PYTHONPATH /app

# install prebuilt dependencies
COPY --from=builder /build /app
COPY --from=builder /usr/lib64/mariadb/plugin/caching_sha2_password.so /usr/lib64/mariadb/plugin/caching_sha2_password.so
COPY --from=builder /lib64/libmariadb.so.3 /lib64/libmariadb.so.3

# copy the lambda entry points
COPY lambdas/api/lambda.py /app/api_lambda.py
COPY lambdas/tasks/lambda.py /app/tasks_lambda.py

# install and configure texlive
RUN yum clean metadata
RUN yum install -y wget tar gzip perl-Digest-MD5
RUN cd /tmp \
	&& wget https://mirror.math.princeton.edu/pub/CTAN/systems/texlive/tlnet/install-tl-unx.tar.gz \
	&& tar xf install-tl-unx.tar.gz \
	&& rm install-tl-unx.tar.gz \
	&& cd install-tl* \
	&& ./install-tl --scheme basic --no-src-install --no-doc-install --no-interaction

# add local texlive install to the path
ENV PATH="/usr/local/texlive/2023/bin/x86_64-linux:${PATH}"
# install required packages
RUN tlmgr install eurosym xcolor babel-french etoolbox csquotes titling caption varwidth multirow datetime fmtcount xkeyval wasysym wasy xltabular ltablex cm-super

# generate fonts
RUN mktexpk --destdir /usr/local/texlive/2023/texmf-var/fonts/pk/ljfour/public/wasy/ --mfmode / --bdpi 600 --mag 1+0/600 --dpi 600 wasy10

# copy application source
COPY src/afelp /app/afelp
# app environment variables
ENV TZ Europe/Paris
ENV LANG fr_FR.UTF-8

CMD ["api_lambda.lambda_handler"]

Wondering if there is any optimization I could do since the build action takes hella time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

9 participants