Dockerfile multiphase build cache issues

[ilyapoz]

High Level Spec

• Package docker-multiphase-handler in deb/rpm files and host on packagecloud.
• Modify Dockerfile building to build in stages, labeling each stage appropriately
• Modify docker_cleanup to avoid removing stages
• Modify the scheduler-docker-cleanup hook to avoid removing stages.

Original Post

Description of problem:
Let's say I use this Dockerfile:

Dockerfile.txt

report.txt

Environment details (AWS, VirtualBox, physical, etc.):
vps

How was Dokku installed?:
znz.dokku ansible galaxy role

How reproducible:
every time

Steps to Reproduce:

1. push an app consisting only of the Dockerfile attached
2. push some meaningless commit to trigger new build

Actual Results:
RUN echo in builder phase is not cached

Expected Results:
RUN echo is cached as in docker build . run

Additional info:
main phase builds are cached correctly by content! I.e. changing the exact method of creating some_file.txt in build phase doesn't affect caching in the main phase.

[josegonzalez]

Can you gist the output of a deploy before and after running dokku trace on? I have a feeling that our cleanup step is removing the intermediate build layer since its not used, but getting confirmation of that would be great.

[ilyapoz]

before: https://gist.github.com/ilyapoz/675d3be5799e2821220ee91857f550af
after: https://gist.github.com/ilyapoz/b9e2c4697db1b599d1193f4a804e3b3a

[josegonzalez]

Can you run dokku trace off and run the deploy?

[ilyapoz]

https://gist.github.com/ilyapoz/a9984e870d01f561f928f20c6907b0a2

[alexquick]

@josegonzalez I took a look and the image from the builder phase is getting removed at: https://github.com/dokku/dokku/blob/master/plugins/common/functions#L650 as you suspected.

[josegonzalez]

Not sure there is a great way around that, given our deployment methodology. Any image cleanup will break this, so perhaps having folks manage that on their own when they are using multi-stage builds is the best way?

[josegonzalez]

Looks like we might be able to leverage dockfmt to list all stages in an image, and then build the image by specifying targets/tagging each layer. We'd then avoid removing tagged layers during the cleanup stage.

[josegonzalez]

So my original plan won't work unless each stage is named, meaning we would have to either drop the cache-fix for non-named stages or rewrite the dockerfile to ensure each stage is named. I don't know how I feel about either, though the later method is probably possible using a fun regex and the post-extract hook.

[hiepxanh]

hello, I think i have solution for this, I'm looking a way to cache my package manager dont have to reinstall every time have CI build or dokku build on server and now I have it https://github.com/dokku/dokku/blob/c1df31b106da6eb8c7897deea67c143e2b5e611a/docs/deployment/builders/dockerfiles.md
I have make an PR for the docs on here
#4635

in my case I'm using pnpm (the same like npm or yarn) and I spend about 10 minutes or 20 minutes for each time package.json file change, and with the new command, It can save me about 5 minutes build time, which save alot of money sir :

FROM node:14-alpine as builder
ENV CYPRESS_INSTALL_BINARY=0
RUN --mount=type=cache,target=/root/.pnpm-store/v3
RUN pnpm install --frozen-lockfile --prod