DO NOT MERGE: buildah vendor treadmill

[edsantiago]

Followup to Cabal discussion 2022-04-07, re: letting us vendor in buildah on a moment's notice.

Instead of cron'ing this, let's see if this works: keep this PR open perpetually, with daily updates to get latest podman and buildah.

See individual commit messages for details.

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: edsantiago

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [edsantiago]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[edsantiago]

The first commit is the important one. This one must be hand-crafted by a human, there is no possible way to automate it. This is the one that, in case of emergency vendor, can be grabbed into a real PR.

The second commit is a throwaway, ignore it. That's just automated fetch-the-latest-buildah.

[edsantiago]

bud test failed, remote only, because the new --identity-label=false option is not getting passed through. Probably an easy fix, but not this late in my day.

This is still a proof-of-concept PR, and my feeling so far is positive: it has quickly identified multiple problems in the buildah merge that are easy to solve iteratively. A nightly cron job, or a non-gating step in buildah CI, would be utterly useless because neither one would have the crucial first-commit. Without that, there's no way to make it past the first or second or third failure.

[rhatdan]

I just pushed a fix for this.

[Luap99]

How is this PR regularly updated?

[edsantiago]

By a human, presumably me. There is actually no other way: this is not possible to solve via cron or PRs.

I am working on a script to automate as much of it as possible.

[edsantiago]

Followup, because I don't want anyone to think this is abandonware: it's not. I'm running my script at least twice a day, one of those times in the evening. One of the nice features I added is:

+---> Podman has bumped, but Buildah is unchanged. There's probably not much point to testing this.

So I'm choosing to save CI cycles by not re-pushing today. Should anyone need to re-vendor buildah tomorrow or this weekend (when I'm unavailable), just cherry-pick the first commit from this PR. It's still valid.

[edsantiago]

Yippee, script has caught a bug! containers/buildah#3919 breaks when vendoring into podman:

+---> Running 'make' to confirm that podman builds cleanly...
CGO_ENABLED=1 GOOS=linux GOARCH=amd64 go build \
        -mod=vendor  \
        -ldflags '-X github.com/containers/podman/v4/libpod/define.gitCommit=3bf05d3113df29addd392032797f4fdf5dae56b7 -X github.com/containers/podman/v4/libpod/define.buildInfo=1650539506 -X github.com/containers/podman/v4/libpod/config._installPrefix=/usr/local -X github.com/containers/podman/v4/libpod/config._etcDir=/usr/local/etc -X github.com/containers/common/pkg/config.additionalHelperBinariesDir= ' \
        -tags "   selinux systemd  exclude_graphdriver_devicemapper seccomp" \
        -o bin/podman ./cmd/podman
# github.com/containers/podman/v4/libpod
libpod/events.go:18:3: cannot use r.config.Engine.EventsLogFileMaxSize (type "github.com/containers/common/pkg/config".eventsLogMaxSize) as type uint64 in field value
make: *** [Makefile:321: bin/podman] Error 2
buildah-vendor-treadmill-x: 'make' failed with new buildah. Cannot continue.

I could just force-push right now and let y'all see the breakage in CI, or I could just report it like I am right here. Or I could file an issue in containers/common since that's where the offending code seems to be?

Anyhow, @rhatdan et al, PTAL because this is not going to vendor into podman.

[edsantiago]

What the heck, I pushed anyway in case someone wants to pull & play.

[packit-as-a-service]

Cockpit tests failed for commit c63d8a0. @martinpitt, @jelly, @mvollmer please check.

[packit-as-a-service]

Cockpit tests failed for commit b8aa0c7. @martinpitt, @jelly, @mvollmer please check.

[packit-as-a-service]

Cockpit tests failed for commit 34fcb47. @martinpitt, @jelly, @mvollmer please check.

[packit-as-a-service]

Ephemeral COPR build failed. @containers/packit-build please check.

[edsantiago]

@containers/buildah-maintainers PTAL. Something in latest buildah is breaking podman int tests. As best I can tell, the common factor is that buildah is now re-pulling images that are already present:

old (good):
$ podman build ...
STEP 1 2 3 etc

new (bad):
$ podman build ...
Trying to pull .....  <<<<<<<<<< SHOULD NOT HAPPEN

(I may be wrong in my analysis. Maybe the common factor is something else I didn't notice)

[nalind]

@containers/buildah-maintainers PTAL. Something in latest buildah is breaking podman int tests. As best I can tell, the common factor is that buildah is now re-pulling images that are already present:

It looks like the image caching save/load sequence is changing the compression used for the image's layer, and thus the manifest, and thus the manifest's digest, and when build checks if the image in the registry has a different digest when compared to the one it has on disk, it sees that it is and pulls it again.

To verify, compare the RepoDigests values you get from podman inspect quay.io/libpod/alpine after freshly pulling the image, and after a podman save/podman rmi/podman load cycle.

[edsantiago]

Thank you. Looks like a reasonable analysis. How do we fix that?

[Luap99]

@containers/buildah-maintainers PTAL. Something in latest buildah is breaking podman int tests. As best I can tell, the common factor is that buildah is now re-pulling images that are already present:

It looks like the image caching save/load sequence is changing the compression used for the image's layer, and thus the manifest, and thus the manifest's digest, and when build checks if the image in the registry has a different digest when compared to the one it has on disk, it sees that it is and pulls it again.

To verify, compare the RepoDigests values you get from podman inspect quay.io/libpod/alpine after freshly pulling the image, and after a podman save/podman rmi/podman load cycle.

But nothing here changed that looking at the diff, it used too work fine and I cannot see any config/compression chnages here

What I do see however is containers/buildah#5478

if platform.OS == "" {
	platform.OS = runtime.GOOS
}

AFAIK the libimage code treats a non empty platform as must pull or something like that if I remember correctly
However I haven't verified that.

[nalind]

Hmm, this has been happening for a while, since I see it happening with both podman 4.9.4 and 5.0.0 as well, so it might not be the only cause.
But to your question, buildah uses a "dir:" location for the cache to avoid changing the image while caching it, and a custom "copy" test helper since it doesn't want to depend on having skopeo available at test time. But podman already expects skopeo to be present, so skopeo copy should be up to the task.

[nalind]

AFAIK the libimage code treats a non empty platform as must pull or something like that if I remember correctly However I haven't verified that.

I had suspected as much, but at most, that should be changing the pull policy to "if-newer" under the covers, and I'm pretty sure that's already the value that's being passed in.

[Luap99]

AFAIK the libimage code treats a non empty platform as must pull or something like that if I remember correctly However I haven't verified that.

I had suspected as much, but at most, that should be changing the pull policy to "if-newer" under the covers, and I'm pretty sure that's already the value that's being passed in.

Yeah not sure, but the fact is main is passing but this PR is not so the problem must be in the diff here and thankfully the diff is small so this is my best guess as of why. I can try to reproduce tomorrow.

[edsantiago]

What I do see however is containers/buildah#5478

if platform.OS == "" {
platform.OS = runtime.GOOS
}

Nice catch, @Luap99 . Removing those three lines (and the import runtime) appears to fix the problem.

[rhatdan]

That was added to make podman build --arch arm64 work

[Luap99]

That was added to make podman build --arch arm64 work

Sure but it fixed it in broken way, the default pull policy is ifmissing and not ifnewer which the code uses now due the well other not so great behavior in libimage.
And reading your PR your intention is to only set TARGETARCH not change the pull policy so this is definitely broken and must be fixed before the buildah release

[edsantiago]

Tests now passing. This is safe to vendor into podman; no changes needed. Thank you @nalind for the quick fix.