Support Buildpacks.io

[helderco]

It would be great if Dagger could support building images with Cloud Native Buildpacks. More specifically, I'd like to integrate with Pack, which is both a Command Line Interface and a Go library.

Apart from executing the pack binary from CUE, either directly or via a custom image, there seems to be 2 ways to support this that I can see.

1. Have pack support a buildkit frontend

There's an issue (buildpacks/pack#768, presentation) investigating the usage of a frontend to make Pack work with BuildKit. It's still a proof of concept right now, the idea being to have this built-in in the future if it's worked out.

A possible downside is that this doesn't support all that Pack can do. But, in any case, I think Dagger should still allow using different frontends.

2. Add cue bindings for pack's Go library

I think this would be ideal. Hope it's ok to paste conversations from Discourse here.

The question from @TomChv is:

We already have a docker package that set a CLI, so I think it's not to hard to create a custom cue definition that will handle build with buildback, but I'm not sure how we will load the image built into the dag.

If it's possible to export an artifact, that could works

@shykes explains further:

The remaining issue is how to pass the complete OCI image produced by a pack Cue component to other arbitrary components. Currently you can pass the buildkit state (ie the filesystem contents of the image) but other OCI metadata might be left behind

Basically we’re missing a feature in our buildkit bindings for Cue, and your use case probably depends on that missing binding. At least we think so.

And @TomChv replied:

After I’ve finish my work on multi-arch, I can start working on that

I’ve gain knowledge during my research that I think can help us find a solution

[helderco]

The buildpacks/pack#768 issue above mentions some findings around export.

It is the Lifecycle and not the Platform (i.e., Pack) that makes the export. Which means that pack exports to the local daemon by default, unless using the --publish option to write to a registry. The exporting is actually done inside the builder container, not by pack itself.

There's an issue buildpacks/imgutil#100 to support saving images as archives. Also from buildpacks/lifecycle#423 (comment):

This may also be useful for the BuildKit integration. Specifically in removing the need to reimplement export.

It would require more thought but maybe the export phase could take an option to export as oci layout format. This oci layout directory or archive can then be read by the BuildKit frontend and processed to include layers and image config (labels, env vars, etc).

Although performance wise, there's also this from buildpacks/lifecycle#423 (comment):

Note from discussion: Generating a tar means having to write all layers (as opposed to being able to skip them during publish)

I would actually be happy to have the CUE binding ask pack to publish to the registry directly instead of trying to pass the image to docker.#Push.

[TomChv]

Maybe a silly question but I want to be sure :

If we can create a standalone package that exposes a definition in cue to build & push to a remote registry, that would be a nice workaround until we support buildpacks artifact export?

E.g

package buildpacks

import (
    "alpha.dagger.io/dagger"
)

// Build with buildpacks and push the result to a remote registry
#RemoteImage: {
     source: dagger.#Artifact

     ref: string

     creds: ...

     #up: [
         // Login to the remote registry
         // Build with buildpack
         // Push the result to the ref
     ]
}

@samalba @grouville Do you think it's a good idea and should it be a standalone package or do we want it in the standard library?

[helderco]

@TomChv yeah, but better than just the registry is if we can have pack send to the local daemon (it's what it does by default). We can use docker.#Load after that if needed.

#PackBuild {
    ...
    publish: bool
    ...
}

Are you thinking on using the Go library? I can always use Exec to the cli but this way there's no need to install pack.

[TomChv]

For the moment I'm not sure, I have to verify what is necessary to do to use the Go library, I'll update you when I have more information.

[helderco]

Now I think it's fine to keep it in CUE and call the pack cli in a container (rootless.... no need for binding host socket, like kpack).

[helderco]

Note: https://blog.codecentric.de/en/2021/10/gitlab-ci-paketo-buildpacks/

TLDR: Use lifecycle directly. Run inside a builder image (which includes everything you need), without having to bind the docker socket (unless you want to build to the local daemon). Also, since it's the lifecycle that does the exporting, we can explore how the output can be integrated with dagger's DAG.

Dagger would become the platform then.

[helderco]

In buildpacks/pack#564 (comment):

Another way of describing this is: the lifecycle is comparable to kaniko or other unprivileged image building tools. The pack CLI is glue code that makes it easy to use lifecycle with the Docker daemon. We could expand the functionality of the pack CLI so that it acts as glue code for other container runtimes, but that glue code is only necessary when containers are not natively accessible already. [...]

Pack is the glue, but we already have a glue 😁

Also, from buildpacks/pack#564 (comment):

There are currently two ways to use the tooling provided by the Cloud Native Buildpacks project:

1. With the pack CLI. This uses Docker (purely as a container runtime, without using docker build) to run a builder image (which contains the lifecycle). Docker is available and easy to install on macOS, Windows, and Linux.

2. Without the pack CLI, by executing a builder image directly on platform that can already run containers (like k8s). Tekton, kpack, and concourse use this strategy. It does not require Docker or privileged containers.

[...] Running pack inside of a container (which creates nested containers) is unnecessarily and decreases performance. The lifecycle can run directly in that container instead. [...]