PROPOSAL: Core Type System

[aluzzardi]

Intro

In its current state, this is more of a brain dump with a straw man proposal rather than a full fledged proposal. Just enough to get some ideas bouncing.

Background

Currently, the concept of computation (e.g. a set of compute operations) and data (e.g. the resulting filesystem from the computation) are mixed together as a single concept, dagger.#Artifact with an open definition:

// An artifact such as source code checkout, container image, binary archive...
// May be passed as user input, or computed by a buildkit pipeline
#Artifact: {
	@dagger(artifact)
	#up: [...op.#Op]
	_
	...
}

Packages expecting data as input actually take a computational pipeline (in the form of an #Artifact with accompanying #up instructions). At runtime, the artifact is computed and replaced by the resulting buildkit state (e.g. filesystem).

For instance, docker.#Build is defined as such:

#Build: {
	// Build context
	source: dagger.#Artifact
	[...]
}

If we were to pass a git.#Repository as the source argument, it would eventually CUE evaluate to something like this:

#Build: {
	// Build context
	source: {
		...
		_
		#up: [op.#FetchGit & { ref: "..." }]
	}
	[...]
}

As soon as #Build attempts to manipulate source (for instance, copying its content), dagger will compute the pipeline on the fly (i.e. #FetchGit operation) and use the resulting filesystem.

Access to local directories (e.g. dagger input dir) works in the same way and is "compiled" to an artifact:

dagger.#Artifact & {
  #up: [ op.#Local & { ... } ]
}

Problem

• The #Artifact concept is not intuitive as it serves different purposes (compute and data).
• Performance: Nested definitions (e.g. mount takes an #Artifact which itself provides mount which itself provides an #Artifact, ...) have caused various performance issues. This led to all #Artifact references being replaced by _ as a workaround. Related performance issues keep coming up [reference needed -- @samalba @grouville?]
• The open definition (e.g. { ... _ }) of #Artifact causes problems. Since anything is technically an artifact, it confuses CUE's type system (e.g. if (source & string) != _|_). Passing a wrong argument (or forgetting to provide it) causes hard to understand CUE errors [reference needed -- @samalba?]
• Caching Issues (Fix caching issues #914). This is actually a broader runtime problem, discovered by caching issues but not limited to (e.g. it's affecting docker.#Load as well). In a nutshell:
  • There is no way to differentiate between an Artifact and a reference to an Artifact
  • This leads to every Artifact being computed every time they are referenced
  • BuildKit helps with deduplication, avoiding unnecessary reference computation. However it is limited to LLB instructions.
  • non-LLB operations (e.g. gRPC calls to read files, call to external frontends, pushing images, etc) are not de-duplicated (see Duplicate log messages #449 Re-use previous Pipeline results rather than relying on BuildKit caching #591)
  • non-idempotent operations will compile to a different LLB state each time they are referenced, defeating BuildKit cache. For instance, operations using the current cue path (e.g. cache mounts Broken BuildKit deduplication when using cache mounts #495) and operations using randomness (e.g. always: true which is re-computed at each reference Duplicate execution with always: true #42)

Proposal

WIP

• Split #Artifact into computation (#Pipeline?) and data (#Directory?)
• Pass data directly (e.g. source: #Directory) rather than computation
• Close definitions (e.g. avoid { ... _})

Directory (data)

// Dagger Directory, represents LLB state
#Directory: {
	@dagger(directory)
	

	// For internal use only.
	// This is used by the runtime to store information during computation.
	// Ideally this could be hidden somehow (perhaps using attributes, _id, $id, ...)
	id: string
}

• A directory represents LLB state (e.g. a filesystem)
• Can be the result of computation or a user provided directory, doesn't matter
• Internally, it maps to computed LLB state. The id field is there to help the runtime perform the mapping, it's an implementation detail
• All definitions requiring data as input will take a #Directory directly (e.g. source: dagger.#Directory). This is true for built-in definitions as well (e.g. #Exec: { mount: [string]: dagger.#Directory })
• A user provided directory (e.g. currently dagger input dir) maps to a #Directory. It's the job of the runtime to figure out how solve the mapping.

Pipeline (computation)

WIP

#Pipeline: {
	@dagger(pipeline)

	#up: [...op.#Op]

	fs: #Directory
}

• Similar to today's #Artifacts (e.g. #up instructions are executed)
• fs is the bridge with #Directory
  • It represents the computed LLB state (the result "image" of the pipeline)
  • Allows the pass the filesystem of the pipeline along, without having to pass the entire pipeline
• The definition is closed. No more ... _

Example:

build: dagger.#Pipeline & {
    #up: [
        op.#FetchContainer & { ref: "alpine" },
        op.#Exec & { args: ["sh", "-c", "echo -n hello world > /output" ] }
    ]
}

deploy: dagger.#Pipeline & {
    #up: [
        op.#FetchContainer & { ref: "alpine" },
        op.#Exec & {
            // `build.fs` points to the filesystem of the `build` pipeline. Passing `build` here would fail.
            mount: "/src": from: build.fs

            args: ["sh", "-c", "cat /src/output" ]
        }
    ]
}

Exporting

WIP

Beside filesystem contents, computation pipelines are passed around for their exported values.

With the current #Artifact it looks like this:

url: {
	string

	#up: [
		[...]
		op.#Export & { ...}
	]
}

• This triggers the same issues as passing around directories
• Doesn't allow #Pipeline to be closed
• Won't work with the additional fs field (e.g. can't be both a string and contain fs)

One potential solution would be to adopt the pattern of os.#Container: there's no built-in export, exporting happens from the filesystem instead.

Example:

// Netlify site
#Site: {
        ctr: dagger.#Pipeline & {
            #up: [
                op.#FetchContainer & { ref: "alpine" },
                op.#Exec & { args: ["sh", "-c", "echo -n http://localhost:8080 > /netlify/url" ] }
            ]
        }

	// Website url
	url: {
		os.#File & {
			from: ctr.fs
			path: "/netlify/url"
		}
	}.contents
}

User-facing Directories (integration with #1058)

Splitting computation and data breaks the current way of injecting user directories usingop.#Local in an #Artifact. It's a happy coincidence since that method doesn't play very nice with #1058 IMO (/cc @talentedmrjones).

A better approach would be "declarative format" for directories:

• User somehow declares a local directory in context.directories
• Runtime computes context directories back to regular #Directory (with the id and such). No need for an op.#Local
• context.directories can be passed around in packages

There are a couple alternative DX here:

• Merging
• Wrapping (disclaimer: my favorite so far)
• Embedding (is it possible?)

Merging

Use the same #Directory definition both for "produced" directories and "user defined" directories. This is achieved by merging the base #Directory fields with additional (optional) user-settable fields.

Example definition:

package dagger

// Dagger Directory
#Directory: {
	@dagger(directory)
	
	// User defined. Can only be provided within `context.directories`, will be ignored otherwise.
	path?: string

	// Other user fields will go here.
	// include?: [string]
	// exclude?: [string]

	// For internal use only.
	id: string
}

Example of a plan:

package main

import (
  “alpha.dagger.io/dagger”
)

dagger.#Plan

context: {
  directories: test: dagger.#Directory & {
    path: "./test"
  }
}

actions: {
  build: yarn.#Build & {
    source: context.directories.test
  }
}

• Pros: Very simple, there's a single type #Directory type
• Cons: User-defined fields (such as path) which will be ignored if set outside of context.directories. Can be confusing to have fields which you're not supposed to set.

Wrapping

Here we keep the original #Directory and add a new definition specifically meant to be used in context. This new definition wraps #Directory

Example definition:

package context

#LocalDirectory: {
	path: string

	contents: #Directory
}

Example of a plan:

package main

import (
  “alpha.dagger.io/dagger”
  “alpha.dagger.io/dagger/context”
)

dagger.#Plan

context: {
  directories: test: context.#LocalDirectory & {
    path: "./test"
  }
}

actions: {
  build: yarn.#Build & {
    source: context.directories.test.contents
  }
}

• Pros: Clean separation between the two types. No need to guess which fields in #Directory one is supposed to fill or not
• Pros: Layer of abstraction between context and code.
  • For instance, in addition to #LocalDirectory we could have #SSHDirectory with entirely different fields.
  • This same pattern would make a lot of sense for secrets (e.g. dagger.#Secret and context.#AWSVaultSecret, context.#SOPSSecret, ...) and services (e.g. dagger.#Stream and context.#UnixSocket, context.#TCPSocket). Merging would require cramming every option into a single type.
  • Better dependency management: contextual types (#LocalDirectory) are only relevant to the runtime to compute a universal type (#Directory). If a package uses an old version of #Directory it shouldn't prevent the plan to use a new contextual type (e.g. #AWSVaultSecret).
• Cons: One more type for the user to learn
• Cons: The syntax context.directories.xxx.contents is a bit cumbersome

Embedding

I'm not sure if this would work.

This is a hybrid between merging and wrapping.

Example definition:

package context

#LocalDirectory: {
	dagger.#Directory

	path: string
}

Example of a plan:

package main

import (
  “alpha.dagger.io/dagger”
  “alpha.dagger.io/dagger/context”
)

dagger.#Plan

context: {
  directories: test: context.#LocalDirectory & {
    path: "./test"
  }
}

actions: {
  build: yarn.#Build & {
    source: context.directories.test
  }
}

• Pros: Gets rid of thecontext.directories.xxx.contents syntax (e.g. context.directories.xxx works)
• Pros: Most of the benefits of wrapping
• Cons: There are still additional types

[samalba]

Overall I like it, it's more explicit (for instance: referencing the .fs key instead of the bare component key).

However we should be careful with root definitions:

For example, instead of:

#Directory: {
	@dagger(directory)
	id: string
}

I would do:

#Directory: {
	@dagger(directory)
	directoryID: string
}

Without it, #Secret and #Directory are likely to have the same definition keys and it won't be possible to use indections such as myKey: #Secret | #Directory.

[samalba]

Given dagger.#Pipeline is now closed, do we still need #up to be a def? I guess not, we could simply rename it to up.

(Maybe it allows to differentiate 2 pipelines in cue by their list of Ops. Example: myKey: yarn.#Pipeline | node.#Pipeline. If #up is a def, it cannot be taken into account in an indirection.)

[TomChv]

That looks smooth, the more we type our cue and pipeline, the more it's safe and easy to use.

There are a couple alternative DX here:

• Merging
• Wrapping (disclaimer: my favorite so far)
• Embedding (is it possible?)

I think wrapping or embedding is the best solution, it feels more natural and we can understand what happens in a single reading.

Given dagger.#Pipeline is now closed, do we still need #up to be a def? I guess not, we could simply rename it to up.

(Maybe it allows to differentiate 2 pipelines in cue by their list of Ops. Example: myKey: yarn.#Pipeline | node.#Pipeline. If #up is a def, it > cannot be taken into account in an indirection.)

That would be amazing! And could help people build complex packages that can use a predefined set of definitions.
I'm sure it will be useful to develop the serverless agnostic package for instance.