Proposal: Dagger Attributes

[verdverm]

We propose using CUE Attributes to mark Dagger concepts. Attributes are designed for tools building on CUE. They are not part of evaluation and can provided metadata for tool authors and users.

Attributes being considered:

• @input() - inputs to modules and libraries
• @output() - outputs from modules and libraries
• @widget() - metadata for presentation in UI / CLI
• @compute() - replace #compute as the required definition
• @artifact() - replace dagger.#Artifact as a required definition
• @debug() - futurology for BuildKit drop in debug sessions
  execution.

Motivation

The initial motivation was for marking inputs. Currently, if a user forgets to set an input, they do not find out until late in Daggers execution, at the point they are used. We would like to be able to detect missing inputs early (before BuildKit) and error, alerting to the user as such.

Types of inputs:

• secrets (apikeys)
• config (domain.com)
• nested, subselection(?)
• required, optional, defaults
• string, number, json/yaml, cue, (format / data type)

The broader motivation is to solidify the UX / DX for both users and library authors, as well as create a means to auto-generate presentation and other interfaces in the UI / CLI.

Implementation

CUE attributes will be used to mark Dagger concepts. We may be able to encode metadata (extra data needed by Dagger to process fields with attributes) using disjunctions and definitions (TBD).

A first implementation will likely focus on inputs, outputs, widgets, and presentation.

web: {
  siteID: string @input(help="Netlify site ID") @widget()
  deployURL: string @output(help="Netlify deployment URL") @widget(read-only)
  #deploy: [ < llb stuff > ] @compute()
}

Questions:

• Can we use a disjunction so that scalar fields can be extended with metadata for presentation?
• How can a user override default presentation provided by a library author?
• Can overrides span many levels of nesting?
• Will the disjunctions impact Dagger performance near-term (from known CUE perf issues)?
• Do we need to limit attribute detection depth? (no, hopefully)
• How and where do the input discovery and checking interact with preserved (dagger input ...) and adhoc (dagger compute --input-* ...) input values?
• How do we handle values which are both inputs and outputs? (shows up with default values for computed fields)

(alternative implementations were considered, they can be found in the edit history of this textarea. Several were discarded for UX, this proposal is trying to merge the remaining 2)

Terms

• user - a consumer of a Dagger module, the person running dagger to get something done, they specify input values
• developer - an author of a Dagger module, the person defining the inputs
• module - the top level CUE provided as the argument to Dagger
• library - CUE code that implements a reusable building block

User eXperience (UX):

• A user should be able to list the inputs needed to execute the current "spec"
• A user should be able to CRUD common inputs for all subsequent calls in the context
• A user should be able to see current, default, and missing values when listing needed inputs
• A user should get early feedback about missing inputs

Developer eXperience (DX):

• A module or library developer needs to specify values that are undefined at time of writing (both required / optional)
• are these only for the main module, can libraries have inputs (config?) (yes)
• can a library also be a main module? (yes)

Other questions:

Do outputs deserve some treatment as well? (would a user or developer like to know what is returned) (yes)

Related CUE issues, discussions, and CLs

• [POC] Dagger Inputs #225 (working branch for POC)
• Walking CUE Values cuelang/cue#862 (walking CUE values across definitions)
• How do attributes propagate through structs? cuelang/cue#888 (attribute propagation)

[shykes]

Developer eXperience (DX):

• are these only for the main module, can libraries have inputs (config?)

Yes, a library can specify inputs as well. The tool is responsible for presenting them all to the user, and clarifying which input goes where.

• can a library also be a main module?

Yes unless we find a specific technical reason why it can’t.

Do outputs deserve some treatment as well? (would a user or developer like to know what is returned)

Yes.

[shykes]

@input()
a small amount of upstream work needs to happen (may happen anyway)

What’s the nature of the upstream work needed?

[verdverm]

It's not a ton of work, @mpvl has already added the necessary helpers to make the more correct version viable

cuelang/cue#529

His comments / proposal are close to what I have written already. He thinks the needed changeset on that PR are minimal

[verdverm]

@shykes looks like it's getting worked on currently

https://cue-review.googlesource.com/c/cue/+/9201

[verdverm]

I'm partial to @input() having used attributes elsewhere. They support key/value pairs within the parens if we need extended details of "inputs" for inputs.

The main drawback is that they are not part of unification

[shykes]

At first glace, here’s how I feel about each option. Not a definitive answer!

No:

• #Input field
• Embed an input struct
• // comments on fields
• #Proxy

Maybe:

• @input() field attributes
• #Wrapper

[aluzzardi]

I feel the same about the different approaches: @input / #Wrapper (as an alternative to @input) feel less intrusive. Could work with "regular" Cue, whereas #Input / #Proxy requires customizing the Cue structure. // comments seems like a workaround for annotations.

[shykes]

To bring back concepts from the first dagger mockups: instead of inputs and outputs, we should use terminology that clarifies that these fields are user-facing.

Any non-concrete field in a dagger config is technically either an input (if its value contains no reference) or an output (if its value contains a reference). These inputs and outputs could be detected without additional annotation. But not all of them are meant to be presented to the user. To identify those user-facing fields is what requires annotations.

So really these annotations are presentation metadata: what UI widget do we use to render this field to the user? Possible types include: text label, test field, checkbox, dropdown select, multi-select.. This could be explicit in an attribute, or inferred. The widgets are bundled together in control panels.

Perhaps there are 2 standardized panel:

• a deployment panel for dagger users
• an admin panel for dagger admins and developers
  Panel assignment is an attribute.

To be continued :)

[verdverm]

They start with underscores, both regular and defs _a and _#b. They are not required for concreteness and are not visible outside of a package

https://cuelang.org/docs/tutorials/tour/references/hidden/

https://cuelang.org/docs/references/spec/#exported-identifiers

[shykes]

Oh I thought those had been deprecated! I just use aliases, seems pretty similar in purpose.

[verdverm]

Aliases have a more local scope, to the scope they are defined in.

The Proxy example above uses a hidden def for implementation and a regular def as the "input interface"

[verdverm]

cuelang/cue#699 (comment)

This is the best writeup about it all

[aluzzardi]

To bring back concepts from the first dagger mockups: instead of inputs and outputs, we should use terminology that clarifies that these fields are user-facing.

👍

So really these annotations are presentation metadata: what UI widget do we use to render this field to the user? Possible types include: text label, test field, checkbox, dropdown select, multi-select.. This could be explicit in an attribute, or inferred. The widgets are bundled together in control panels.

I think it would be more powerful/flexible to separate data and presentation. The UI layer has more context on how to best render an input, e.g. a boolean input can be rendered as a checkbox in HTML, a toggle switch on iOS, a boolean in a config file, a flag in a CLI program, ...

It also leaves more wiggle room for the UI to decide which "widget" to use (e.g. if there many options use a multi-select, if there's only 1-3 use a series of checkboxes). And domain specific decisions: on a responsive design, perhaps when rendering on a mobile screen you want to use a multi-select rather than a series of checkboxes because of lack of screen real estate. The UI can improve without having to patch the underlying Cue libraries.

I believe data/presentation split is the current trend, reminds me of WordPress themes which mixed data and presentation (e.g. "use this CSS for <input>"), versus headless CMS (e.g. Gatsby, Strapi) approach "here's the data and metadata render however you please".

[aluzzardi]

#Wrapper
hmm, need to remember what defined this one or made it special...

I think #Wrapper was meant as an alternative to using @attributes, e.g. dagger.#Input & { ... }, if we decide we don't want to expose another concept to the user. No idea what the #Input definition would entail.

[verdverm]

If I am understanding this correctly

• #Wrapper would be per field (like @attrs())
• #Input would be a struct with all inputs