Explicit dependency between actions

[shykes]

Overview

This proposes a design in Dagger 0.2 to solve #940.

Problem

Most of the time, dependencies between Dagger tasks are implicit, thanks to the magic of CUE references. If a config value for task foo is referenced by a config value for task bar, Dagger implicitely knows that foo must run before bar.

But sometimes, foo must run before bar even though there is no dependency between their configurations. For example, foo might upload an artifact at a known URL, which bar must later download. In this case, developers need a way to control dependencies between tasks explicitly.

Currently this is achieved by various hacks (cc @grouville who is carrying the oppressive weight of these soul-destroying hacks on his shoulders).

Solution

Instead of hacks, there should be a reliable and intuitive API to achieve this with minimal effort. There are 3 design options being considered:

	description	Synchronization primitive	Wait for one task	Wait for multiple tasks	Synchronize with high-level actions	Works with cue/flow
Option 1	#Wait + id	Explicit calls to engine	engine.#Wait & { target: foo , completed: { …}}	engine.#Wait & { targets: [foo, bar] }	High-level action must expose id: #ID (can be #Wait on multiple sub-tasks)	Yes
Option 2	if + status	CUE control flow + CUE computed values	if (foo.status == “completed”) { … }	if (foo.status == “completed”) && (bar.status == “completed”) { … }	High-level action must expose status: #Status (can be CUE expression combining status of multiple sub-tasks)	Maybe?
Option 3	Salt-style convention	Engine scans the tree for special fields	require: foo	require: [foo, bar]	High-level action must implement one or more of require, require_any, onfail, onfail_any	No (requires patching cue/flow to confer special meaning to Salt-style convention)

See below for details on each option.

Option 1: #Wait + id convention

Spec:

package engine

// Wait for one or more tasks to complete
#Wait: {
  {
    target: #TaskID
  } | {
    targets: […#TaskID]
  }
  // Execute a task when all targets have completed
  completed?: #Task
  // Execute a task when all targets have started
  started?: #Task
}

// Match any engine task
#Task: {
  id: #TaskID
}

#TaskID: string

Example usage:

sleepShort: engine.#Exec & {
  args: [“sleep”, “1”]
}

sleepLong: engine.#Exec & {
  args: [“sleep”, “10”]
}

message: engine.#Wait & {
  targets: [sleepShort.id, sleepLong.id]
  completed: engine.#Exec & {
    args: [“echo”, “done!”]
  }
}

User-defined actions can also be waited on: they simply need to implement the id: #TaskID interface.

For example docker.#Run would do:

package docker

#Run: {
  id: _exec.id
  // …
  _exec: engine.#Exec & {
    // …
  }
}

Option 2: if + status convention

This option requires the CUE if statement to be a reliable synchronization primitive.

All tasks have a status field filled by the runtime:

package engine

// The status of a task
#Status: “running” | “completed” | “failed” | “cancelled”

#Exec: {
  status: #Status
  …
}

#Pull: {
  status: #Status
  …
}
// etc.

Then waiting for a task is as simple as:

sleepShort: engine.#Exec & {
  args: [“sleep”, “1”]
}

sleepLong: engine.#Exec & {
  args: [“sleep”, “10”]
}

if sleepShort.status == “completed” && sleepLong.status == “completed” {
  completed: engine.#Exec & {
    args: [“echo”, “done!”]
  }
}

High-level actions could also be waited on, by implementing the status: engine.#Status convention. For example:

package docker

#Run: {
  status: _exec.status
  _exec: engine.#Exec & {
    …
  }
}

Option 3: Salt-style convention

Copied from @helderco’s comment below

• require: Requires that a list of target tasks succeed before execution
• require_in: Reverse dependency, is this even possible? Would be awesome if so.
• require_any: What if you need an OR logic?
• onfail: Execute only if a target task fails
• onfail_in
• onfail_any

As a global, every task would have to follow a convention, but looks cleaner when used (no need for wrapping):

package engine

#Task: {
    id: #TaskID
    #Requisites
    ...
} 

#TaskID: string

#Requisites: {
    require: [...#Task]
    require_any: [...#Task]
    onfail: [...#Task]
    onfail_any: [...#Task]
    ...
}

#Exec: #Task & {
    args: ...
    ....
}

Example usage:

sleepShort: engine.#Exec & {
  args: [“sleep”, “1”]
}

sleepLong: engine.#Exec & {
  args: [“sleep”, “10”]
}

engine.#Exec & {
    args: [“echo”, “done!”]
    require: [sleepShort, sleepLong]
}

And in user-land:

package docker

#Run: #Requisites & {
    id: _exec.id
    require: _
    // …
    _exec: engine.#Exec & {
        // …
        "require": require
    }
}

Copying all requisites like this doesn't look good, but maybe there's a simpler way to implement?

[aluzzardi]

@shykes The problem @grouville is facing (if I understand correctly) is: how does this work with higher level packages?

Let's say you want yarn.#Package to wait for a docker.#Run to complete.

The problem in dagger mainstream is you have no idea which fields are the output of tasks (e.g. can be used as completed field) and which ones are inputs of tasks, since you're working with an abstraction (multi layered at that, e.g. docker.#Run -> docker.#Command -> os.#Container -> actual task).

It's worse when a package contains a set of tasks. How do you wait on all of them to complete?

[shykes]

@shykes The problem @grouville is facing (if I understand correctly) is: how does this work with higher level packages?

Let's say you want yarn.#Package to wait for a docker.#Run to complete.

The problem in dagger mainstream is you have no idea which fields are the output of tasks (e.g. can be used as completed field) and which ones are inputs of tasks, since you're working with an abstraction (multi layered at that, e.g. docker.#Run -> docker.#Command -> os.#Container -> actual task).

I see. What if we made it a convention to expose an id: #TaskID field for high-level actions? If you don’t, everything still works but developers can’t use task synchronization primitives against your action: wait for completion, terminate, etc.

It's worse when a package contains a set of tasks. How do you wait on all of them to complete?

@helderco had the exact same request :)

I will update the design proposal based on this, thanks!

[helderco]

Most of the time, dependencies between Dagger tasks are implicit, thanks to the magic of CUE references. If a config value for task foo is referenced by a config value for task bar, Dagger implicitely knows that foo must run before bar.

This makes me think any reference is valid.

The problem in dagger mainstream is you have no idea which fields are the output of tasks (e.g. can be used as completed field) and which ones are inputs of tasks, since you're working with an abstraction (multi layered at that, e.g. docker.#Run -> docker.#Command -> os.#Container -> actual task).

But this says it's not.

How does the go implementation know to follow a CUE reference as a dependency? Do tasks need to reference other tasks somehow (through the higher level abstractions)?

[helderco]

Reading on cue flow...

[helderco]

At first glance I like how the if option reads for a simple case, but soon I hit some limitations.

• More verbose to check for status;
• I may have a dynamic set of tasks (template) so I'd need to make a list and filter somehow in the condition;

Taking inspiration from Salt, which I'm used to, every state has a name (aka id) and some global requisites. These are powerful, here's some I like:

• require: Requires that a list of target tasks succeed before execution
• require_in: Reverse dependency, is this even possible? Would be awesome if so.
• require_any: What if you need an OR logic?
• onfail: Execute only if a target task fails
• onfail_in
• onfail_any

As a global, every task would have to follow a convention, but looks cleaner when used (no need for wrapping):

package engine

#Task: {
    id: #TaskID
    #Requisites
    ...
} 

#TaskID: string

#Requisites: {
    require: [...#Task]
    require_any: [...#Task]
    onfail: [...#Task]
    onfail_any: [...#Task]
    ...
}

#Exec: #Task & {
    args: ...
    ....
}

Example usage:

sleepShort: engine.#Exec & {
  args: [“sleep”, “1”]
}

sleepLong: engine.#Exec & {
  args: [“sleep”, “10”]
}

engine.#Exec & {
    args: [“echo”, “done!”]
    require: [sleepShort, sleepLong]
}

And in user-land:

package docker

#Run: #Requisites & {
    id: _exec.id
    require: _
    // …
    _exec: engine.#Exec & {
        // …
        "require": require
    }
}

Copying all requisites like this doesn't look good, but maybe there's a simpler way to implement?

So... for me, either this (option 3?) or option 1, with some notes:

• I prefer a single targets, even if with only one element;
• Can we list #Task instead of #TaskID (making the access to the id field implicit)?
• The need for wrapping (i.e., #Wait.completed) feels restrictive, but that's just a gut reaction. I assume #Exec has to be inside #Wait for you to control execution;

[grouville]

I don't have a strong opinion on all these alternatives, but I like the UX of the third option (@helderco's one) the most (even though his current example isn't fully functional atm [theoritically]). From what I understand, helderco's proposition might be possible with your Option 1: id convention.

The second option seems more "risky" because it relies on Cue to expose subfields in the scope of the if (which led to some issues in the past). @helderco's comment is also relevant, it may become very heavy in case of 3+ dependencies involved.

Having a require field is pretty neat and understandable :

sleepShort: engine.#Exec & {
  args: [“sleep”, “1”]
}

sleepLong: engine.#Exec & {
  args: [“sleep”, “10”]
}

engine.#Exec & {
    args: [“echo”, “done!”]
    require: [sleepShort.id, sleepLong.id]
}

Btw, these directives are indeed very powerful and could bring a lot of potential use-cases. They may all fit your id convention.

require: Requires that a list of target tasks succeed before execution
require_in: Reverse dependency, is this even possible? Would be awesome if so.
require_any: What if you need an OR logic?
onfail: Execute only if a target task fails
onfail_in
onfail_any

Curious to know your opinion on the subject

[samalba]

Duplicate of #940

[shykes]

Picking this up.

• I updated the top comment, with a summary table of our 3 options.
• Per discussion with @jlongtine : it’s not clear which options work out of the box within the limitations of cue/flow. It’s possible that none do. There is a chance that option 1 (#Wait) does work if designed correctly: Joel is testing this now
• Overall my favorite candidate is Option 1 at the moment.
  • An explicit engine API call remains the cleanest way to synchronize IMO. Since it’s the same core primitive as everything else, it’s familiar to Dagger developers, fits in the same documentation system, can be extended and/or deprecated in the same way as all other calls, etc.
  • All options require implementing a field convention in high-level actions, so that they can be synchronized in the same way as low-level tasks. Option 1 is better in this dimension, because it only requires adding a single field. The Salt-style convention of Option 3 as a larger footprint, and if we want to implement richer synchronization primitives, the only way to do that is extend the footprint further. By contrast, Option 1 can be enriched by adding more engine API calls: the convention remains the same: just add id.
  • Option 2 is out, for several reasons. Using CUE control flow is risky, it gets quite verbose when waiting for multiple tasks. Nobody seems to love it.

[lyzhang1999]

I'm look forward to this new feature

[dolanor]

Engineering wise, the option 1 makes more sense, but I don't really like the syntax/implementation presented.
I like option 3 better in a usage point of view. I would have thought it could work out of the box with CUE, but thinking about it, I guess CUE will eval the dependency correctly, but it would have 0 idea of the state of the dependency, only the runtime/engine would.
I don't have any pointer for now as I'm still discovering all the concepts as well.

[shykes]

@aluzzardi you mentioned a possible core.#Pipeline or core.#Sequence recently. Could that be an alternative to the other options mentioned here? wdyt?