engine: support for system proxy settings (#7255)

* engine: support for system proxy settings

Signed-off-by: Erik Sipsma <[email protected]>

* networking: remove no-hosts from dnsmasq conf

This setting meant that containers could not resolve names that were
resolvable in the engine (even if direct IPs were still usable).

While supporting that is incredibly obscure, the need has arisen in
integ tests for proxy support, where the test setup has:
* a proxy service
* a nested engine with dep on the proxy service
* containers running in that nested engine that automatically get proxy
  settings from their engine

In this case, being able to use the service binding alias for squid,
which is set in the engine's /etc/hosts, massively simplifies TLS
testing since the proxy's certs can all reference just that alias.

From poking around, it does not seem that this setting was load-bearing
for us and was just inherited from the original `dnsname` configuration
we picked up (and in turn `dnsname` had just set that from the beginning
with no context).

Looking around online for use cases that call for this setting in
dnsmasq, it seems to generally be obscure/complex networking setups
where entries in /etc/hosts end up pointing to IPs local to a
container/machine but they want nested clients to point to a different
IP.

My imagination is limited, but I cannot currently think of any situation
in which this would apply to engine containers; the networking settings
of the engine itself should dictate the networking settings of the
containers so any aliases found in the engine's /etc/hosts can apply to
resolving names queried by containers.
* To be clear: this does *not* result in containers' /etc/hosts having
  different values from before. It just makes all names resolvable by
  the engine also resolvable by containers, which is consistent with the
  behavior of DNS names outside of aliases.

Signed-off-by: Erik Sipsma <[email protected]>

* cleanup integ tests

The previous commit that tweaked dnsmasq conf enables just the squid
alias to be used for the proxy endpoint, which massively simplifies the
rest of the test.

Signed-off-by: Erik Sipsma <[email protected]>

* add integ tests that use basic auth for proxy

Signed-off-by: Erik Sipsma <[email protected]>

* add integ tests for NO_PROXY

Signed-off-by: Erik Sipsma <[email protected]>

* add support for GOPROXY setting on the Go SDK

This is implemented in such a way that it's a bit more generic but
hidden from external clients for now.

The overall plumbing is:
1. A hidden (via `__` prefix) API on Container for specifying env vars
   that should be inherited from the engine container, where they must
   be prefixed with `_DAGGER_ENGINE_SYSTEMENV_` (e.g.
   `_DAGGER_ENGINE_SYSTEMENV_GOPROXY`).
2. The Container implementation sets metadata that the Executor needs
   via LLB *metadata*. Importantly, this is very different from the
   previous `ftp_proxy` hack in that this is just metadata and thus
   doesn't impact LLB digests. We specifically use the `Description` LLB
   metadata, which is explicitly doc'd as never being interpreted by the
   solver and thus safe to use for this.
3. The Worker's ResolveOp method registers the solver.Vertex (which
   includes that metadata) so that it can be read by the Executor as
   needed.
4. The Executor uses that metadata to set any requested env vars from
   the system.

Once we've fleshed out the design of support for these system env vars
we can open up the API.

This plumbing of executorMetadata is also intended to be easy to re-use
for any other needs that arise in the future similar to this.

Integ tests are also included here with a pretty nice GOPROXY
implementation usable a library.

Signed-off-by: Erik Sipsma <[email protected]>

* chore: tidy go.mod

Signed-off-by: Justin Chadwell <[email protected]>

* add clarifying comments to proxy env setting code

Signed-off-by: Erik Sipsma <[email protected]>

---------

Signed-off-by: Erik Sipsma <[email protected]>
Signed-off-by: Justin Chadwell <[email protected]>
Co-authored-by: Justin Chadwell <[email protected]>

core/container.go

@@ -93,6 +93,10 @@ type Container struct {
 
  // The args to invoke when using the terminal api on this container.
  DefaultTerminalCmd DefaultTerminalCmdOpts `json:"defaultTerminalCmd,omitempty"`
+
+ // (Internal-only for now) Environment variables from the engine container, prefixed
+ // with a special value, that will be inherited by this container if set.
+ SystemEnvNames []string `json:"system_envs,omitempty"`
 }
 
 func (*Container) Type() *ast.Type {
@@ -157,6 +161,7 @@ func (container *Container) Clone() *Container {
  cp.Sockets = cloneSlice(cp.Sockets)
  cp.Ports = cloneSlice(cp.Ports)
  cp.Services = cloneSlice(cp.Services)
+ cp.SystemEnvNames = cloneSlice(cp.SystemEnvNames)
  return &cp
 }
 
@@ -1214,14 +1219,25 @@ func (container *Container) WithExec(ctx context.Context, opts ContainerExecOpts
 
  execSt := fsSt.Run(runOpts...)
 
- execDef, err := execSt.Root().Marshal(ctx, llb.Platform(platform.Spec()))
+ execMD := buildkit.ExecutionMetadata{
+ SystemEnvNames: container.SystemEnvNames,
+ }
+ execMDOpt, err := execMD.AsConstraintsOpt()
+ if err != nil {
+ return nil, fmt.Errorf("execution metadata: %w", err)
+ }
+ marshalOpts := []llb.ConstraintsOpt{
+ llb.Platform(platform.Spec()),
+ execMDOpt,
+ }
+ execDef, err := execSt.Root().Marshal(ctx, marshalOpts...)
  if err != nil {
  return nil, fmt.Errorf("marshal root: %w", err)
  }
 
  container.FS = execDef.ToPB()
 
- metaDef, err := execSt.GetMount(buildkit.MetaMountDestPath).Marshal(ctx, llb.Platform(platform.Spec()))
+ metaDef, err := execSt.GetMount(buildkit.MetaMountDestPath).Marshal(ctx, marshalOpts...)
  if err != nil {
  return nil, fmt.Errorf("get meta mount: %w", err)
  }
@@ -1236,7 +1252,7 @@ func (container *Container) WithExec(ctx context.Context, opts ContainerExecOpts
  mountSt := execSt.GetMount(mnt.Target)
 
  // propagate any changes to regular mounts to subsequent containers
- execMountDef, err := mountSt.Marshal(ctx, llb.Platform(platform.Spec()))
+ execMountDef, err := mountSt.Marshal(ctx, marshalOpts...)
  if err != nil {
  return nil, fmt.Errorf("propagate %s: %w", mnt.Target, err)
  }