Nix and/or Dagger

[nat-418]

I already use Nix to manage my software dependencies, create development environments, build my software, and create OCI containers. After listening to Solomon speak on the Changelog podcast, I am not sure how or if Dagger could fit into my workflow. My CI/CD pipelines are currently in GitHub Actions and GitLab CI. All they do is say "use this image and run that test". All the software is already prebuilt etc. in the container. How can Dagger improve this? Will it play nicely with Nix, or not?

[gerhard]

👋 @nat-418

To answer your first question: would non-Nix users be able to run your workflows the same way you do without installing Nix?

In my experience, Nix on Mac would stop working after every reboot. I tried various suggestions & stuck with it for several months in late 2021, but in the end I went back to brew. After that experience, I personally cannot recommend Nix on Mac.

As for your second question, Dagger works on NixOS (and with Nix) similar to how Docker does. While there is no Dagger Nix SDK, it wouldn't be too difficult to implement it. We actually spoke with @tazjin about this idea some months after https://changelog.com/shipit/37 & a few months before our other SDKs came out:

• 🐹 https://dagger.io/blog/go-sdk
• 🐍 https://dagger.io/blog/python-sdk
• ⬢ https://dagger.io/blog/nodejs-sdk

While my Nix on Mac experience was not great, I have been running a dedicated NixOS host since 2023.03. It's been a mostly positive experience, and while I still didn't get past the Nix learning curve, I am running all my homelab stuff on that host (including Dagger Engine, Docker, Grafana, Prometheus, etc.). I am curious to experience Pop OS next, on the same hardware.

FWIW, my other bare metal Linux host is Talos OS.

And while I get @mitchellh's passion & mastery of Nix - https://mitchellh.com/writing/nix-with-dockerfiles - and I suspect that you share some of that too, most of the people that I speak to just want to use their programming language + preferred code editor for CI/CD.

I see Dagger as a good choice for everyone that doesn't want to learn another DSL, and/or wants to keep YAML at a minimum.

Do you have some GitHub workflows to share @nat-418? What about some Nix code? I am curious to see what that looks like in practice.

---

cc @vito. There is some cool Nix here: https://github.com/vito/bass

[nat-418]

Thanks for the reply @gerhard. The code I am talking about is in a proprietary GitLab, but I will try and give a sense of it.

Consider for this example a Python application. there is a deps.nix file in the project's root directory that specifies the various build tools, libraries, etc. required for the project. There is a shell.nix that imports from deps.nix and allows for the emphemeral virtual dev/testing environment. There is also a docker.nix that imports from deps.nix and specifies how to build an OCI container.

# deps.nix
{ nixpkgs ? import <nixpkgs> {} }:

let
  pyPi = { name, version, hash, deps }:
    nixpkgs.python3Packages.buildPythonPackage {
      pname = name;
      version = version;
      src = nixpkgs.python3Packages.fetchPypi {
        pname = name;
        version = version;
        sha256 = hash;
      };
      doCheck = false;
      propagatedBuildInputs = deps;
    };
in
{
  generalDependencies = with nixpkgs; [
    git-cliff
    pre-commit
    ruff
  ];

  pythonDependencies = with nixpkgs.python3Packages; [
    black
    mypy
    pdoc
    (pyPi {
      name = "xdoctest";
      version = "1.1.1";
      hash = "sha256-LqyBMb3N8ngbTlpi1t6H8ES3MMyNuK8UKlG7KcJF53k=";
      deps = [ six ];
    })
    (pyPi {
      name = "code2flow";
      version = "2.5.1";
      hash = "sha256-rZhxeSDaZZjaw4oZUfOLhIkfSqS9LzbBGbDbrHKfR4U=";
      deps = [
        nixpkgs.graphviz
        nixpkgs.python3Packages.graphviz
      ];
    })
  ];
}

# shell.nix
{ pkgs ? import <nixpkgs> {}, deps ? import ./deps.nix {}}:

pkgs.mkShell {
  packages = [
    deps.generalDependencies
    (pkgs.python3.withPackages (x: deps.pythonDependencies))
  ];
}

# docker.nix
# Note: this produces < 200mb docker image without any upstream FROM
{ pkgs ? import <nixpkgs> {}, deps ? import ./deps.nix {} }:

with pkgs;

dockerTools.buildImage {
  name = "example/foo";
  tag = "latest";
  created = "now";
  copyToRoot = buildEnv {
    name = "image-root";
    paths = deps.generalDependencies ++ deps.pythonDependencies ++ [
      bash
      coreutils
    ];
    pathsToLink = [
      "/bin"
    ];
  };
  config = {
    # Provides compatibility with GitLab CI/CD runners
    Entrypoint = [ "${bash}/bin/bash" "-c" ];
  };
}

#!/bin/sh
# example shell script to be called by GitLab CI
ruff .

# Example GitLab CI
type-check:
  image:
    name: example/foo
  script:
    - ./bin/type-check.sh

lint:
  image:
    name: example/foo
  script:
    - ./bin/lint.sh

run-tests:
  image:
    name: example/foo
  script:
    - ./bin/run-tests.sh

What I currently do is use Nix to build that OCI container and push it to my company's container registry, where GitLab can then pull it and run the CI jobs. The jobs themselves are just shell scripts, calling ruff, mypy, etc. on the Python codebase. This is the only part of my DevOps workflow that isn't integrated into Nix, and I was wondering if this is where Dagger could/should go?

Personally, I think the Nix language and ecosystem is purpose-built for dealing with the particular complexities of building and configuring software—and I prefer that mature, clean approach over trying to wire up the same thing in Go or whatever.

I don't use macOS so I can't comment on the macOS issues you were having.

[helderco]

Hooohh... I've been wondering lately about trying Nix to replace Homebrew (mostly), and conda (only use it for managing venvs and python versions) in macOS. Reading this makes me pause.

@nat-418, can you have Nix in a container and run from there? A bit like the dockerfile I see here: https://mitchellh.com/writing/nix-with-dockerfiles, but in dagger. This way a team member that doesn't even need to have Nix installed would get the same builds, and CI as well.

And thanks for sharing how you manage a Python project with Nix! 🙌

[nat-418]

@nat-418, can you have Nix in a container and run from there?

There is an official nixos/nix Docker image that you could use with a volume mount etc.

If there is a way to run macOS on KVM/QEMU I am willing to try and rig up a demo to help you and Gerhard get Nix working on macOS.

[vito]

It's true that right now Dagger is mostly used as a build + test tool, so if you've already bought into Nix it probably seems a bit redundant, since that's also Nix's bread + butter.

I could ramble about this for ages, but I think the tl;dr is that Nix is mostly focused on building things, while Dagger is interested in a more generic approach that can be applied to every stage of the app development lifecycle:

1. bootstrapping the dev environment
2. building the app
3. running all of its test suites and misc checks/validations
4. running containers in those test suites (testcontainers style)
5. running the app locally during development, including dependent services
6. arbitrary scripts and automation, replacing bash
7. and of course, CI/CD

Nix covers a lot of this, but it comes with some strong opinions and a steep learning curve.

Dagger is essentially a specialized tool for running cached, lazily-evaluated containers and content-addressed services. It blurs the line between build + run, so you just think about code most of the time, and what you can do with it. As an example of things you can do with this specialized tool, you can implement Docker Compose in 199 lines of Go. I could probably implement a decent CI system, or re-implement Concourse. Lots of people could, in their preferred language.

As for how they relate to one another: beneath all of this, ideally you are using a reproducible toolchain. Dagger encourages reproducible builds and has APIs built expressly to support it (WithTimestamps), but it doesn't enforce it. In my experience a lot of that comes down to tackling the base container image, which Nix is great at building. But I also have the option of using something like Wolfi or just YOLOing apt-get until I care enough.

Anyway, that's all my 2c; I'm a big fan of Nix and use it as my daily OS. 😄

[nat-418]

Thanks for clarifying.

Nix covers a lot of this, but it comes with some strong opinions and a steep learning curve.

I haven't used Dagger, but based on Solomon's Changelog interview, it seems that by switching from Q (cue?) / YAML++ to just whatever programming language you are gonna need CI for your CI etc. I honestly don't think learning Nix is very hard, and the beauty of it is that you don't need to reinvent the wheel.

I'll keep an eye on Dagger, but for now I guess I am not the target market.