Zenith: How to use private dependencies with the Python SDK

[skycaptain]

I would like to utilize Dagger along with the Python SDK to configure my CI pipelines across multiple projects. In order to avoid unnecessary repetition, I intend to externalize certain functions into a shared Python library called my-private-library. However, since this code is intended to remain private, I opt to utilize a private PyPI registry. My pyproject.toml looks like this:

[project]
name = "main"
version = "1.0.0"
description = ""
dependencies = [
    "my-private-library",
]

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

[tool.hatch.build.targets.wheel]
packages = ["src/main.py"]

[tool.hatch.envs.default.env-vars]
PIP_INDEX_URL = "https://token:{env:GITLAB_API_TOKEN}@gitlab.com/api/v4/groups/<group1_path>/-/packages/pypi/simple/"

{env:GITLAB_API_TOKEN} is evaluated when the package is installed here: https://github.com/dagger/dagger/blob/8fd92e367c304694b01615dc95aa5467bb8d9ca1/sdk/python/runtime/main.go#L27C10-L27C61

As it is not currently possible to pass environment variables from the caller's host (see #6112), what alternative solutions could be utilized for using private dependencies?

[skycaptain]

Answering my own question: It seems that the sdk defined in dagger.json is also just a module (great design choice). Hence, it is possible to fully customize the runtime container, where my-private-library is installed in, e.g.:

{
  "name": "potato",
  "sdk": "../sdk/python/my-custom-runtime",
  "root": "../"
}

This way I can use a pre-built image that has all batteries included.

[helderco]

Yes, that's what I'd suggest for now, the problem I see with this is how do pass the secret value (for gitlab's token)? You'll have to cache that token into the runtime container in the my-cutom-runtime module as a normal env var or explicitly in the source code if you're using Dagger's native secrets. I'd suggest using an external secret store like infisical (website) but then you need the token for that service!

[helderco]

Btw, have you already tried doing the custom runtime? Does it work? A few weeks back I tried it quickly and remember that there was a piece missing to make it work but don't remember what. So good feedback if you got it to work or not 🙂

[skycaptain]

Thank you for your insights. At the moment, I am in the process of exploring Dagger modules to ascertain their suitability for our needs and to identify areas where adjustments to my CX libraries may be necessary. My use case is that I have a Python library so that I can reuse code (I know that is kind of contrary to the modules approach) in my CX pipelines in multiple projects with the CX pipelines being Dagger modules. The question now is, how do I include my library from a private repository when installing the Dagger module in the SDK runtime. My first thought was to just build a custom image, that has this library already installed, so no need to install it from the private repository and use that image instead of the default python:3 image in the SDK runtime.

My second though was - and please note that this is not yet production code and given our use of Gitlab CI, where secret management is somewhat compromised anyways due to just being global environment variables - to echo all secrets into a .env file in the modules root and parse this file prior to the module's installation. Although I haven't done thorough testing, it appears to be working: https://github.com/skycaptain/dagger/tree/python-sdk-runtime-with-env/sdk/python/runtime.

{
  "name": "potato",
  "sdk": "github.com/skycaptain/dagger/sdk/python/runtime@5cc76c6939e1fb09497ff1721b92f74bd38e367e",
  "root": "../",
  "exclude": ["**/.venv"]
}

In Gitlab CI

build:
  [...]
  script:
    - echo "PIP_INDEX_URL='$PIP_INDEX_URL'" > .env
    - dagger call ...

[skycaptain]

Just to let you know. One issue I encountered with the custom SDK is this:

49: exec /runtime
49: exec /runtime ERROR: secret core.Secret:eyJuYW1lIjoiQVJUSUZBQ1RPUllfVVNFUiJ9: not found

It occurs sporadically after a few runs of my pipeline, indicating a potential connection to caching. I will do further investigation.