feature: on demand docker environments #1796
Conversation
…anges to conda_environment
|
|
| # On Demand Docker image build configuration | ||
| ### | ||
| # Image builder service url | ||
| DOCKER_IMAGE_BAKERY_URL = from_conf("DOCKER_IMAGE_BAKERY_URL", None) |
There was a problem hiding this comment.
nit - change this to FAST_BAKERY_URL? and similar changes below?
| def init_environment(self, echo): | ||
| if self._setup_conda_fallback(): | ||
| print( | ||
| "Some steps would execute locally. Had to fallback to a conda environment" |
There was a problem hiding this comment.
we can skip the print statement for now? ideally, we can have a debug flag that will print out what kind of environment is being created for which step. we can make it a follow up with a TODO in the code.
| return True | ||
|
|
||
| def init_environment(self, echo): | ||
| if self._setup_conda_fallback(): |
There was a problem hiding this comment.
it seems that we fall back to conda environments for all steps if any step is running locally. can we make it such that any step that is marked to run on kubernetes can benefit from fast bakery?
| if self._setup_conda_fallback(): | ||
| print( | ||
| "Some steps would execute locally. Had to fallback to a conda environment" | ||
| ) |
There was a problem hiding this comment.
a flow that has a single step marked with @kubernetes, fails right now
Some steps would execute locally. Had to fallback to a conda environment
Bootstrapping virtual environment(s) ...
Virtual environment(s) bootstrapped!
2024-05-02 08:48:39.662 Workflow starting (run-id 217204):
2024-05-02 08:48:42.651 [217204/start/1333327 (pid 68470)] Task is starting.
2024-05-02 08:48:47.021 [217204/start/1333327 (pid 68470)] [pod t-f10eeb12-c9h8b-bv7gc] Task is starting (Pod is pending, Container is waiting - ContainerCreating)...
2024-05-02 08:48:46.955 [217204/start/1333327 (pid 68470)] [pod t-f10eeb12-c9h8b-bv7gc] Setting up task environment.
2024-05-02 08:48:56.265 [217204/start/1333327 (pid 68470)] [pod t-f10eeb12-c9h8b-bv7gc] Downloading code package...
2024-05-02 08:48:57.143 [217204/start/1333327 (pid 68470)] [pod t-f10eeb12-c9h8b-bv7gc] Code package downloaded.
2024-05-02 08:48:57.175 [217204/start/1333327 (pid 68470)] [pod t-f10eeb12-c9h8b-bv7gc] Task is starting.
2024-05-02 08:48:57.658 [217204/start/1333327 (pid 68470)] [pod t-f10eeb12-c9h8b-bv7gc] bash: line 1: /conda-prefix/bin/python: No such file or directory
2024-05-02 08:49:00.594 [217204/start/1333327 (pid 68470)] Kubernetes error:
2024-05-02 08:49:00.683 [217204/start/1333327 (pid 68470)] Error (exit code 127). This could be a transient error. Use @retry to retry.
2024-05-02 08:49:00.683 [217204/start/1333327 (pid 68470)]
2024-05-02 08:49:00.912 [217204/start/1333327 (pid 68470)] Task failed.
|
|
||
|
|
||
| class FastBakeryException(MetaflowException): | ||
| headline = "Docker Image Bakery ran into an exception" |
There was a problem hiding this comment.
minor nit - error messages can also reference fast bakery
Adds support for on demand Docker images. Open for discussion
open todo's
condaenvironment when executing locally with--environment docker