Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Make Kubernetes default resources configurable #1800

Merged
merged 5 commits into from
May 6, 2024
Merged

Make Kubernetes default resources configurable #1800

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
6b75357
Make Kubernetes default resources configurable
martinhausio Apr 11, 2024
7c6b7fa
use proper key value parsing
martinhausio Apr 11, 2024
63f27e7
refine to separate resources environment variables
martinhausio May 2, 2024
bae650c
validate if values are set in decorators
martinhausio May 4, 2024
cbd618d
change from empty string to None
martinhausio May 4, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
4 changes: 4 additions & 0 deletions metaflow/metaflow_config.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -314,6 +314,10 @@
KUBERNETES_SHARED_MEMORY = from_conf("KUBERNETES_SHARED_MEMORY", None)
# Default port number to open on the pods
KUBERNETES_PORT = from_conf("KUBERNETES_PORT", None)
# Default kubernetes resource requests for CPU, memory and disk
KUBERNETES_CPU = from_conf("KUBERNETES_CPU", None)
KUBERNETES_MEMORY = from_conf("KUBERNETES_MEMORY", None)
KUBERNETES_DISK = from_conf("KUBERNETES_DISK", None)

ARGO_WORKFLOWS_KUBERNETES_SECRETS = from_conf("ARGO_WORKFLOWS_KUBERNETES_SECRETS", "")
ARGO_WORKFLOWS_ENV_VARS_TO_SKIP = from_conf("ARGO_WORKFLOWS_ENV_VARS_TO_SKIP", "")
Expand Down
12 changes: 12 additions & 0 deletions metaflow/plugins/kubernetes/kubernetes_decorator.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,9 @@
KUBERNETES_SERVICE_ACCOUNT,
KUBERNETES_SHARED_MEMORY,
KUBERNETES_PORT,
KUBERNETES_CPU,
KUBERNETES_MEMORY,
KUBERNETES_DISK,
)
from metaflow.plugins.resources_decorator import ResourcesDecorator
from metaflow.plugins.timeout_decorator import get_run_time_limit_for_task
Expand Down Expand Up @@ -174,6 +177,15 @@ def __init__(self, attributes=None, statically_defined=False):
except (NameError, ImportError):
pass

# parse the CPU, memory, disk, values from the KUBERNETES_ environment variable (you would need to export the METAFLOW_KUBERNETES_CPU, METAFLOW_KUBERNETES_MEMORY and/or METAFLOW_KUBERNTES_DISK environment variable with the desired values before running the flow)
# find the values from the environment variables, then validate if the values are still the default ones, if so, then replace them with the values from the environment variables (otherwise, keep the values from the decorator)
if self.attributes["cpu"] == self.defaults["cpu"] and KUBERNETES_CPU:
self.attributes["cpu"] = KUBERNETES_CPU
if self.attributes["memory"] == self.defaults["memory"] and KUBERNETES_MEMORY:
self.attributes["memory"] = KUBERNETES_MEMORY
if self.attributes["disk"] == self.defaults["disk"] and KUBERNETES_DISK:
self.attributes["disk"] = KUBERNETES_DISK

# If no docker image is explicitly specified, impute a default image.
if not self.attributes["image"]:
# If metaflow-config specifies a docker image, just use that.
Expand Down