Releases: Netflix/metaflow
2.11.15
Features
Displaying task attempt logs
When running a task with the @retry
decorator, previously we were able to only view the logs of the latest attempt of the task.
With this release it is now possible to target a specific attempt with the --attempt
option for the logs command
python example.py logs 123/retry_step/456 --attempt 2
Scrubbing log contents
This release introduces a new command for scrubbing log contents of tasks in case they contain sensitive information that needs to be redacted.
Simplest use case is scrubbing the latest task logs. By default both stdout
and stderr
are scrubbed
python example.py logs scrub 123/example/456
There are also options to target only a specific log stream
python example.py logs scrub 123/example/456 --stderr
python example.py logs scrub 123/example/456 --stdout
when using the@retry
decorator, tasks can have multiple attempts with separate logs that require scrubbing. By default only the latest attempt is scrubbed. There are options to make scrubbing multiple attempts easier
# scrub specific attempt
python example.py logs scrub 123/retry_step/456 --attempt 1
# scrub all attempts
python example.py logs scrub 123/retry_step/456 --all
# scrub specified attempt and all prior to it (this would scrub attempts 0,1,2,3)
python example.py logs scrub 123/retry_step/456 --all --attempt 3
The command also accepts only specifying a step for scrubbing. This is useful for steps with multiple tasks, like a foreach split.
python example.py logs scrub 123/foreach_step
all the above options also apply when targeting a step for scrubbing.
Note: Log scrubbing for running tasks is not recommended, and is actively protected against. There can be occasions where a task has failed in such a way that it still counts as not completed. For such a case you can supply the --include-not-done
option to try and scrub it as well.
What's Changed
- feature: scrub logs by @saikonen in #1802
- Fix issue #1805 by @narayanacharya6 in #1807
- bug fix: unused arg for batch step --help by @mae5357 in #1817
- bump version to 2.11.15 by @saikonen in #1829
New Contributors
- @narayanacharya6 made their first contribution in #1807
- @mae5357 made their first contribution in #1817
Full Changelog: 2.11.14...2.11.15
2.11.14
What's Changed
- Increase Azure Blobstore connection_timeout. by @shrinandj in #1827
- bump version to 2.11.14 by @shrinandj in #1828
Full Changelog: 2.11.13...2.11.14
2.11.13
Features
Configurable default Kubernetes resources
This release introduces configuration options for setting default values for cpu / memory / disk
when running on Kubernetes. These can be set either with environment variables
METAFLOW_KUBERNETES_CPU=
METAFLOW_KUBERNETES_MEMORY=
METAFLOW_KUBERNETES_DISK=
or in a Metaflow profile
{
"KUBERNETES_CPU": "",
"KUBERNETES_MEMORY": "",
"KUBERNETES_DISK": "",
}
These values will be overruled by specifying a value through the @kubernetes
or @resources
decorators.
Improvements
Support for wider foreach flows with Argo Workflows
This release changes the way task ids are generated on Argo Workflows in order to solve an issue where extremely wide foreach splits could not execute correctly due to hard limits on input parameters size on Argo Workflows.
What's Changed
- fix: Deterministic foreach task id's for Argo Workflows by @saikonen in #1704
- Make Kubernetes default resources configurable by @martinhausio in #1800
- Add METAFLOW_ESCAPE_HATCH_WARNING for s3op by default. by @romain-intel in #1825
- bump version to 2.11.13 by @saikonen in #1826
New Contributors
- @martinhausio made their first contribution in #1800
Full Changelog: 2.11.12...2.11.13
2.11.12
What's Changed
- Fix: JSON Reference Path Error in AWS Step Functions Distributed Map by @nidhinnru in #1822
- fix import of the new escape hatch flag by @wangchy27 in #1823
New Contributors
- @nidhinnru made their first contribution in #1822
Full Changelog: 2.11.11...2.11.12
2.11.11
What's Changed
- Add support for getting EC2 metadata through IMDSv2. by @trhodeos in #1808
- Initialize token for IMDSv1 ec2-metadata by @trhodeos in #1809
- allow @conda to accept nvidia as a compute target by @savingoyal in #1811
- fix: github tests by @saikonen in #1813
- fix: R tests by @saikonen in #1814
- Add a flag to control escape hatch log by @wangchy27 in #1819
- upgrade package versions by @wangchy27 in #1821
New Contributors
Full Changelog: 2.11.10...2.11.11
2.11.10
Improvements
Argo Events trigger improvements for parameters with default values
This release fixes an issue where partial or empty argo event payloads would incorrectly overwrite the default values for the parameters of a triggered flow.
For example a flow with
@trigger(events=["params_event"])
class DefaultParamEventFlow(FlowSpec):
param_a = Parameter(
name="param_a",
default="default value A",
)
param_b = Parameter(
name="param_b",
default="default value B",
)
will now correctly have the default values for its parameters when triggered by
from metaflow.integrations import ArgoEvent
ArgoEvent('params_event').publish()
or a default value for param_b and the supplied value for param_a when triggered by
ArgoEvent('params_event').publish({"param_a": "custom-value"})
What's Changed
- [Ready for review] replace pull_request_target by @darinyu in #1790
- [bug fix] Flow decorator click option names fix by @valayDave in #1775
- [@kubernetes port] Allow configurable port number by @valayDave in #1793
- Bump vite from 5.0.12 to 5.0.13 in /metaflow/plugins/cards/ui by @dependabot in #1791
- feature: add metadata for argo workflows template owner by @saikonen in #1798
- fix: support default parameter values with argo events by @saikonen in #1797
- release 2.11.10 by @saikonen in #1799
Full Changelog: 2.11.9...2.11.10
2.11.9
What's Changed
- check for falsy values instead of just None by @madhur-ob in #1786
- Bump version to 2.11.9 by @madhur-ob in #1787
Full Changelog: 2.11.8...2.11.9
2.11.8
2.11.7
What's Changed
- fix access to name and kwargs by @madhur-ob in #1773
- fix log options by @madhur-ob in #1777
- Bump version to 2.11.7 by @madhur-ob in #1780
Full Changelog: 2.11.6...2.11.7
2.11.6
What's Changed
- Generate stubs properly for extensions with lazy loaded modules by @romain-intel in #1765
- ability to customise log driver and log options by @madhur-ob in #1767
- Fix an issue when unpickling functions through the escape hatch by @romain-intel in #1770
Full Changelog: 2.11.5...2.11.6