Skip to content

Releases: Netflix/metaflow

2.9.4

12 Jun 19:27
@saikonen saikonen
2.9.4
4a3f51a
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.
Compare
Choose a tag to compare
2.9.4

Improvements

Fix using email addresses as usernames for Argo Workflows

Using an email address as the username when deploying with a @project decorator to Argo Workflows is now possible. This release fixes an issue with some generated resources containing characters that are not permitted in names of Argo Workflow resources.

The secrets decorator now supports assuming roles

This release adds the capability to assume specific roles when accessing secrets with the @secrets decorator. The role for accessing a secret can be defined in the following ways

As a global default

By setting the METAFLOW_DEFAULT_SECRET_ROLE environment variable, this role will be assumed when accessing any secret specified in the decorator.

As a global option in the decorator

This will assume the role secret-iam-role for accessing all of the secrets in the sources list.

@secrets(
  sources=["first-secret-source", "second-secret-source"],
  role="secret-iam-role"
)

Or on a per secret basis

Assuming a different role based on the secret in question can be done as well

@secrets(
  sources=[
    {"type": "aws-secrets-manager", "id": "first-secret-source", "role": "first-secret-role"},
    {"type": "aws-secrets-manager", "id": "second-secret-source", "role": "second-secret-role"}
  ]
)

In case you need any assistance or have feedback for us, ping us at chat.metaflow.org or open a GitHub issue.

What's Changed

Full Changelog: 2.9.3...2.9.4

Contributors

tuulos, pjoshi30, and 4 other contributors
Assets 2

2.9.3

01 Jun 18:13
@romain-intel romain-intel
2.9.3
68a27b6
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.
Compare
Choose a tag to compare
2.9.3

Improvements

Ignore duplicate Metaflow Extensions packages

Duplicate Metaflow Extensions packages were not properly ignored in all cases. This release fixes this and will allow the loading of extensions even if they are present in duplicate form in your sys.path.

Fix package leaks for the environment escape

In some cases, packages from the outside environment (non Conda) could leak into the Conda environment when using the environment escape functionality. This release addresses this issue and ensures that no spurious packages are imported in the Conda environment.

In case you need any assistance or have feedback for us, ping us at chat.metaflow.org or open a GitHub issue.

What's Changed

Full Changelog: 2.9.2...2.9.3

Contributors

dhpollack, savingoyal, and 2 other contributors
Assets 2

2.9.2

22 May 16:17
@savingoyal savingoyal
2.9.2
4c3b771
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.
Compare
Choose a tag to compare
2.9.2

Features

Introduce support for image pull policy for @kubernetes

With this release, Metaflow users can specify image pull policy for their workloads through the @kubernetes decorator for Metaflow tasks.

@kubernetes(image='foo:tag', image_pull_policy='Always') # Allowed values are Always, IfNotPresent, Never
@step
def train(self):
    ... 
    ...

If an image pull policy is not specified, and the tag for the container image is :latest or the tag for the container image is not specified, image pull policy is automatically set to Always.

If an image pull policy is not specified, and the tag for the container image is specified as a value that is not :latest, image pull policy is automatically set to IfNotPresent.

In case you need any assistance or have feedback for us, ping us at chat.metaflow.org or open a GitHub issue.

What's Changed

Full Changelog: 2.9.1...2.9.2

Contributors

tuulos, savingoyal, and 3 other contributors
Assets 2

2.9.1

16 May 02:15
@savingoyal savingoyal
2.9.1
86b99fe
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.
Compare
Choose a tag to compare
2.9.1

Features

Introduce Slack notifications support for workflow running on Argo Workflows

With this release, Metaflow users can get notified on Slack when their workflows succeed or fail on Argo Workflows. Using this feature is quite straightforward

  1. Follow these instructions on Slack to set up incoming webhooks for your Slack workspace.
  2. You should now have a webhook URL that Slack provides. Here is an example webhook: 
    https://hooks.slack.com/services/T0XXXXXXXXX/B0XXXXXXXXX/qZXXXXXX
  3. To enable notifications on Slack when your Metaflow flow running on Argo Workflows succeeds or fails, deploy it using the --notify-on-error or --notify-on-success flags: 
    python flow.py argo-workflows create --notify-on-error --notify-on-success --notify-slack-webhook-url <slack-webhook-url>
  4. You can also set METAFLOW_ARGO_WORKFLOWS_CREATE_NOTIFY_SLACK_WEBHOOK_URL=<slack-webhook-url> in your environment instead of specifying --notify-slack-webhook-url on the CLI everytime.
  5. Next time your workflow succeeds or fails on Argo Workflows, you will get a helpful notification on Slack.

FAQ

I deployed my workflow following the instructions above, but I haven’t received any notifications yet?

This issue may very well happen if you are running Kubernetes v1.24 or newer.

Since v1.24, Kubernetes stopped automatically creating a secret for every serviceAccount. Argo Workflows relies on the existence of these secrets to run lifecycle hooks responsible for the emission of these notifications.

Follow these steps for explicitly creating a secret for the service account that responsible for executing Argo Workflows steps:

  1. Run the following command, replacing service-account.name with the serviceAccount in your deployment. Also change the name of the secret to correctly reflect the name of the _serviceAccount _for which this secret is 
    cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Secret
metadata:
  name: default-sa-token #change according to the name of the sa
  annotations:
    kubernetes.io/service-account.name: default #replace with your sa
type: kubernetes.io/service-account-token
EOF
  2. Edit the serviceAccount object so as to add the name of the above secret in it. You can use kubectl edit for this. The serviceAccount yaml should look like the following 
    $ kubectl edit sa default -n mynamespace
...
apiVersion: v1
kind: ServiceAccount
metadata:
  creationTimestamp: "2023-05-05T20:58:58Z"
  name: default
  namespace: jobs-default
  resourceVersion: "6739507"
  uid: 4a708eff-d6ba-4dd8-80ee-8fb3c4c1e1c7
secrets:
- name: default-sa-token # should match the secret above
  3. That’s it! Try executing your workflow again on Argo Workflows. If you are still running into issues, reach out to us!

In case you need any assistance or have feedback for us, ping us at chat.metaflow.org or open a GitHub issue.

What's Changed

Full Changelog: 2.8.6...2.9.0

Contributors

savingoyal, kubernetes, and saikonen
Assets 2

2.9.0

16 May 00:39
@savingoyal savingoyal
2.9.0
4965335
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.
Compare
Choose a tag to compare
2.9.0

Features

Introduce support for composing multiple interrelated workflows through external events

With this release, Metaflow users can architect sequences of workflows that conduct data across teams, all the way from ETL and data warehouse to final ML outputs. Detailed documentation and a blog post to follow very shortly! Keep watching this space.

In case you need any assistance or have feedback for us, ping us at chat.metaflow.org or open a GitHub issue.

What's Changed

Full Changelog: 2.8.6...2.9.0

Contributors

savingoyal, kubernetes, and saikonen
Assets 2

2.8.6

10 May 17:51
@savingoyal savingoyal
2.8.6
0386164
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.
Compare
Choose a tag to compare
2.8.6

Features

Introduce support for persistent volume claims for executions on Kubernetes

With this release, Metaflow users can attach existing persistent volume claims to Metaflow tasks running on a Kubernetes cluster.

To use this functionality, simply list your persistent volume claim and mount point using the persistent_volume_claims arg in @kubernetes decorator - @kubernetes(persistent_volume_claims={"pvc-claim-name": "mount-point", "another-pvc-claim-name": "another-mount-point"}).

Here is an example:

from metaflow import FlowSpec, step, kubernetes, current
import os

class MountPVCFlow(FlowSpec):

    @kubernetes(persistent_volume_claims={"test-pvc-feature-claim": "/mnt/testvol"})
    @step
    def start(self):
        print('testing PVC')
        mount = "/mnt/testvol"
        file = f"zeros_run_{current.run_id}"
        with open(os.path.join(mount, file), "w+") as f:
            f.write("\0" * 50)
            f.flush()
        
        print(f"mount folder contents: {os.listdir(mount)}")
        self.next(self.end)

    @step
    def end(self):
        print("finished")

if __name__=="__main__":
    MountPVCFlow()

In case you need any assistance or have feedback for us, ping us at chat.metaflow.org or open a GitHub issue.

What's Changed

Full Changelog: 2.8.5...2.8.6

Contributors

savingoyal, tfurmston, and 2 other contributors
Assets 2

2.8.5

05 May 22:53
@romain-intel romain-intel
2.8.5
e47f5d7
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.
Compare
Choose a tag to compare
2.8.5

Improvements

Improvements

Make pickled Metaflow client objects accessible across namespaces

The previous release resulted in disabling a sequence of user operations that worked previously:

  1. Pickle a Metaflow object
  2. Access this Metaflow object in a different namespace
  3. Access a child or parent object of this object

This release restores the previous behavior.

In case you need any assistance or have feedback for us, ping us at chat.metaflow.org or open a GitHub issue.

What's Changed

Full Changelog: 2.8.4...2.8.5

Contributors

pjoshi30, romain-intel, and saikonen
Assets 2

2.8.4

03 May 01:54
@savingoyal savingoyal
2.8.4
61dbc58
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.
Compare
Choose a tag to compare
2.8.4

Features

Introduce support for tmpfs for executions on Kubernetes

It is typical for the user code in a Metaflow step to download assets from an object store, e.g. S3. Examples include serialized models and raw input data, such unstructured media or structured Parquet files. The amount of data loaded in a task is typically 10-100GB, allowing even terabytes to be handled in a foreach.

To reduce IO bottlenecks in such tasks, we provide an optimized client for S3, metaflow.S3 that makes it possible to download data using all available network bandwidth. Notably, in a modern instance the available network bandwidth can be higher than the local disk bandwidth. Consider: SATA 3.0 provides 6Gbit/s whereas a large instance can have 20Gbit/s network throughput. Even Gen3 NVMe provides just 16Git/s. To benefit from the full network bandwidth, local disk IO must be bypassed. The metaflow.S3 client accomplishes this by relying on the page cache: Nominally files are downloaded in a temporary directory on disk but practically all data stays in the page cache. This is assuming that the downloaded data can fit in memory, which can be ensured by having a high enough @resources(memory=) setting.

The above setup, which can provide excellent IO performance in general, has a small gotcha: The instance needs to have enough local disk space to back all the data, although no data actually hits the disk. Increasingly, instances may have more memory than local disk space available, so this superfluous requirement becomes a problem. This puts users in a strange situation: The instance has enough RAM to hold all the data in memory, and there are ways to download it quickly from S3, but the lack of local disk space (that is not even needed), makes it impossible to access the data.

Kubernetes supports mounting a tmpfs filesystem on the fly. Using this feature, the user can create a memory-backed file system which can be used as a temporary space for downloaded data. This removes the need to have to deal with any local disks. One can simply use a minimal root filesystem, which greatly simplifies the infrastructure setup.

With this release, we introduce a new config option - METAFLOW_TEMPDIR, which, if defined, is used as the default metaflow.S3(tmproot). If METAFLOW_TEMPDIR is not defined, tmproot=’.’ as before. In addition, a few new attributes are introduced for @kubernetes decorator -

Attribute (default) Default behavior Override semantics
use_tmpfs=False tmpfs disabled use_tmpfs=True enables tmpfs
tmpfs_tempdir=True sets METAFLOW_TEMPDIR=tmpfs_path tmpfs_tempdir=False doesn't set METAFLOW_TEMPDIR
tmpfs_size=None sets tmpfs size to 50% of @resources(memory) tmpfs size in megabytes
tmpfs_path=None use /metaflow_temp as tmpfs_path custom mount point

Examples

Handle large amounts of data in-memory with Kubernetes:
@kubernetes(memory=100000, use_tmpfs=True)

In this case, at most 50GB is available for tmpfs and it is used by S3 by default. Note that tmpfs only consumes the amount of memory corresponding to the data stored, so there is no downside in setting a large size by default.

Increase tmpfs size:
@kubernetes(memory=100000, tmpfs_size=100000)

Let tmpfs use all available memory. Note that use_tmpfs=True doesn’t have to be specified redundantly.

Custom tmpfs use case:
@kubernetes(memory=100000, tmpfs_size=10000, tmpfs_path=’/data’, tmpfs_tempdir=False)

Full control over settings - metaflow.S3 doesn’t use the tmpfs volume in this case.

Besides metaflow.S3, the user may want to use the tmpfs volume for their own use cases. In particular, many modern ML libraries require a local cache. To support these use cases, tmpfs_path is exposed through the current object, as current.tempdir.
This allows the user to leverage the volume straightforwardly:

AutoModelForSeq2SeqLM.from_pretrained(
            model_path,
            cache_dir=current.tempdir,
            device_map='auto',
            load_in_8bit=True,
        )

Introduce current.run and current.task_ in current singleton

With this release, you can access current.run and current.task within a running flow, allowing for use cases like

from metaflow import current

# add tags from inside a run
current.run.add_tag('foobar')

Improvements

Make metaflow client objects backward compatible

The previous release broke backward compatibility in cases where the metaflow client object is deserialized from an older version of Metaflow. This release preserves the functionality and provides explicit compatibility guarantees going forward.

In case you need any assistance or have feedback for us, ping us at chat.metaflow.org or open a GitHub issue.

What's Changed

New Contributors

Full Changelog: 2.8.3...2.8.4

Contributors

dhpollack, pjoshi30, and 7 other contributors
Assets 2

2.8.3

12 Apr 17:52
@savingoyal savingoyal
2.8.3
74247ed
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.
Compare
Choose a tag to compare
2.8.3

Features

Introduce support for tmpfs for executions on AWS Batch

It is typical for the user code in a Metaflow step to download assets from an object store, e.g. S3. Examples include serialized models and raw input data, such unstructured media or structured Parquet files. The amount of data loaded in a task is typically 10-100GB, allowing even terabytes to be handled in a foreach.

To reduce IO bottlenecks in such tasks, we provide an optimized client for S3, metaflow.S3 that makes it possible to download data using all available network bandwidth. Notably, in a modern instance the available network bandwidth can be higher than the local disk bandwidth. Consider: SATA 3.0 provides 6Gbit/s whereas a large instance can have 20Gbit/s network throughput. Even Gen3 NVMe provides just 16Git/s. To benefit from the full network bandwidth, local disk IO must be bypassed. The metaflow.S3 client accomplishes this by relying on the page cache: Nominally files are downloaded in a temporary directory on disk but practically all data stays in the page cache. This is assuming that the downloaded data can fit in memory, which can be ensured by having a high enough @resources(memory=) setting.

The above setup, which can provide excellent IO performance in general, has a small gotcha: The instance needs to have enough local disk space to back all the data, although no data actually hits the disk. Increasingly, instances may have more memory than local disk space available, so this superfluous requirement becomes a problem. The issue is further amplified by the fact that as of today, it is impossible to add ephemeral volumes on the fly on AWS Batch. This puts users in a strange situation: The instance has enough RAM to hold all the data in memory, and there are ways to download it quickly from S3, but the lack of local disk space (that is not even needed), makes it impossible to access the data.

AWS Batch supports mounting a tmpfs filesystem on the fly. Using this feature, the user can create a memory-backed file system which can be used as a temporary space for downloaded data. This removes the need to have to deal with any local disks. One can simply use a minimal root filesystem, which greatly simplifies the infrastructure setup.

With this release, we introduce a new config option - METAFLOW_TEMPDIR, which, if defined, is used as the default metaflow.S3(tmproot). If METAFLOW_TEMPDIR is not defined, tmproot=’.’ as before. In addition, a few new attributes are introduced for @Batch decorator -

Attribute (default) Default behavior Override semantics
use_tmpfs=False tmpfs disabled use_tmpfs=True enables tmpfs
tmpfs_tempdir=True sets METAFLOW_TEMPDIR=tmpfs_path tmpfs_tempdir=False doesn't set METAFLOW_TEMPDIR
tmpfs_size=None sets tmpfs size to 50% of @resources(memory) tmpfs size in megabytes
tmpfs_path=None use /metaflow_temp as tmpfs_path custom mount point

Examples

Handle large amounts of data in-memory with Batch:
@batch(memory=100000, use_tmpfs=True)

In this case, at most 50GB is available for tmpfs and it is used by S3 by default. Note that tmpfs only consumes the amount of memory corresponding to the data stored, so there is no downside in setting a large size by default.

Increase tmpfs size:
@batch(memory=100000, tmpfs_size=100000)

Let tmpfs use all available memory. Note that use_tmpfs=True doesn’t have to be specified redundantly.

Custom tmpfs use case:
@batch(memory=100000, tmpfs_size=10000, tmpfs_path=’/data’, tmpfs_tempdir=False)

Full control over settings - metaflow.S3 doesn’t use the tmpfs volume in this case.

Besides metaflow.S3, the user may want to use the tmpfs volume for their own use cases. In particular, many modern ML libraries require a local cache. To support these use cases, tmpfs_path is exposed through the current object, as current.tempdir.
This allows the user to leverage the volume straightforwardly:

AutoModelForSeq2SeqLM.from_pretrained(
            model_path,
            cache_dir=current.tempdir,
            device_map='auto',
            load_in_8bit=True,
        )

Introduce auto-completion support for metaflow client in ipython notebooks

With this release, Metaflow client objects will support autocomplete in ipython notebooks

from metaflow import Flow, Metaflow

Metaflow().flows
>>> [Flow('HelloFlow'), Flow('MovieStatsFlow')]

flow = Flow('HelloFlow') # No autocomplete here
flow._ipython_key_completions_()
>>> 
['1680815181013681',
 '1680815178214737',
 '1680432265121345',
 '1680430310127401']

run = flow["1680815178214737"]
run._ipython_key_completions_()
>>> ['end', 'hello', 'start']

step = run["hello"]
step._ipython_key_completions_()
>>> ['2']

task = step["2"]
task._ipython_key_completions_()
>>> ['name']

Improvements

Reduce metadata service network calls for faster execution of flows

With this release, Metaflow flows should execute a tad bit faster since a few network calls to Metaflow's metadata service are now cached. Expect continued further improvements in flow execution times over the next few releases.

Handle unsupported data types for pandas.DataFrame gracefully for Metaflow's default card

With this release, Metaflow card creation will handle non-JSON parseable types gracefully by replacing the column values with UnsupportedType : <TYPENAME>.

In case you need any assistance or have feedback for us, ping us at chat.metaflow.org or open a GitHub issue.

What's Changed

New Contributors

Full Changelog: 2.8.2...2.8.3

Contributors

jimbudarz, Batch, and 8 other contributors
Assets 2

2.8.2

28 Mar 23:03
@savingoyal savingoyal
2.8.2
2136a57
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.
Compare
Choose a tag to compare
2.8.2

Features

Introduce support for Metaflow sandboxes for Metaflow tutorials

With this release, the Metaflow tutorials can now be executed within the Metaflow sandboxes, making it trivial to evaluate whether Metaflow is a good fit for your organization without committing to deploying the necessary cloud infrastructure upfront.

Display Metaflow UI URL on the terminal when a flow is executed via step-functions trigger or argo-workflows trigger

With this release, if the Metaflow config (in ~/.metaflow_config) includes a reference to the deployed Metaflow UI (assigned to METAFLOW_UI_URL), the user-facing logs in the terminal will indicate the direct link to the relevant run view in the Metaflow UI.

image (6)

In case you need any assistance or have feedback for us, ping us at chat.metaflow.org or open a GitHub issue.

What's Changed

New Contributors

Full Changelog: 2.8.1...2.8.2

Contributors

savingoyal, romain-intel, and 2 other contributors
Assets 2