Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow arbitrary AWS tags based on metaflow config or batch decorator #1627

Open
Limess opened this issue Nov 8, 2023 · 3 comments · May be fixed by #1628
Open

Allow arbitrary AWS tags based on metaflow config or batch decorator #1627

Limess opened this issue Nov 8, 2023 · 3 comments · May be fixed by #1628

Comments

@Limess
Copy link

Limess commented Nov 8, 2023

We have some cost allocation tags we'd like to add which are not tied to a given user.
As such, we'd like to be able to tag and then allocate AWS batch usage from metaflow correctly based on these tags.

Currently Metaflow supports METAFLOW_BATCH_EMIT_TAGS but this only supports ['metaflow.flow_name', 'metaflow.run_id', 'metaflow.step_name', 'metaflow.version', 'metaflow.run_id.$', 'metaflow.user', 'metaflow.owner', 'metaflow.production_token'] (https://github.com/Netflix/metaflow/pull/631/files).

Ideally it'd be great to be able to do this both at the metaflow config level (so a team can set/forget it) and in the @batch decorator, but the latter would be the most useful.

@Limess Limess linked a pull request Nov 8, 2023 that will close this issue
Limess added a commit to Limess/metaflow that referenced this issue Nov 8, 2023
Limess added a commit to Limess/metaflow that referenced this issue Nov 8, 2023
Limess added a commit to Limess/metaflow that referenced this issue Nov 8, 2023
@tonyszhang
Copy link

This is needed for Autodesk usage as well, as we have many new teams use Metaflow and need a way to manage EC2 tags per user/team via the .metaflowconfig/config.json file (or within batch decorator)

@mvarma-caris
Copy link

This looks like a great feature to have!

@anentropic
Copy link

+1 my org uses standardised attribution tags across all the cloud infra, would want to be able to automatically tag metaflow resources with the standard org tags for the parent project, not just metaflow's own metadata tags

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants