conda list and clean commands

[madhur-ob]

TODOs:

• How to associate step names with environment IDs?
• Where are dependency lock files located? Since we need to delete those as well..

[iamsgarg-ob]

LGTM.. just not sure how easy or hard it is to add tests.

[romain-intel]

haven't looked yet in detail but I think we should name it environment to keep consistency with the extension.

[savingoyal]

@romain-intel, this is more of a hidden utility to clean out the underlying conda environments than providing a top-level environment manipulation command. there is a good reason that we may want to have environment as yet another command in the near future.

[romain-intel]

I mean, it would be exposed to the user and there is similarity with this: https://github.com/Netflix/metaflow-nflx-extensions/blob/main/metaflow_extensions/netflix_ext/plugins/environment_cli.py so I wanted to try to minimize (more) divergence. There is also an argument for doing this in metaflow environment as opposed to myflow.py --environment=conda environment but that may be a larger conversation. If this is just something hidden, we should hide it better or provide a utility. Also, we could start by providing these commands and then expand to others. I just feel a bit uncomfortable providing a user-visible "hidden utility" like this I guess.

[madhur-ob]

I remember asking @savingoyal if we wanna add this at the metaflow conda level or perhaps python flow.py conda level.
The latter was probably chosen because we kind of operate on a per flow basis i.e. conda environments for each step etc. and we want the ability to list them + clean them etc.

At the metaflow conda level, it would be a bit difficult to get the ctx.obj I guess -- which allows all that functionality. I could be wrong though.

[romain-intel]

I remember asking @savingoyal if we wanna add this at the metaflow conda level or perhaps python flow.py conda level. The latter was probably chosen because we kind of operate on a per flow basis i.e. conda environments for each step etc. and we want the ability to list them + clean them etc.

At the metaflow conda level, it would be a bit difficult to get the ctx.obj I guess -- which allows all that functionality. I could be wrong though.

Totally agree that it doesn't necessarily have to be at the metaflow level. However, I would prefer it be named environment to be more compatible with the CLI in the extension (it's one or the other which makes more sense).

[saikonen]

same problem as with list_envs. Not getting any environments when default datastore is not local.

[madhur-ob]

fixed, can you check again please

[saikonen]

this one is still bugging out, though cause is my messy conda manifest:

TypeError: stat: path should be string, bytes, os.PathLike or integer, not NoneType

[saikonen]

getting an error with --show-all with my local manifest:

    print("{:<18} {:<67} {:<8} {:<12} {:<12}".format(row))
TypeError: unsupported format string passed to NoneType.__format__

[madhur-ob]

can you help me with debugging this...
to start with, what is the value of row?

also, what are the values of item, envs_data etc.

[saikonen]

shared my conda manifest for testing purposes. The failing row for displaying is this:

['6bf4a51d2573ea3', None, '', 'osx-32', 'conda']

which most likely is leftovers from my development with the PyPI deco.

[saikonen]

f-string borking python 3.5 tests. Note the other occurrences as well.

[saikonen]

shared my conda manifest for testing purposes. The failing row for displaying is this:

['6bf4a51d2573ea3', None, '', 'osx-32', 'conda']

which most likely is leftovers from my development with the PyPI deco.