Update conda-store explanations #726
Conversation
✅ Deploy Preview for kaleidoscopic-dango-0cf31d ready!
To edit notification comments on pull requests, go to your Netlify site configuration. |
pavithraes
added
area: documentation 📖
| Each individual user has a separate namespace, which has the same name as their username (used while logging in). All environments in this namespace are private to the individual. | ||
|
|
||
| A user can be a part of several other "shared" namespaces, and based on the level of access given to them, they can view and use the environment, edit the environment, or delete it all together. The permission are dictated by "role mappings". | ||
|
|
||
| ## Role mappings |
There was a problem hiding this comment.
@peytondmurray - We can document the permissions for each role in this section
|
@pavithraes What's the status of this? Is there something I can do here to move this along? Those role permissions would be really nice to have. |
|
@peytondmurray I'd appreciate a review on this PR if you have time, and any notes on the role mapping (I'm not familiar with the recent updates, so any notes will be helpful to complete that section). This PR is mainly blocked on an initial review. |
|
Here are the current role bindings: _viewer_permissions = {
schema.Permissions.ENVIRONMENT_READ,
schema.Permissions.NAMESPACE_READ,
schema.Permissions.NAMESPACE_ROLE_MAPPING_READ,
}
_editor_permissions = {
schema.Permissions.BUILD_CANCEL,
schema.Permissions.ENVIRONMENT_CREATE,
schema.Permissions.ENVIRONMENT_READ,
schema.Permissions.ENVIRONMENT_UPDATE,
schema.Permissions.ENVIRONMENT_SOLVE,
schema.Permissions.NAMESPACE_READ,
schema.Permissions.NAMESPACE_ROLE_MAPPING_READ,
schema.Permissions.SETTING_READ,
}
_admin_permissions = {
schema.Permissions.BUILD_DELETE,
schema.Permissions.BUILD_CANCEL,
schema.Permissions.ENVIRONMENT_CREATE,
schema.Permissions.ENVIRONMENT_DELETE,
schema.Permissions.ENVIRONMENT_READ,
schema.Permissions.ENVIRONMENT_UPDATE,
schema.Permissions.ENVIRONMENT_SOLVE,
schema.Permissions.NAMESPACE_CREATE,
schema.Permissions.NAMESPACE_DELETE,
schema.Permissions.NAMESPACE_READ,
schema.Permissions.NAMESPACE_UPDATE,
schema.Permissions.NAMESPACE_ROLE_MAPPING_CREATE,
schema.Permissions.NAMESPACE_ROLE_MAPPING_READ,
schema.Permissions.NAMESPACE_ROLE_MAPPING_UPDATE,
schema.Permissions.NAMESPACE_ROLE_MAPPING_DELETE,
schema.Permissions.SETTING_READ,
schema.Permissions.SETTING_UPDATE,
} |
|
|
||
| A pinned YAML file is generated for each environment ta is built. | ||
| This includes pinning of the `pip`` packages as well. | ||
| YAML file that follows the conda specification is a common way to create environments. |
There was a problem hiding this comment.
| YAML file that follows the conda specification is a common way to create environments. | |
| YAML files that follow the conda specification are a common way to create environments. |
| In rare cases, the pinned packages may not solve because packages are | ||
| routinely marked as broken and removed. |
There was a problem hiding this comment.
| In rare cases, the pinned packages may not solve because packages are | |
| routinely marked as broken and removed. | |
| In rare cases, building environments from "pinned" YAML files may not solve because packages are routinely marked as broken and removed at the repository level. |
| Assuming you have `conda` installed, to create a conda environment (on any machine) using this file: | ||
| Click on **"Show yml file"** link in the conda-store UI to open the file in a new browser tab. You can download the file[^1] and share with someone or use it to create an environment on a different machine. | ||
|
|
||
| [^1]: Concretely, download the browser page displaying the file. For example, on macOS: Right-click on the page -> Select "Save As" -> Give the file a meaningful name (like `environment.yml`). |
There was a problem hiding this comment.
I'd argue this footnote is probably not necessary; if you click on this link it'll just take you to the raw text anyway, right?
| ```bash | ||
| conda env create --file <environment.yml> | ||
| ``` |
There was a problem hiding this comment.
IMO we should be extremely careful with mixing conda and conda-store terminology in user documentation. There's already a lot of confusion amongst experienced conda/conda-store users about what falls under the purview of conda-store and what falls under the purview of conda; there was a recent issue raised about the output of conda config because we haven't been super careful about this in the past.
I think there's a chance that users will think that running conda env create --file <environment.yml> will somehow create an environment managed by conda-store, which it won't. I think we need to be super careful about this if we want to avoid a lot of support tickets moving forward.
| ```shell | ||
| conda install -c conda-forge lockfile | ||
| conda install -c conda-forge conda-lock | ||
| ``` | ||
|
|
||
| Install the locked environment file from conda-store. | ||
| Create an environment using the lockfile generated by conda-store: | ||
|
|
||
| ```shell | ||
| conda-lock install <lockfile-filename> | ||
| conda-lock install <lockfile.yml> | ||
| ``` |
There was a problem hiding this comment.
See note above about mentioning conda/conda-lock/conda-store related terminology in the same context. I think that there's an extremely high chance that users will see this documentation, run these commands, and then expect these environments to be managed by conda-store.
| 1. Conda downloads `channeldata.json` from each of the channels which | ||
| ## Environment creation | ||
|
|
||
| Given an `environment.yaml` file, this is how conda perform a build (in brief): |
There was a problem hiding this comment.
| Given an `environment.yaml` file, this is how conda perform a build (in brief): | |
| Given an `environment.yaml` file, this is how conda performs a build (in brief): |
|
|
||
| 4. The specific packages are downloaded | ||
| 5. Conda does :sparkles: magic :sparkles: to fix the path prefixes of the installs, which is beyond the scope of this page. |
There was a problem hiding this comment.
Unless we really want to expand upon this, let's omit this note. There's really not enough information here to explain what is meant by "fix the path prefixes of the installs" or why it's important.
| * In step (2) `repodata.json` is updated regularly. When conda solves for a user's environment it tries to use the latest version of each package. Since `repodata.json` could be updated the very next minute, the same solve for the same | ||
| `environment.yaml` file can result in different solves. To enable reproducibility, conda-store auto-generates certain artifacts like lockfiles and tarballs that capture the actual versions of packages and can be used reliably re-create the same environment. Learn more about them in the [artifacts documentation][artifacts]. |
There was a problem hiding this comment.
I know that conda-store handles this by downloading/keeping repodata.json up to date, but another way to talk about this without getting into too many implementation details is to say that conda channels get updated with new package versions regularly, for example:
| * In step (2) `repodata.json` is updated regularly. When conda solves for a user's environment it tries to use the latest version of each package. Since `repodata.json` could be updated the very next minute, the same solve for the same | |
| `environment.yaml` file can result in different solves. To enable reproducibility, conda-store auto-generates certain artifacts like lockfiles and tarballs that capture the actual versions of packages and can be used reliably re-create the same environment. Learn more about them in the [artifacts documentation][artifacts]. | |
| * When a user creates an environment, conda tries to use the latest version of each package requested in the environment specification. Conda channels are constantly being updated with new package versions, so the same solve for the same `environment.yaml` file can result in different dependencies being downloaded. To enable reproducibility, conda-store auto-generates certain artifacts like lockfiles and tarballs that capture the actual versions of packages and can be used reliably re-create the same environment. Learn more about them in the [artifacts documentation][artifacts]. |
| conda-store always re-builds an environment from scratch when edits are detected, which is required for ensuring truly reproducible environments. | ||
| Version control is very useful in any collaborative setting, and environments are no exception. | ||
| Hence, conda-store keeps older versions of the environment for reference, and allows you to select different versions when needed. conda-store-ui also provides a graphical way to [switch between versions][conda-store-ui-version-control]. |
There was a problem hiding this comment.
I think here it would be nice to state that the user can always revert to a previous build at any time. This is one of the key defining features of conda-store that you can't get anywhere else, so we should be say it loudly here!
| - Developer (to be changed to Editor) | ||
| - Admin | ||
|
|
||
| ## Environment versions |
There was a problem hiding this comment.
I know I talked about this before, but internally we refer to different environment versions as "builds". It might be a good time to update the documentation to match, what do you think?
Addresses #748
Description
This pull request updates:
Pull request checklist
Additional information
This PR needs some more work, but is ready for an initial review.