Programmatic creation of workspaces?

[ncdc]

We are starting to get a few requests to be able to create workspaces programmatically, from controllers that talk to an APIExport view where the export claims Workspaces and the binding accepts the claim. This poses some challenges, such as:

1. The APIExport view impersonates a service account in the target workspace when issuing requests to kcp. The service account is assigned the system:kcp:admin group. When creating a workspace this way, the new workspace's owner annotation lists this service account as the owner. But, the service account is in the parent workspace, and we do not allow service accounts to operate outside of their own workspace. This means that workspace initialization cannot succeed, because that flow impersonates the workspace owner to perform actions. Because the owner is a service account in a different workspace, this impersonation is always rejected by the authorization chain.
2. There is no way for the APIExport owner to indicate who the owner of the newly created workspace should be.
3. There is no way for the APIBinding consumer to influence the ownership or to allow an APIExport owner to create a subworkspace on the APIBinding consumer's behalf (e.g. via accepting an impersonation claim).

We do not want to change the restriction that a service account may only operate inside its workspace. Therefore, we need to come up with some new ideas for how to resolve this.

When we thought about workspace creation initially, we assumed it would be something that the user (an actual human) would do, mainly because there are aspects such as what's described above related to workspace ownership and permissions that are required for workspace initialization.

For folks interested in programmatically creating workspaces, we'd love to hear your use cases. Especially details around ownership, given that you want to have a controller create workspaces for users.

cc @fabriziopandini @yastij

[yhrn]

Chiming in with a use case here. We're building an internal developer platform for a pretty large enterprise. Our users (application developers) declaratively specify that they have a "system" by creating a configuration file in a central repo. In there they specify some fundamental aspects of their system, such as ownership, what environments it has (prod, staging, etc.), in what regions they want to deploy their services, what cloud vendor they need to use.

Given this we want to create one KCP workspace per environment and make sure that the relevant APIs are available there. We don't want our users to have to understand KCP concepts, to them it should just look like they are getting their own k8s cluster with a few less standard resources and a set of custom resources.

It's worth noting that creating this system configuration results in more things happening, e.g. the system gets registered in our software catalog and becomes visible in out developer portal, etc.

[stevekuznetsov]

Do you expect the users to have identities in kcp and access those workspaces with a $KUBECONFIG? Should they have the ability to delete the runtime state (Workspace object) from the system, and would you expect your GitOps flow to re-create it? Is the state of objects inside of the workspace derived from your central repo?

[lionelvillard]

we want to create one KCP workspace per environment and make sure that the relevant APIs are available there

We are working on the exact same use case.

[yhrn]

Do you expect the users to have identities in kcp and access those workspaces with a $KUBECONFIG?

Yes, definitely for viewing. Possibly also to do write operations for non-prod environments.

Should they have the ability to delete the runtime state (Workspace object) from the system, and would you expect your GitOps flow to re-create it?

They should not have the ability to delete workspaces directly. But if they change their system definition our controller might decide to delete it if we believe it is safe.

Is the state of objects inside of the workspace derived from your central repo?

It should typically be derived from repos owned by our users. We would set up something like ArgoCD for them for this purpose.

[MikeSpreitzer]

I have not thought through the details of permissions carefully yet, but we are doing this in the mailbox controller in edge-mc. See the run-through in https://github.com/kcp-dev/edge-mc/blob/main/docs/poc2023q1/mailbox-controller.md , recently successfully tested against v0.11.0 of kcp-dev/kcp. It runs the mailbox controller as the user in the current context of the admin.kubeconfig created by kcp start , which is the user named kcp-admin.

[lionelvillard]

I don't understand this: where the export claims Workspaces and the binding accepts the claim.. How can you create bindings without a workspace? Is it to allow APIExport to create sub-workspaces? What's the use case?

[ncdc]

• Service provider is in workspace sp, it has APIExport e which claims workspaces
• Consumer is in workspace c, it has an APIBinding b to APIExport e. The binding accepts the claim on workspaces.
• Service provider wants to do POST /services/apiexport/$cluster/$export/clusters/c/apis/tenancy.kcp.io/v1alpha1/workspaces to create a child workspace under c

[fabriziopandini]

Thank you @ncdc for opening this issue, I really appreciate the openness of the team in discussing this use case and sharing the internals of how the system works.

we'd love to hear your use cases. Especially details around ownership, given that you want to have a controller create workspaces for users

We are still defining requirements and learning what we can do, but also in our case we are currently playing around with the idea of offering “workspace as a service”; as per the discussion thread we can describe it also as a “a higher-level button for creating a workspace” or “some sort of catalog / workspace builder wizard that spits out a kubeconfig with APIs preconfigured”.

We do not want to change the restriction that a service account may only operate inside its workspace. Therefore, we need to come up with some new ideas for how to resolve this.

I'm not sure I fully grasp yet what workspace ownership entails, but I think that a mechanism to specify the ownership for the workspace being created makes sense not only for controllers.

For humans, think e.g. platform administrators, it means they can create orgs or high-level workspaces and then end-over ownership to someone else. For controllers, it could be a very specific and explicit way out of the service account restriction, which makes sense per se but in this specific use case it is limiting how we can programmatically control the system.

If I try to think about what we want the ownership to be set to, I can think at two models: one, with ownership always assigned to a highly privileged user, which is probably ok only for single-tenant deployments/less complex organizations, or another model when you can programmatically assign ownership to different users.
One example that comes to my mind for the second use case is a controller that creates "root workspaces" for different orgs, which AFAIK is something that won't be possible from the CLI.

[ncdc]

Workspace ownership primarily exists as a concept to handle workspace initialization. When a workspace is created, there are a set of controllers that bootstrap the new workspace with things such as APIBindings, default resources (such as the default namespace), and potentially other things as well. The rules that we follow are:

• programmatic workspace initialization is a fast-path optimization/shortcut to avoid having to do the same work manually
• everything a controller does to init a workspace must be doable as a normal cluster-admin user inside an empty workspace
• a controller that initializes a workspace must not be a highly privileged user (it cannot be system:masters); it must have the same set of privileges as the user who created the workspace

[stevekuznetsov]

One tiny bit of context - if some $user creates a new workspace with an initializer that binds some APIExport - the "owner" concept allows us to enforce that the code doing initialization may only bind that APIExport if the $user has that privilege.

[stevekuznetsov]

@yhrn @fabriziopandini @lionelvillard I'm writing a KEP for kcp to support the concept of Entitlements which will remove the need to model cross-workspace concerns using Kubernetes RBAC, allowing us to provide workspaces-as-a-service without needing to pile workarounds on workarounds. kcp-dev/enhancements#4 PTAL and any & all feedback appreciated!

[fabriziopandini]

Thanks! I will take a look
cc @yastij