Private Plugins

[EugeneTseitlin]

Private Plugins

Summary

Amplication uses plugins as a major way to provide customizability for users' projects. Plugins are also allow to apply same changes or structure to a different services.
Currently Amplication has several plugins the were developed by Amplication Team and are available to the community.
Plugins can also be developed by the community or our customers. And not always these plugins are universal and sharable, sometimes it's a company specific code that is not supposed to be publicly accessible due to privacy, licensing or other reasons.
The idea of Private Plugins is to provide a way to create custom plugins that are going to be securely saved and accessible for use only to defined workspaces.

Requirements

Requirements are based on #5121

1. Plugin developer should be able to upload his plugin to Amplication.
2. Plugin developer should be able to mark a plugin he uploaded as private.
3. Plugin which is marked as "Private" should be securely stored and accessible only to a specified scope.
4. Plugin developer should be able to specify a workspace that has an access to a plugin.
5. Private Plugins can be used only by services in projects that belong to a workspace with an access to it.
6. Private Plugins should be visually distinguishable in the UI from public plugins.
7. Private Plugins should have a dedicated area in the UI for a user to quickly view and use his org private plugins.
8. Only members of the specified workspace can see and use private plugins.

Architecture

Here are the major topics that have to be addressed.

Format

Expected format for uploaded plugins artifact is tgz. That's the format that is currently used to import public plugins.

Authentication

User has to be authenticated in order to access Private Plugins. Currently we have authentication only in amplication-server. There are two options here, the first is to add authentication to another existing or new service, the second is to use amplication-server as an entry point for every request.

Storage

Storage that we are going to use has to persist plugins securely. Here are the possible options: EFS, S3, package registry.
We're already using EFS with encryption enabled. It can be accessible via PV and PVC that already exist in our Kubernetes cluster. Using EFS we will have to keep an appropriate order for storing plugins, possibly migrate in case we decide to change the structure, which might be effort consuming. Also EFS bucket can grow in volume within a time, which can make it hard to backup and replicate.
S3 is another managed service that is optimized for object storage. There is no need to create a backups or restore it, which means less maintenance. Objects can be migrated by chunk using AWS Console, which is simpler and safer than migrations in EFS. Each object can be accessed by direct link.
3rd party package registry are tools that are dedicated for storing npm artifacts. Here are the examples: AWS CodeArtifact, JFrog NPM Registry, Verdaccio, Sonatype. These systems may persist packages, manage versions of packages, provide a granular access to a package. Some of them are managed, some of them are self-deployed. Even when it's self-deployed helm charts are provided, so installation won't require large initial investment of time, but will add maintenance.

Separation of concerns

Lately we are trying to add new functionality in any other service, rather than amplication-server. So there are two candidates: an existing plugin api service or a new private plugin service. In case plugin api is chosen, it's database will have to be modified in order to support new functionality. In case if new service is chosen, a new database, infrastructure resources such as Kubernetes deployment, config map, service will have to added.

Usability

There should be a straightforward way to import plugin, assign it to a workspace, view private plugins and assign them to a service easily.
@GreenMachine01 @alexbass86 Please, elaborate on this part.

Component architecture

Link to Component Architecture: https://drive.google.com/file/d/1KrR1b7KGuNWlEDjXVd-NRhTqfTdA-dgT/view?usp=sharing
Please, feel free to leave your comments in the diagram.

Here is the list of proposed configurations:

1. server + s3
2. server + registry
3. server + plugin api + s3
4. server + plugin api + registry
5. server + private plugin + s3
6. server + private plugin + registry

In my perspective the most promising options are 3 and 4.

[GreenMachine01]

For security, it needs to work both ways.

1. Only the plugin developer or someone that has access to the plugin code can add a private plugin to a workspace.
2. Only someone that has access to the workspace can add a private plugin to the workspace

I see 3 ways this can happen:
Option#1 - The user uploads a private plugin via the UI directly to his workspace
Option#2 - Plugin Developer has a way to mark a workspace as having access to a private plugin. A team member accepts using the private plugin.
Option#3 - Similar to option#2 but with a key/secret/code

[yuval-hazaz]

@GreenMachine01 as for security - in my opinion, it should be very straight forward - the settings are per workspace - meaning that a user can only register a private plugin to their current workplace, and the plugin can only be used in that workspace.

[yuval-hazaz]

@EugeneTseitlin - one more thing we should consider is the license of the service used to manage private plugins. This is an enterprise feature, and as such, it should be created under the ee folder - so unless there is a considerable overhead or cost, we probably need to go with a new service.

The cost to use a new service is that currently, all authenticated requests are sent via amplication-server, so we need to think about two things:

1. schema federation so the client can use a single proxy
2. centralized auth service that will allow peripheral service to authenticate users

[overbit]

We could consider a temporary solution where server will act as a proxy for requests related to private plugins. It will proxy requests to a new private-plugin-api service that could or not share the db with plugin-api (both are valid architecture in this case microservice architecture). The private-plugin-api will be part of our ee services and it will be hosted internally so only server can access it.

A more complex solution is to use an event driven architecture but it could be an overkill since once we have in place a proper IdP authentication and federation would be easier.

[yuval-hazaz]

We should also consider the impact of using two sources for the client - when fetching the data for the list - unless we design the client in a way where the public catalog and the private catalog are totally separated and are not shown in a single list (which can make pagination and filtering very complicated)

[yuval-hazaz]

one more option that was discussed with the team, is to use the org git repo to store the private plugins instead of using our own storage service.
@GreenMachine01 we probably want to talk to customers to understand if this is a relevant option.
We may allow the org to connect to a separate repo that will hold the plugins