Storage v2

[elitan]

in this post I'm outlining Storage version 2. We're discussed this internally to better support storage with Hasura and we think you'll like it.

What we'd like to get is feedback on things we have missed and suggestions on improvements. Or if you like the approach, please tell us that too. Any feedback is appreciated.

---

TLDR

We'll create a new files table in a new storage schema.

This would be the new approach for uploading a file:

1. Upload the file: nhost.storage.upload(file).
2. File is stored in S3
3. File metadata is stored in storage.files.
4. A file_id is returned referencing the newly inserted record in storage.files.
5. The client insert or updates a file_id column in the database with the file_id.

This approach uses only Hasura permissions. If you can read the file in the storage.files table (using Hasura permissions) you can also construct the URL to read the file.

---

Full version

Store all file metadata in a new storage schema. This means that all file upload must go via the Storage API so the file metadata and the actual files are in sync.

File storage

Files are stored in an S3 server in a single predefined bucket. One bucket per Storage service.

Restrictions

All file upload and download go via the Storage server. Never directly via S3.

Schema

Schema:

• storage

Tables:

• files
• migrations (for feature schema changes)

Files table

The files table in the storage schema will have the following columns:

• id uuid PK
• created_at timestampz
• updated_at timestampz
• access_token uuid UNIQUE
• name text (filename)
• pathname text {access-token}/file_name combined with a computed fields.
• size int (file size in bytes)
• mime_type text (ex image/png)
• cache_control text (ex max-age=3600)
• uploaded_by_ip text

How to store a file

Let's say you have two tables:

products
- id
- name
- description

product_images
- id
- product_id (FK to public.products.id)
- file_id (FK to storage.files.id)

The way you'd add a file to a product is via the product_images table. After uploading a file and getting a file_id back you need to insert or update the product_images table with the file id.

Permissions

Anyone can upload a file.

All other permissions is handled by Hasura permissions using GraphQL:

create - inserting a new row with the file_id
read - read the files' metadata (to construct the URL to read the actual file)
update - updating an existing row with a new file_id
delete - delete the file from storage.files.

Access Token

We use the access_token saved in the storage.files to construct our URL like this: /storage/file/${file.accessToken}/${file.name}.

This means if you can read the metadata in the storage.files table you're allowed to also get the actual file.

This is secure because the accessToken is a non-guessable UUID.

CDN

Since the accessToken is part of the URL there is no need to do "look-up requests" on incoming file requests. Instead the file can be served directly which means a CDN can easily be put in front of Storage.

Endpoints and Functions

• /storage/upload - POST - to upload a new file
• /storage/file - GET - to get a file

Code example

storage.files:
- id
- created_at: timestampz
- updated_at: timestampz
- access_token: uuid
- file_name: text
- url: text
- size: int
- mime_type: text
- cache_control: text
- uploaded_by_ip: IP number


public.products:
- id
- created_at
- name
- description

public.product_images
- id
- product_id uuid FK to public.products.id
- file_id uuid FK to storage.files.id

function Files() {
  const [file, setFile] = useState(null);

  function addFile(e) {
    setFile(e.target.files[0]);
  }

  async function upload() {

    try {
      // specifying a file name is optional. The original file name will be used if not filename is specified.
      // const { fileId } await storage.upload(file, "cat.jpg"); (change the name of the file)
      const { fileId } await storage.upload(file); // (use the original file name)
    } catch (error) {
      console.log({ error });
      return alert("Upload failed");
    }
			
		try {
      await insertProductImage({
        variables: {
          product_image: {
            product_id: product_id,
            file_id: file_id,
          },
        },
      });
    } catch (error) {
      console.error(error);
      return alert("Failed adding item");
    }

    alert("Upload successful");
  }

Get products and the products' images:

query {
  products {
    id
    name
    description
    product_images {
      id
      file {
        id
        pathname
        name
      }
    }
  }
}

Display images:

product_images.map(images => {
    return (
      <img src={`${process.env.STORAGE_BASE_URL}/${images.file.pathname}`} />
      // or
      // <img src={`${process.env.STORAGE_BASE_URL}/${images.file.access_token}/${images.file.name}`} />
    )
}

Overview

[RubenVanEldik]

Interesting idea!

I already use part of this in my current codebase (I create a uuid for each file and store that with the item it belongs to), so I'm definely open for it!

I'll think about it more in the coming days and will try to write up my thoughts in the weekend. :)

[elitan]

Good feedback that you're already doing something similar.

[aaronhayes]

You could also track the user id in the files table. This is is available in the JWT token in the authorization header. Can be nullable if uploading files with admin secret.

storage.files

user_id uuid FK to auth.accounts.user_id nullable

[aaronhayes]

Yeah there's a bunch of use cases for tracking the user id - more useful than the ip adresss IMO.

If a user wanted to delete all their data would make it much easier to find/delete files.

[elitan]

Right, but generally the record you delete should generally be on the table (in the public schema) where the file_id is at. Deleting such a record will remove the file information from the GraphQL API and the file is practically not accessible anymore.

Then we could automatically delete "dangling" files in the storage.files table periodically. Which we need to do anyway because some files will probably be uploaded without inserted into the database for various reasons.

There is also another thing. We're thinking of splitting HBP into two separate services for Auth and Storage. Having database migrations from one service that depends on the other service can be tricky.

Having a database dependency from Storage to Auth such as a uploader_user_id FK to public.users.id will require Auth for Storage to run. And you would not be able to run Storage separately if you're using a different Auth strategy.

I like the suggestion. I'm just raising some points to think about.

[aaronhayes]

Right, but generally the record you delete should generally be on the table (in the public schema) where the file_id is at. Deleting such a record will remove the file information from the GraphQL API and the file is practically not accessible anymore.

Practically not accessible may not compile with GDPR delete requests.

Under this system, when fetching a file does the storage API hit hasura on every GET request or does it just serve the file directly? If a link to a file has been shared, would it still be fetch-able if the row in storage.files was deleted?

Then we could automatically delete "dangling" files in the storage.files table periodically. Which we need to do anyway because some files will probably be uploaded without inserted into the database for various reasons.

Could be risky if the application uploads files directly/outside of hbp storage. This option should be configurable at a minimum.

There is also another thing. We're thinking of splitting HBP into two separate services for Auth and Storage. Having database migrations from one service that depends on the other service can be tricky.

Oh okay, in that case drop the FK restraint on the user_id column, the developer can then choose which (if any) tables that has a relationship to.

[elitan]

Practically not accessible may not compile with GDPR delete requests.

Under this system, when fetching a file does the storage API hit Hasura on every GET request or does it just serve the file directly? If a link to a file has been shared, would it still be fetch-able if the row in storage.files was deleted?

The file would be served directly. When uploading a file the following would happen:

1. Client upload file
2. Storage receives the request
3. Storage insert the metadata in storage.files and get the inserted and generated id and access_token.
4. Uploads the actual file to the S3 server at /${access_token}/${file.name}.

So when a user GET the file they hit ${baseURL}/storage/file/${access_token}/${file.name}. This mean there is no database requests when getting the file.

(Maybe we can skip access_token and use id directly) And rotate the "access_token" (id) by updating the id. And always set ON UPDATE CASCADE on the file_id FK in other tables.

If the record in the storage.files is deleted we should have a webhook (Hasura even trigger on delete) that will send a request to this Storage server to delete the actual file in the S3 server.

Could be risky if the application uploads files directly/outside of hbp storage. This option should be configurable at a minimum.

Files should always be uploaded through HBP. Never to S3 directly.

Oh okay, in that case drop the FK restraint on the user_id column, the developer can then choose which (if any) tables that has a relationship to.

That's an option. Or that we say the Storage has a dependency on Auth. We'll see how do with the splitting and dependencies cross Storage and Auth. I'm actually starting to like the storage.files.uploader_user_id column.

[aaronhayes]

(Maybe we can skip access_token and use id directly) And rotate the "access_token" (id) by updating the id. And always set ON UPDATE CASCADE on the file_id FK in other tables.

If you're using the access token in the path ie. ${access_token/${file_name} it essentially becomes an id. I don't know enough about rotating access tokens/specifics here to comment further.

Files should always be uploaded through HBP. Never to S3 directly.

That's fair I was trying to think of situations where users of the app would need to let their clients upload files to S3 which can be done using signed urls. Granted that's will be a rare use case. Was trying to avoid somebody getting 85% of the way with nhost and not being able to configure it to their needs. But I overlooked a very simple solution to that - second S3 bucket.

If the record in the storage.files is deleted we should have a webhook (Hasura even trigger on delete) that will send a request to this Storage server to delete the actual file in the S3 server.

I quite like this solution; very "hasura".

[elitan]

I'll just add to SELECT permissions.

What we can do is allow all users to SELECT everything in storage.files but limit the number of rows to 0.

That means users are only allowed to read the metadata of the file in storage.files if they know a file_id. This means the developer only need to set SELECT permissions on the table where the file_id is.

[nunopato]

What we can do is allow all users to SELECT everything in storage.files but limit the number of rows to 0

I am not sure I understand how this works. Even if you know the file_id, wouldn't this make it impossible to retrieve that row from storage.files ?

[elitan]

No. There is a difference between listing and getting individual rows.

So if we set permissions to allow users to select every column but we limit the number of rows to list to 0 the following happen:

• Users can not list all rows in storage.files
• User can access rows in storage.files via GraphQL if they have read access to a file_id in another table

That means that the this:

query {
  storage_files {
    id
    access_token
    created_at
  }
}

will return :

{
  "storage_files": []
}

but the following:

query {
  product_images {
    id
    file {
      id
      access_token
      created_at
    }
  }
}

will return:

{
  "product_images": [{
    "id": "..",
    "file": {
       "id": "...",
       "access_token": "...",
       "created_at": "...",
     }
  }]
}

[nunopato]

makes sense now 👍

[elitan]

Relevant for the future:

hasura/graphql-engine#696
hasura/graphql-engine#4110

[elitan]

On INSERT and UPDATE permissions. We can set those directly in Hasura too.

This is an example of permissions on a table (ex product_images) with file_id FK to storage.files.id.

[jmarbutt]

I like this, it makes sense to me/

[SkinyMonkey]

Not sure if this will be of interest/inspiration or anything but i created a filesystem (file and folders download/uploads) based on hasura here: https://github.com/SkinyMonkey/hasura-fs
For their hackaton.
I built it as composable service : to work with it, you spin an instance and create a fs_user for each user.
This way you keep the fs id in your own user table with whatever data you want in another hasura and can do a remote join when you need to access the filesystem.

It's aiming at some features that are not needed by HBP but just in case it gives you some ideas.
For example i developed a streaming/on the fly zip creation/reading from/to a folder.

I did exactly the model you described (events to delete file on s3 etc)
I was planning on creating a cron job for dangling files etc.

[elitan]

Nice, that's useful.

The big difference with our approach is that we don't have a file system model that the developer should think about. Instead you insert the file_id into your database and get all file metadata (inc. URL) from that.

[SkinyMonkey]

I also abstracted away the storage backend, it could be S3, Azure blob, A local filesystem etc.
'not too hard to at least give the opportunity to fork and add whatever backend is needed easily by implementing a simple interface

[elitan]

Our proposed solution above uses Hasura permissions. If a user can read the row with the file_id the user can also read all metadata of the file and create the URL to read the actual file too.

Do we want to complement this approach with a "signed URL" approach where a user can get a signed URL that is only valid for x amount of time? Do we allow one or the other (access key vs signed URLs)? Or do we skip signed URLs for now?

Personally, I've never used signed URLs when developing an app. So from personal experience, I think the access key approach is enough.

cc @aaronhayes @nunopato @plmercereau

[elitan]

How do we handle permissions with both access tokens and pre-signed URLs?

With access tokens, if a user can read the file_id the user can read all file metadata in the storage.files table.

How do we handle this permission dilemma with pre-signed URLs?

Do we add an option when uploading the file that this file can only be accessed via pre-signed URLs? And add a only_pre_signed: true to the storage.files?
Should some users be able to use the access token, and other users can only access the file via a pre-signed URL?

[elitan]

Just dumping some thoughts here for an approach:

If a user has access to a file (via Hasura permissions) they can do two things:

• Generate the direct file URL: https://xxx.nhost.app/storage/file/${accessToken}/${fileName}
• Generate a pre-signed URL: nhost.storage. createSignedUrl(accessToken, 120) (120 means link is valid for 120 sec) that will generate a unique link that is signed like this: https://xxx.nhost.app/storage/file/${randomStr}/${fileName}?token=<sign-token>

The important thing is that the pre-signed URL does not contain the accessToken because the accessToken will give direct and unlimited access to the file.

randomStr could be a hash like this:

const randomStr = encrypt(accessToken, secret)

That means we can resolve the accessToken when receiving the request of a pre-signed URL:

const accessToken = decrypt(randomStr, secret)

The main question is: Is it OK to allow users who have access to the file to generate the pre-signed URLs or should it be accommodated in some other way?

[nunopato]

If a user has access to the file, then I do not see a reason for not allowing them to generate the pre-signed URL.

[jmarbutt]

I agree, if a user has access to the file, then they should be able to generate the pre-signed URL.

[ArieJones]

Yes, I agree with this as well .. this is the way I would expect it to work.

[nunopato]

1. Upload the file: nhost.storage.upload(file).
2. File is stored in S3
3. File metadata is stored in storage.files.
4. A file_id is returned referencing the newly inserted record in storage.files.
5. The client insert or updates a file_id column in the database with the file_id.

vs

1. Client upload file
2. Storage receives the request
3. Storage insert the metadata in storage.files and get the inserted and generated id and access_token.
4. Uploads the actual file to the S3 server at /${access_token}/${file.name}.

I see these 2 versions throughout the discussion, not sure if it is intended but I personally think we should do the second version. Users should be able to define permissions for uploading the file itself (not everyone should be able to upload a file). When storage receives an upload request, it tries to insert the file's metadata in storage.files. 2 things can happen here:

1. The user has no permission to upload, so the file is discarded and an error is sent back to the client
2. permissions are passed and the file is actually uploaded to S3.

If the record in the storage.files is deleted we should have a webhook (Hasura even trigger on delete) that will send a request to this Storage server to delete the actual file in the S3 server.
👍

That's an option. Or that we say the Storage has a dependency on Auth. We'll see how do with the splitting and dependencies cross Storage and Auth. I'm actually starting to like the storage.files.uploader_user_id column.

I would avoid having references from different schemas that need to be tracked (migrations, metadata, etc) simply because this might prove to be hard if we split the repos. Ideally, schemas are a way of scoping specific features/services.

[elitan]

That's an option but I don't think it will work very well.

What if you want to insert data + a file in the same request?

How will the permissions be able to resolve if it only gets the table and the column? Permissions can be much more general like, you're only allowed to insert if user_id=x-hasura-user-id and file.mime_type=image/* and other_field < 4. There is no way for these kinds of permissions to be able to resolve during storage.upload() in the example above because you're not doing the full GraphQL mutation during storage.upload().

The problem you're trying to solve is: Check permission before uploading the file.

I get that and it makes sense to do that. But it's too complicated and will hurt DX in my opinion.

We can manage the downside of allowing anyone to upload a file and check permissions when inserting or updating the file_id with a GraphQL request by:

• Automatically remove dangling files (file_ids that was never inserted/updated into any table.)
• Implement abuse detection that can identify if someone is spamming storage.upload() without inserting any file_id

99% of requests to a Storage service will come from "good" actors. I think it's important that we make it easy for developers and "good" actors to use this Storage service.

For "bad" actors we instead implement abuse detection and automatically remove dangling files.

[kratam]

What if the user can insert a row to product_images with a nested file object and it returns a signed_upload_url where the user can upload the file to? The signed URL can have some pre-defined conditions to restrict the mime types, the length of the file or to set a specific key.

insert_product_images_one(object: { product_id: 1, file: { data: { name: "foo", mime_type:"image/png" } } } ) {
  id
  file { signed_upload_url }
}

Then the user can upload the file to the signed URL.

[elitan]

INSERT and UPDATE permissions can be added like this:

#516 (comment)

Clients should never add file data directly.

Always:

1. Upload file and get back file_id
2. Insert file_id (ex in product_images.file_id)

[kratam]

Clients should never add file data directly.

Why not? S3 will refuse the upload operation if the actual file doesn't match the provided mime type (or key, content length, etc) encoded into the signed url (which is generated by the server). And hasura permissions can be used to allow/disallow certain mime types based on the user's id, roles, etc.

You cannot upload without a signed url so we don't have to allow anonymous upload and we can also limit the max file size (possibly based on the user's permissions).

[elitan]

Why not?

Because we'll recommend the following workflow:

1. Upload file and get back file_id
2. Insert file_id (ex in product_images.file_id)

If you want to resolve permissions in step 1 it gets very complicated and the code will not be intuitive.

Instead, we'll allow anyone to upload any file. Then resolve all permissions when the user does the GraphQL mutation (insert/update). During the mutation permission for the file can be resolved.

[SkinyMonkey]

Are you aiming for small or big files?
Allow to resume a file upload could be a plus for big files.
The more time it takes for a file to upload, the more possible network error can happen.

[elitan]

I have not thought about it. How would that work?

[mrinalwahal]

Checkout tusd protocol (https://github.com/tus/tusd/) I agree with @SkinyMonkey. Resumable file uploads might be have a niche audience, but an audience that would directly convert as an Nhost customer if they get this one small feature along with the entire Nhost stack.

I can implement it in Golang. They actually have an SDK. It can be integrated easily with any http client that would authenticate incoming file upload requests in our stack and perform the upload.

It works with both S3 and Minio.

[RubenVanEldik]

I think it's a very interesting idea, it solves a lot of limitations that the current storage has!

I do have a few small questions/remarks:

1. The file security is basically security through obscurity right? If someone finds out a link to a file they'll be able to access it. A UUID makes it unguessable, but the URL will be send over networks and could be intercepted, since its not encrypted with HTTPS. Are there any other large platforms that handle files this way?
2. What is the purpose of the IP field in the files table? I try to store the least amount of PII as possible and I'm not sure what the benefits of this field currently are.
3. I think it would be handy to have 3 levels of 'upload permission': everyone, logged-in users, and admin-secret only. This permission is global and could be set by an ENV variable. Almost all applications only need users to upload files and can therefore use the second level. This would make it a bit harder for bad actors to upload files and would allow it to automatically store the user_id in the files table. (as Aaron suggested)

[elitan]

Thanks for the feedback @RubenVanEldik !

1. Yes, it's security through obscurity. We recommend that you always use HTTPS with this to encrypt the path [link].
2. It's for future abuse detection to ban IPs that upload files in bad faith. This could be combined with a future user_id colum field on the storage.files table.
3. I like that idea with the env var. I think we should have it and default to "logged-in". 👍

[escamoteur]

Maybe I just over-read it but how will the Storage console change with this new API?

[elitan]

You'll see the file in the database UI just like other data types such as text, integer, boolean.

[escamoteur]

so no separate storage management like we have now in the NHost console?

[elitan]

Our current thinking is that there will be a storage management UI but without any folder structure because files will be in the storage.files table and attached, via a file_id, directly to the data in the database.