How to copy collections (schema & data) data anytime (production to staging)

[rflieb]

Hi Community,

we have two instances of directus running and wonder about how we can copy collections (incl. data) from one env. to another.
Example: Production > Staging without interferring Roles, Users, Permissions and especially API Tokens, which differ between Prod and Staging.

Any suggestions or other how went down this road?

Thx in advance
Ralf

[pavelstianko]

We are using Directus CLI for export and importing schema (npx directus schema snapshot) and DB data dump (only tables other than directus_*). But we have problem with different users UUIDs at dev and production. I hope someone from Directus will explain any Best practice for migrating schema and data :)

Also I found https://www.npmjs.com/package/directus-migrator but I haven't tried it yet.

[rokdd]

Well I made a cronjob which is completly copying the database and the S3 storage of directus, and then we have a config file where the project and tokens for the api get updated from the secrets of the container. It is not absolutely perfect because the versions of directus might differ and the S3 bucket cannot keep the original date after copying ;-)

[pavelstianko]

We are also using .env config files for different environments. Are there any problems when you are copying the whole database (also with directus tables)? Thank you

[rokdd]

We are also using .env config files for different environments. Are there any problems when you are copying the whole database (also with directus tables)? Thank you

We have it running in docker and config with the ENV variables.. Our system runs on postgres and you need to make sure to copy the whole schema and database including all permissions and triggers etc. The best is to really have two seperate databases with their own public schema. I have sometimes some not planned issues but production is always untouched so I don't see a problem or better solution. A problem which we not solved yet how to apply a new version of a schema to the data of prod in an integration stage.

[pavelstianko]

Thank you, we have exactly the same setup (docker compose, env files and postgres), so will try this aproach as well.

[rflieb]

Thank you for your comments so far we will try to check/verify them and come back in terms of any result
We already tried to dump the whole DB to our staging system by keeping the different API Tokens without any success.

As mentioned we will check your suggestions
Thx and appreciated

[rokdd]

Maybe you can write a bit more detailed which configuration you use (db, storage,..) and what means "no success". For example you need always to restart directus after copying else it will apply only half or crash.

[EdouardDem]

Hi, I've developped a CLI for this purpose: https://github.com/tractr/directus-sync
It allows to pull all schema, dashboards, flows, folders, operations, panels, permissions, presets, roles, settings, translations, and webhooks into JSON files (in your repository), and then push them back to another Directus instance.
This is a non-destructive operation and only apply changes.
I have also written a tutorial : https://medium.com/tractr/streamlining-directus-workflows-with-directus-sync-a-comprehensive-guide-baf33b9cc399
Hope this helps