Medusa-operator usage post 1.0.0

[burmanm]

These are some of the features I'd like to see us improving post-1.0.0 for taking backups. This includes mostly usability and does not as much touch the actual implementation details (although I do have an idea how to implement these - most of them are quite simple). Some API names probably need thinking, this is more of a "how-to" approach in using Medusa inside Kubernetes. These additional options are not necessarily incompatible with the helm, but they also allow configuring the Medusa without using helm.

Configure target repository

Allow separate target repositories where to store the data, such as daily backups to long term storage and hourly to faster s3 storage:

apiVersion: medusa.cassandra-medusa.io/v1alpha1
kind: CassandraBackupRepository
metadata:
  name: cassandrabackuprepository-sample
spec:
  provider: s3
  bucketName: "daily-backups"
  secret: my-aws-secret
  params:
    storageClass: "glacier"
  retentionPolicy:
    ttl: "720d"

apiVersion: medusa.cassandra-medusa.io/v1alpha1
kind: CassandraBackupRepository
metadata:
  name: cassandrabackuprepositoryshort-sample
spec:
  provider: s3
  bucketName: "hourly-backups"
  secret: my-aws-secret
  params:
    storageClass: "ia"
  retentionPolicy:
    ttl: "14d"

Automated backups

The use of CronJob's format is not necessary, we could simplify these to parameters that are more easily understood such as "schedule, everyHour".

Daily backups:

apiVersion: medusa.cassandra-medusa.io/v1alpha1
kind: CassandraBackupSchedule
metadata:
  name: cassandrabackupschedule-sample
spec:
  schedule: "0 0 * * *"
  name: "cass-daily"
  cassandraDatacenter: dc1
  targetRepository: cassandrabackuprepository-sample

Hourly backups:

apiVersion: medusa.cassandra-medusa.io/v1alpha1
kind: CassandraBackupSchedule
metadata:
  name: cassandrabackupschedule-sample
spec:
  schedule: "0 * * * *"
  name: "cass-hourly" 
  cassandraDatacenter: dc1
  targetRepository: cassandrabackuprepositoryshort-sample

View the backups

Whenever medusa creates a backup, it would also create a new object to the Kubernetes so that they could be viewed. These should have necessary information such as TTL that allows expiring the data from S3 as well as medusa-operator to delete the backup object.

➜  storage git:(master) ✗ kubectl get cassandrabackup
NAME                        AGE
cass-daily-202101290000     13h
cass-hourly-202101291300    21m
cass-hourly-202101291200    1h
➜  storage git:(master) ✗

Restore the backup

There's no real need to modify the restore process. Although, one additional option could be to modify the CassandraDatacenter and add a annotation restore.medusa.cassandra-medusa.io; cass-hourly-202101290000 and the the medusa-operator would do the rest. That would be inplace restoration for the cluster, for more complex scenarios we would still require creating a Restore object.

[burmanm]

An additional view would be to merge the schedule and the repository to a single object, that would define the target repository properties as well as schedule.

[burmanm]

Additional ideas, the schedule/repository combination should have status which could allow STOPPED or PAUSED etc to temporarily stop the backup schedule from running. Similar to how we can control CassandraDatacenter these days.

[burmanm]

Also, for metrics (I don't remember if these were documented somewhere - so I'm adding them here) we should probably add a push-gateway to the medusa-operator instance and push some statistics from these jobs (such as success count, failed counts, time it took, etc for each of the automated backup schedules).

[burmanm]

And obviously, if we want to reduce the helm dependency, we could create a kubectl plugin to allow certain operations to happen from there also. I think similar requirement was asked for the cass-operator (by the casskop guys), so this would allow same approach between our components.