Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Introduce cold storage for saving storage costs #21

Open
yrliou opened this issue Jun 9, 2020 · 4 comments · Fixed by #22
Open

Introduce cold storage for saving storage costs #21

yrliou opened this issue Jun 9, 2020 · 4 comments · Fixed by #22
Labels
enhancement New feature or request

Comments

@yrliou
Copy link
Member

yrliou commented Jun 9, 2020

To save storage cost, we are going to set the TTL attribute to 90 days away when we're soft-deleting an item so it will be automatically deleted by dynamoDB without extra cost.
cc @jsecretan

ref: https://aws.amazon.com/blogs/aws/new-manage-dynamodb-items-using-time-to-live-ttl/

@yrliou yrliou added the enhancement New feature or request label Jun 9, 2020
@yrliou yrliou self-assigned this Jun 9, 2020
@yrliou yrliou changed the title Set TTL when soft-deleting items for removing obsolete items Set TTL when soft-deleting items to automatically remove obsolete items Jun 9, 2020
@yrliou yrliou closed this as completed in #22 Jun 9, 2020
@yrliou
Copy link
Member Author

yrliou commented Jun 10, 2020

PR reverted in #23

@yrliou
Copy link
Member Author

yrliou commented Jun 10, 2020

As pointed out in the reverted PR, we cannot simply delete them directly because it
would break users who had a device that have been offline for more than 90 days, when that device is back online, it cannot receive updates for items that have been deleted for more than 90 days, so this device will get conflict that it cannot resolve when it tries to change any of those items and keep backoff. It could be reset by leave and rejoin sync chain on this device, but the deleted items in this local will be committed back to server and sync to other devices which might not be what user would want.
We need to revisit this topic with some more advanced solutions like putting data into S3 when ttl expires and include data objects from S3 too when client is asking for updates more than 90 days ago.

Good news here is when client committing a delete, they will remove specifics (expect bookmark now is an exception which still send a full specific), so the size of deleted records are not that bad, only their meta data are left in the storage.

@yrliou yrliou changed the title Set TTL when soft-deleting items to automatically remove obsolete items Set TTL when soft-deleting items for minimizing cost of storing obsolete items Jun 10, 2020
@yrliou yrliou changed the title Set TTL when soft-deleting items for minimizing cost of storing obsolete items Set TTL when soft-deleting items for minimizing cost of storing cold items Jun 10, 2020
@yrliou yrliou removed their assignment Jun 10, 2020
@yrliou
Copy link
Member Author

yrliou commented Jun 17, 2020

Another thought is that we can always set TTL when creating/updating the item, and put them into cold storage when that expired. And of course we need to be able to get those items back when needed by our server.

@yrliou yrliou changed the title Set TTL when soft-deleting items for minimizing cost of storing cold items Introduce cold storage for saving storage costs Jun 17, 2020
@yrliou
Copy link
Member Author

yrliou commented Jun 17, 2020

We could also consider saving specifics binary in S3 instead of directly in Dynamo too.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant