Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

🚧 feat(Mobile): Use client side hash to check if asset exists in remote before backup #9306

Draft
wants to merge 3 commits into
base: main
Choose a base branch
from

Conversation

dramikei
Copy link

@dramikei dramikei commented May 7, 2024

Allows Client to check for asset existence in remote server by making use of asset SHA-1 checksum along with API implemented in #2072

Fixes #1553

Problem statement:

When reinstalling immich app, or when asset backup already exists on immich server (through icloudpd and immich-cli for example) the client app uploads all the assets from the device which then get deduplicated from the server-side. Resulting in unnecessary upload of 20k+ assets, wasted bandwidth and time.

Change:

Assets that are already on the server are ignored and not uploaded to the server

Tests:

  • Tested on Android (emulator)
  • Tested on iOS (physical device)

@dramikei
Copy link
Author

dramikei commented May 7, 2024

Found some issues while testing on iOS:

  1. Hashing a large amount of files takes quite some time, so its better to convert hashing before backup into a cancellable "job" of its own with persistence in case app gets killed
  2. If we have assets uploaded to icloud with Optimize storage turned on, the hashing function sometimes fails to fetch the original file because of final file = await assetEntities[i].originFile; in hash service. converting this to final file = await assetEntities[i].loadFile(withSubtype: Platform.isIOS); seems to fix the issue, but raises another concern (see point 3)
  3. when using loadFile PhotoManagerPlugin downloads each asset to hash, which slows down the entire backup preparation.

final bulkUploadCheckRequest = assetList
.map((e) => AssetBulkUploadCheckItem(
checksum: e.checksum,
id: e.localId != null ? e.localId! : e.id.toString()))

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
id: e.localId != null ? e.localId! : e.id.toString()))
id: e.localId ?? e.id.toString()))

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Move to client side hashing
3 participants