Duplicates not resolving (initial upload via CLI)

[wutr]

After finally setting up immich I imported all 8000+ photos and videos from my android phone backup using the CLI. I then installed the immich android app a couple of days later. I've selected one album to be automatically backed up: camera. It contains 1401 items.

The thing is, apart from the photos taken in-between importing via the CLI and installing the android app, all other items show twice. One with a cloud icon (no tick), and one with a crossed out cloud icon. The files are duplicates, not reduced resolution versions or anything like that.

I was under the impression an automatic duplicate check is carried out but that seems to have not been the case.

Am I missing something here?

[etnoy]

Deduplication is currently being worked on, but in the end what decides is the file checksum. If the checksum already exists the file won't be uploaded. Any changes like changed metadata will allow a duplicate upload.

Could you compare the sha1 checksums before they are uploaded and post the result?

[wutr]

Thanks for the quick reply. I've compared the sha1 sums for one file that shows up twice in the immich android app. The android original, the file in my backup, and the file as it is after being imported in immich are all identical:

7184f793d0e9351ce6e41ac436e06766c2900191  android/IMG_20230322_161643.jpg
7184f793d0e9351ce6e41ac436e06766c2900191  backup/IMG_20230322_161643.jpg
7184f793d0e9351ce6e41ac436e06766c2900191  immich/IMG_20230322_161643.jpg

[alextran1502]

There are some sync issues on the mobile app, but they are mostly cosmetic. Can you confirm that the duplication also shows on the web?

[alextran1502]

I would say yes, until the client-side hashing is implemented

[alextran1502]

The workaround is to choose a different album with one or two assets so that the local assets don't get rendered on the main timeline.

[wutr]

The workaround is to choose a different album with one or two assets so that the local assets don't get rendered on the main timeline.

Won't that prevent new files from being synced to the server if they're not part of that dummy album?

[fyfrey]

yes. that would prevent new files from being backed up.
you could move all assets already backed up via CLI to a new device album (and not select this in Immich). Then your newly taken photos still go to the Camera album (and you select only that to be backed up)

[wutr]

I think I'll nuke-and-pave my immich setup, and do the following:

1. Reset the immich android app and let it upload all desired items (the camera album)
2. Once finished, use jdupes to identify which pictures from my backup were not present on my phone and create a symlink to each file in a temporary directory.
3. Use immich CLI to upload all the symlinked items

I'll have to double check the volume mapping allows for my symlink strategy, I'll probably run into some issues.