Skip to content
This repository has been archived by the owner on Nov 20, 2023. It is now read-only.

Duplicate counts and write to file #92

Open
wants to merge 4 commits into
base: main
Choose a base branch
from
Open

Duplicate counts and write to file #92

Changes from 1 commit
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
21f641b
Added functionality to count how many files were reported as duplicat…
Jun 8, 2023
0e62a05
Changes added from PR conversation. Allowed it to delete the duplicat…
Jun 8, 2023
78d418a
Update bin/index.ts
adoreparler Jun 8, 2023
a966adf
Updated output log statement for writing the duplicates file
Jun 8, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 27 additions & 0 deletions bin/index.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -257,6 +257,8 @@ async function upload(

const limit = pLimit(uploadThreads ?? 5);

const duplicates: Array<string> = [];
adoreparler marked this conversation as resolved.
Show resolved Hide resolved

for (const asset of localAssets) {
const album = asset.filePath.split(path.sep).slice(-2)[0];
if (!assetDirectoryMap.has(album)) {
Expand All @@ -271,6 +273,11 @@ async function upload(
const res = await startUpload(endpoint, key, asset, deviceId);
progressBar.increment(1, { filepath: asset.filePath });
if (res && (res.status == 201 || res.status == 200)) {
if (res.data && res.data.duplicate) {
const filePathSplit = asset.filePath.split('/');
jrasm91 marked this conversation as resolved.
Show resolved Hide resolved
duplicates.push(filePathSplit[filePathSplit.length-1]);
return;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This return probably breaks the album feature. I think it would work fine without the return.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I returned there because I did not want it to delete local asset and was not sure if it should proceed with anything else if it is a duplicate. Let me know your thoughts

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think if they set delete to true it should delete the files when they've been backed up to the server.

It probably doesn't make sense to log the output of duplicates when delete is used though.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have feelings against deleting the asset if its identified as a duplicate. I feel like it technically has not been backed up to the server and should not be deleted. Its either already on there or like in my case google created a bunch of duplicates of the same file and it matched to the original that was uploaded. Logging these cases out so that investigation can happen I feel is what I would want at least. I can remove the return and disable the delete if duplicate, or just remove the return and let it delete.

I'll let you make the decision.

Copy link
Contributor

@jrasm91 jrasm91 Jun 8, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The only situation that can cause duplicate: true is if the file, byte for byte, exists exactly on the server. I personally never use the CLI with the delete option, so I don't really care one way or the other, but I would expect it to delete it (which is what it does today btw - this is existing functionality).

}
if (deleteLocalAsset == 'y') {
fs.unlink(asset.filePath, (err) => {
if (err) {
Expand Down Expand Up @@ -356,6 +363,26 @@ async function upload(
}

log(chalk.yellow(`Failed to upload ${errorAssets.length} files `), errorAssets);

// Ask to create a file that contains the duplicate file names.
if (duplicates.length) {
log(chalk.yellow(`${duplicates.length} files were not uploaded because they were identified as duplicates on the server`));
const writeDupes = await new Promise((resolve) => {
rl.question('Do you want to write the file names of the duplicates to a file? (duplicates.txt) (y/n) ', resolve);
});
jrasm91 marked this conversation as resolved.
Show resolved Hide resolved

if (writeDupes == 'y' || writeDupes == 'Y') {

try {
fs.writeFileSync('duplicates.txt', duplicates.join('\n'));
log(chalk.green(`duplicates.txt created`));
} catch(ex) {
chalk.red(`Error writing duplicates.txt: ${ex}`)
}

}

}

if (errorAssets.length > 0) {
process.exit(1);
Expand Down