cli 2.4.4 and 2.9.5 syncs a subfolder "data" even though I --exclude *.*

[DaleSchultz]

I have been using CLI for some years without any issues.

For one of my files I use:
aws s3 sync . s3://remotesign/ --exclude "*.*" --include "DB.exe" --acl public-read

To upload just that single file to my bucket, and this has worked well for years.

I recently added a subfolder to my folder called "data"

I noticed that the same command now includes the data subfolder and all it subfolders.

There are six other folders that have never uploaded and still do not. They do not exist in my bucket.

If I rename my data folder to datax or xdata it still grabs the files, so its not a hard coded folder name issue.

I have tried to exclude the data folder with:

--exclude "data"
--exclude "data\"
--exclude "data/"

with no success. Here is an example using --dryrun

• Is this a bug?
• Why only that newly created folder?
• How can I exclude a folder?

Version information: aws-cli/2.9.15 Python/3.9.11 Windows/10 exe/AMD64 prompt/off
(aws 2.4.4 also behaved the same way so I upgraded to 2.9.15)

[RyanFitzSimmonsAK]

Hi @DaleSchultz, thanks for reaching out. I wasn't able to reproduce this behavior. In your exclusion parameter, could you try --exclude "data/*"? If that doesn't work, could you provide debug logs with sensitive info redacted? You can get debug logs by adding --debug to your input. Thanks!

[DaleSchultz]

Thanks Ryan,

Indeed --exclude "data/*" does exclude the data folder. But I still should not have to right?

Here is some of the output with --dryrun and --debug:

2023-01-25 22:26:46,237 - MainThread - awscli.customizations.s3.filters - DEBUG - C:\trains\vb\db\data\8000266\D-2023-01-14-4.txt matched exclude filter: C:\trains\vb\db*.*
2023-01-25 22:26:46,237 - MainThread - awscli.customizations.s3.filters - DEBUG - C:\trains\vb\db\data\8000266\D-2023-01-14-4.txt did not match exclude filter: remotesign*.*
2023-01-25 22:26:46,237 - MainThread - awscli.customizations.s3.filters - DEBUG - C:\trains\vb\db\data\8000266\D-2023-01-14-4.txt did not match include filter: C:\trains\vb\db\DB.exe
2023-01-25 22:26:46,237 - MainThread - awscli.customizations.s3.filters - DEBUG - C:\trains\vb\db\data\8000266\D-2023-01-14-4.txt did not match include filter: remotesign\DB.exe
2023-01-25 22:26:46,237 - MainThread - awscli.customizations.s3.filters - DEBUG - =C:\trains\vb\db\data\8000266\D-2023-01-14-4.txt final filtered status, should_include: False
2023-01-25 22:26:46,237 - MainThread - awscli.customizations.s3.filters - DEBUG - C:\trains\vb\db\data\8000266\Wuppertal Hbf did not match exclude filter: C:\trains\vb\db*.*
2023-01-25 22:26:46,237 - MainThread - awscli.customizations.s3.filters - DEBUG - C:\trains\vb\db\data\8000266\Wuppertal Hbf did not match exclude filter: remotesign*.*
2023-01-25 22:26:46,237 - MainThread - awscli.customizations.s3.filters - DEBUG - C:\trains\vb\db\data\8000266\Wuppertal Hbf did not match include filter: C:\trains\vb\db\DB.exe
2023-01-25 22:26:46,237 - MainThread - awscli.customizations.s3.filters - DEBUG - C:\trains\vb\db\data\8000266\Wuppertal Hbf did not match include filter: remotesign\DB.exe
2023-01-25 22:26:46,237 - MainThread - awscli.customizations.s3.filters - DEBUG - =C:\trains\vb\db\data\8000266\Wuppertal Hbf final filtered status, should_include: True
2023-01-25 22:26:46,237 - MainThread - awscli.customizations.s3.syncstrategy.base - DEBUG - syncing: C:\trains\vb\db\data\8000266\Wuppertal Hbf -> remotesign/data/8000266/Wuppertal Hbf, file does not exist at destination
(dryrun) upload: data\8000266\Wuppertal Hbf to s3://remotesign/data/8000266/Wuppertal Hbf

So it is excluding files with an extension, and including files with no extension.

I suspect that instead of excluding "." I need to exclude "*" It certainly seems to work correctly - I am used to DOS file masks and perhaps this is more Regex where I presume . means the file must have a dot - thus does not exclude files without an extension....

Certainly from all this I see that aws is having to process thousands of files, so my new batch file copies the one file I want into a subfolder and syncs just that folder.

[RyanFitzSimmonsAK]

Yep, excluding "*" should work as well. The full explanation of what each special character does in the filter can be found in the Use of Exclude and Include Filters section of the documentation.