filer.backup fails: InvalidDigest: The Content-MD5 you specified was invalid

[stibi]

Hello, I'd like to ask for help"

Describe the bug

Error when running weed filer.backup:

~ # weed filer.backup
I0509 07:21:31.983159 s3_sink.go:78 sink.s3.region: eu-central-1
I0509 07:21:31.983501 s3_sink.go:79 sink.s3.bucket: my-backup-bucket-on-aws-s3
I0509 07:21:31.983514 s3_sink.go:80 sink.s3.directory: /
I0509 07:21:31.983517 s3_sink.go:81 sink.s3.endpoint: https://s3.eu-central-1.amazonaws.com
I0509 07:21:31.983520 s3_sink.go:82 sink.s3.acl:
I0509 07:21:31.983524 s3_sink.go:83 sink.s3.is_incremental: false
I0509 07:21:31.983535 s3_sink.go:84 sink.s3.s3_disable_content_md5_validation: true
I0509 07:21:31.983544 s3_sink.go:85 sink.s3.s3_force_path_style: true
I0509 07:21:31.983549 s3_sink.go:86 sink.s3.keep_part_size: true
I0509 07:21:31.983554 s3_sink.go:92 sink.s3.uploader_max_upload_parts: 1000
I0509 07:21:31.983564 s3_sink.go:94 sink.s3.uploader_part_size_mb: 0
I0509 07:21:31.983572 s3_sink.go:95 sink.s3.uploader_concurrency: 8
I0509 07:21:31.983663 filer_replication.go:129 Configure sink to s3
I0509 07:21:31.986668 filer_backup.go:116 resuming from 2024-04-17 13:17:58.576231682 +0000 UTC
I0509 07:21:31.992931 filer_backup.go:135 backup localhost:8888 progressed to 2024-04-17 13:17:58.577248919 +0000 UTC 0.33/sec
E0509 07:21:32.080584 filer_pb_tail.go:89 process directory:"/buckets/test" event_notification:{new_entry:{name:"helloworld.txt" chunks:{file_id:"10,05eda397e3" size:300 modified_ts_ns:1713359891999390853 e_tag:"IuKKhawu/U/VBp1lqLy1jg==" fid:{volume_id:10 file_key:5 cookie:3986921443} is_compressed:true} attributes:{file_size:300 mtime:1713359891 file_mode:432 uid:1000 gid:1000 crtime:1713359891 mime:"text/plain" md5:"\"⊅\xac.\xfdO\xd5\x06\x9de\xa8\xbc\xb5\x8e"} extended:{key:"X-Amz-Meta-Mtime" value:"1713359891"}} delete_chunks:true new_parent_path:"/buckets/test" signatures:-171941028} ts_ns:1713359892003071868: create entry1 : InvalidDigest: The Content-MD5 you specified was invalid.
	status code: 400, request id: TN1Z15MXR6ZMZ6VG, host id: Z6XHKVl7DuXXVM6o4zdHwIKriLbZ5H7ED3Gp86cFmgzIH59WacJBiTHEDK2qhKiOQ7JM5Cj5qc8=
E0509 07:21:32.142823 filer_pb_tail.go:89 process directory:"/buckets/test" event_notification:{new_entry:{name:"test.txt" chunks:{file_id:"9,07cbfc4e6d" size:6 modified_ts_ns:1715238002023166326 e_tag:"j+shH+90SkNGnPB0AAlMcA==" fid:{volume_id:9 file_key:7 cookie:3422309997} is_compressed:true} attributes:{file_size:6 mtime:1715238002 file_mode:432 uid:1000 gid:1000 crtime:1715238002 mime:"text/plain" md5:"\x8f\xeb!\x1f\xeftJCF\x9c\xf0t\x00\tLp"} extended:{key:"X-Amz-Meta-Mtime" value:"1715238002"}} delete_chunks:true new_parent_path:"/buckets/test" signatures:-171941028} ts_ns:1715238002026926467: create entry1 : InvalidDigest: The Content-MD5 you specified was invalid.
	status code: 400, request id: TN1Q3GBJ8725SE9Z, host id: 5FuY4lOsyc3hvtbeapeCP6bQiI1UNJNHxmDyNasQ517TW53+bPTgr5o+A1sLGOIk5rGoeSx2Lvk=

System Setup

• weed s3 -filer=localhost:8888 -ip.bind=0.0.0.0
• weed filer -master=localhost:9333 -ip.bind=0.0.0.0
• weed master -ip.bind=0.0.0.0 -defaultReplication=00
• OS version - Alpine Linux 3.19.1
• output of weed version: version 30GB 3.67 d3032d1 linux amd64

Expected behavior

Working backup of seaweed to S3.

Any idea if a bug or problem with configuration?

Thanks a lot.

[stibi]

Well I got it working (#5596), but I'm not really confident about it, because it would mean that it doesn't work for anyone, meaning, or nobody is using the s3 replication? Strange. ... so, what do you think?

[kmlebedev]

Yes, it’s kind of strange, I’m definitely using replication swfs => s3 sink => swfs
It's worth checking out

[Sepidemirsoltani]

Well I got it working (#5596), but I'm not really confident about it, because it would mean that it doesn't work for anyone, meaning, or nobody is using the s3 replication? Strange. ... so, what do you think?