Problems with pikpak storing hash after uploads

[djniche]

My version of rclone

rclone version
rclone v1.66.0

• os/version: debian 11.9 (64 bit)
• os/kernel: 5.10.0-22-amd64 (x86_64)
• os/type: linux
• os/arch: amd64
• go/version: go1.22.1
• go/linking: static
• go/tags: none

My Rclone config

[pp]
type = pikpak
user = *******
pass = *******
token = ****
hash_memory_limit = 150G

I found this problem when I moved the files to pikpak, after uploading the files I only keep the pikpak hash value, and deleted the source files from pikpak, then I found that the hash values of several files were not recorded by the server, so I could not recover the files from the server by the hash value.
Later I encountered the problem recorded hash value, and so after a while (about an hour), found that the recorded hash value has changed.
The hash value can be grabbed from the play link, which is provided in the play link, or from a third party client.
Since rclone's synchronization is working fine, I've enabled -v, and there's no error on the console.
And I found that when uploading with rclone, some files are deleted (emptied from recycle bin), but also space is not returned, it takes several days to return the space, when I use alist project to copy pikpak from googledrive it does not happen with caching and hash problems.

I apologize for using a translator, if it's hard to read please bear with me!

[djniche]

Add: only occasionally a few files will appear this problem, not 100%, when I have been retained pikpak source files, do not delete the case, after a period of time will be changed once the hash value, this time the value is correct. After deleting the file I can restore the file by the hash value.
Files with changing hash values can also be played normally, provided they are not deleted from pikpak.

PikPak://MEYD-868-4K.mp4|24055707717|3927AE19823CE8E020402C4C7650CD8FD37E14D1 The hash of the error file that I synchronized with rclone before
PikPak://MEYD-868-4K.mp4|24055707717|6C97D756D7622262EA75F6257738D6E39FF5368E This is the hash I generated after copying pikpak via alist googledrive. The former is invalidated by the deletion of the source file.

[wiserain]

At the time of writing pikpak for rclone, hash used for uploading and verifying file existence wasn't clearly discovered. So I just let it a simple sha1. However, I recently found that it is sha1 of sha1 of file blocks which is similar with torrent piece hash.

Once another fix #7787 which is closely related to this one is sorted out, I will make a new PR for it. Thanks.

[djniche]

At the time of writing pikpak for rclone, hash used for uploading and verifying file existence wasn't clearly discovered. So I just let it a simple sha1. However, I recently found that it is sha1 of sha1 of file blocks which is similar with torrent piece hash.

Once another fix #7787 which is closely related to this one is sorted out, I will make a new PR for it. Thanks.

I know, it's named gcid. The algorithm is written in some Chinese article
Where xunlei is pikpak's algorithm, because pikpak bought the back-end architecture from xunlei, so the calculation method is the same.
https://binux.blog/2012/03/hash_algorithm_of_xunlei/
https://post.smzdm.com/p/awzm3o3m/
(alist project contains pikpak, alist is an open source project , you can look at the specific implementation of the code .
https://github.com/alist-org/alist/tree/main/drivers/pikpak)
def gcid_hash_file(path):
h = hashlib.sha1()
size = os.path.getsize(path)
psize = 0x40000
while size / psize > 0x200 and psize < 0x200000:
psize = psize << 1
with open(path, 'rb') as stream:
data = stream.read(psize)
while data:
h.update(hashlib.sha1(data).digest())
data = stream.read(psize)
return h.hexdigest().upper()

[wiserain]

Thanks for the useful link. Alist is my source of information as well :)