Windows: Destination Encoding characters are improperly escaped

[iseahound]

Hi Nick,

It feels like I'm the only person here using rclone to do local Windows → Windows transfers so here's another bug.

Basically, the Full Width Pipe character seems to be escaped with your backtick sometimes resulting in very odd behavior.

I can't fully describe it, but here's the batch files that makes things easy for you to debug.

control.bat - This file contains expected behavior where ｜.txt is successfully copied from folder a to folder b.

@echo off
chcp 65001

mkdir a
mkdir b
echo "hello world" > a/｜.txt
rclone --config="" copy a b
dir b
pause

bug.bat - Here we expect ｜.txt to be copied from folder c to folder ｜ but it ends up in folder ‛｜ for some reason. Note that despite the folder path ending up escaped, the text file ｜.txt is NOT escaped.

@echo off
chcp 65001

mkdir c
mkdir ｜
echo "hello world" > c/｜.txt
rclone --config="" copy c ｜
dir ｜
pause

Specifically, it causes statements like rclone copy <somewhere> . to spontaneously miss if . expands to a path with a Full Width replacement character.

[URenko]

This is an expected behavior, except for your last paragraph.

rclone assumes your file path in the local filesystem is already encoded in a way specified in local-encoding.
Pipe (| -> ｜) is one of the default encodings for rclone on Windows.
So in both of your examples, for the file ｜.txt, it will first be decoded to |.txt, and then encoded to ｜.txt.¹

At the same time, rclone does NOT assume your command line input is already encoded in a way specified in local-encoding. ²
So in your second example, rclone will bring your input to internality and encode to ‛｜.txt, write to the file system, according to local-encoding.

Therefore, if you want your second case to work properly.
Either you can write your command line input in a way decoded with local-encoding. In your case, is rclone --config="" copy c ^|. (^ to escape for Windows command line)
Or you can disable the Pipe encoding by remove it from your local-encoding:

rclone --config="" copy c ｜ --local-encoding "Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,BackSlash,Ctl,RightSpace,RightPeriod,InvalidUtf8,Dot"

Footnotes

1. In fact, it is first decoded from local-encoding to "Standard" and then encoded to local-encoding, but "Standard" does not contain Pipe, so it can be ignored here. ↩

2. In fact, rclone will assume your command line input is already encoded in "Standard" way. ↩

[iseahound]

Thanks for the explanation. I think I am still a little confused, for the command line parameters and the remote each have their own defaults, yet can be both overwritten by local-encoding. Is there a separate flag that I overlooked? Also what does the Standard encoding contain?

Ah I reread it and I now understand. I suppose the main reason is to handle batch files which expand the file path. If that's the case then handling "." in a different way may pose a problem.

[URenko]

for the command line parameters and the remote each have their own defaults, yet can be both overwritten by local-encoding

local-encoding only specifies the encoding for the local filesystem.
The encoding for the command line parameters cannot be specified (but there is a trick, see #7456), see #7456 for the meaning of "Standard".
The flag for other remote backend is --backend-encoding where backend is the name of the backend.

See https://rclone.org/overview/#encoding for detail, but there are some errors (#7791) currently.

[URenko]

Specifically, it causes statements like rclone copy <somewhere> . to spontaneously miss if . expands to a path with a Full Width replacement character.

This is actually a bug, I may open a pr later.