GB1030 dotnet info

[agocke]

1. copy a group of GB18030 characters(level 3 long sample data)
   𫚭﹫𪛒𫍲𫟼𬘭𮯠⾢)

2. Use the supported string to create folder. 龯蝌灋齅ㄥ䶱

3. unzip dotnet-sdk-9.0.100-preview.4.24218.26-win-x64.zip to above folder

4. access to the folder and run dotnet --info

Expected Result:

the contents can be displayed well.

Actual Result:

GB18030 characters displays incorrectly.

[dotnet-policy-service]

Tagging subscribers to this area: @vitek-karas, @agocke, @VSadov
See info in area-owners.md if you want to be subscribed.

[elinor-fung]

I had looked into this last week, thinking it would be quick (spoiler: it wasn't). Here's where I got:

The host prints using the C functions (fputws, vfwprintf), which do the conversion from wide to multi-byte as if using wcrtomb - that means it is using C locale (specifically LC_CTYPE). This does not map to what the user may have set through something like chcp 65001 so what may be programmatically set through SetConsoleOutputCP.

This affects all tracing in the host on Windows - so basic output (like --info called out by this issue), any error messages, and any detailed logging to stderr or to a file. These are the functions of interest:

runtime/src/native/corehost/hostmisc/pal.h

Lines 162 to 164 in 778cc84

	inline void file_vprintf(FILE* f, const char_t* format, va_list vl) { ::vfwprintf(f, format, vl); ::fputwc(_X('\n'), f); }
	inline void err_fputs(const char_t* message) { ::fputws(message, stderr); ::fputwc(_X('\n'), stderr); }
	inline void out_vprintf(const char_t* format, va_list vl) { ::vfwprintf(stdout, format, vl); ::fputwc(_X('\n'), stdout); }

Options I looked at:

• explicitly setting locale (setlocale) to ".utf8" and reverting it after
  • setlocale only started supporting using a UTF-8 code page with Win 10 version 1803.
• using _l variants of the functions that take a locale
  • would need to create/free the locale that is passed in
• directly using WriteConsoleW instead of the C functions, which should respect the user's code page settings
  • we'd need to explicitly format before instead of using vwfprintf to handle the arg formatting like we do now

Just trying things out in a console app, I found that for the functions based on C locale, I had to set/pass the locale with UTF-8 as the code page and set the console code page (chcp or SetConsoleOutputCP). WriteConsoleW would respect what code page is set for the console. On my Windows 11 machine, I didn't actually have to explicitly change my code page (which was not set to UTF-8) to have WriteConsoleW display the expected output - maybe some of the improved UTF-8 support that has been coming to Windows over the years.

[jtschuster]

@elinor-fung Thank you so much for the info, I would have been pretty lost trying to fix this issue without it.