Properly support for Windows long file paths through Unicode Win32 API calls. #2410
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
UTF8ToWin32Unicode() converts an UTF-8 string into the corresponding Windows UCS-2 one, and will be used to call Unicode Win32 API directly, in order to properly support long file paths everywhere. Win32UnicodeToUTF8() does the opposite, and will generally be used to display such native Win32 paths in error messages.
Add an intermediate DiskInterface derived class named SystemDiskInterface that performs real disk i/o without any intermediate caching. RealDiskInterface is now a derived class of SystemDiskInterface that adds a caching layer on Win32 only. Keeping the same name reduces the number of changes in this commit to the strict minimum. NullDiskInterface is provided to simplify custom DiskInterface implementations used in tests, in particular because new DiskInterface methods are going to be added in future commits, and adding the same method overrides NullDiskInterface will lower the changes to do in those implementations.
Ensure Win32 Unicode APIs are called when trying to perform real disk i/o. This is necessary to properly support long file paths on Windows. For example, FindFirstFileExW() used in RealDiskInterface has not MAC_PATH restriction, though FindFirstFileExA() still does, even when long path support is enabled on the host machine. Note that generally speaking, it is unknown whether Windows CRT functions, such as fopen(), rename(), _unlink(), always properly support long file paths (it may very well depend on the version of MSVCRT linked to the executable), so it is better to err on the side of caution and always try to use the wide-char versions of these functions, when available, or to fall back to Unicode Win32 API functions otherwise. This applies here and the commits following this one in the same pull request.
Use CreateFileW() to ensure proper long path support on Win32.
+ Allow it to compile with cross-toolchains like Mingw.
Make the BuildLog class take a DiskInterface reference in its constructor, to ensure that all i/o operations use the same interface. + Adjust call sites accordingly. For tests, always use a SystemDiskInterface instead of the VirtualFileSystem instance when the latter is available, as this is exactly what the previous code was doing.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is an attempt to properly support Windows long file paths by only using Win32 Unicode API calls, and getting rid of the ANSI ones (some of which are still plagued by MAX_PATH limitations, even when long paths support is enabled on the machine).
This is achieved in the following ways (see individual commits for details):
Add
UTF8ToWin32Unicode()andWin32UnicodeToUTF8()functions to util.hFor simplicity, introduce SystemDiskInterface and NullDiskInterface, and make RealDiskInterface a derived class of SystemDiskInterface that adds a caching layer.
(this opens the door to other caching implementations on Posix).
Augment the
DiskInterfaceAPI to addRenameFile()andOpenFile()methods to ... rename a file (just likerename() / _rename()), and open an stdio FILE instance (just likefopen()).Modify the DiskInterface implementation to only use Unicode Win32 APIs.
Modify sources that rely on direct calls to
fopen(),unlink(),rename()to use a DiskInterface instance. This requires injecting such an instance in the constructors of BuildLog and DepsLog btw.Modify other misc Win32-specific sources to use Win32 Unicode API calls directly (e.g. subprocess-win32.cc)
NOTE: VirtualFileSystem::OpenFile() only supports read access for now. Unlike its real SystemDiskInterface implementation, it doesn't perform \r\n to \n conversion on Windows (do we really want this?). That is why many tests (e.g. build_log_test.cc) do not use it, but rely on a real DiskInterface instance. This can be fixed by another PR after this one.
This should fix issue #1900 once and for all (crossing fingers here :-))