You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Graphic projects can differ in size tremendously. From a single Photoshop file up to a 50 GB file set of 3D scenes, textures, and assets. These project types come with their own set of problems. In the following, I want to clear up some misconceptions about the topic around file locking.
File Locking
Take a look at the code snippet below.
// Process 1
fd = fs.openSync("~/foo", "w");
// Process 2
fd = fs.openSync("~/foo", "w");
Imagine more than one process wants to open the same file at the same time. What do you think will happen?
Answer: It depends on the OS and if you're the maintainer of all processes.
When you call fs.openSync NodeJS will forward the call behind the scenes to an OS function as you can see from this C code
The function open(..) is an OS function and available in all operating systems. But the internals of this function differs between Windows, Linux and macOS so I will cover them separately.
macOS/Linux
Technically, neither macOS nor Linux have true file-locking mechanisms. Although you can read or write-lock a file using a function called fcntl, only programs that use this function regard and respect the file lock. This means, any other process which doesn't use fcntl and directly wants to open a file can acquire a file handle and manipulate the content as long as the file permissions allow it. What a bummer.
Windows is more complicated in that matter. Windows offers two functions to open a file. Either through the Windows API function called CreateFile (yes, that's really the name to open files),...
...or through open(..). But the open(..) function on Windows is a POSIX extension and uses CreateFile internally as well.
As we've seen above NodeJS uses open(..), but since we know that this is just a wrapper for CreateFile, let's check out that function:
// The low-level open function of Windows.
HANDLE CreateFile(
LPCSTR lpFileName,
DWORD dwDesiredAccess,
DWORD dwShareMode,
LPSECURITY_ATTRIBUTES lpSecurityAttributes,
DWORD dwCreationDisposition,
DWORD dwFlagsAndAttributes,
HANDLE hTemplateFile
);
CreateFile has a parameter called dwShareMode. A file that is opened with dwShareMode=0cannot be opened again until its handle has been closed.
So if you use open(..) on a file that was already open by another process with CreateFile(…, dwShareMode=0) you receive this error message:
The process cannot access the file because it is being used by another process
On the other hand, if you use fs.openSync in NodeJS, or open(..) in C/C++, to open a file that hasn't been opened yet, you cannot prevent another application from modifying it*.
* Unless you use file permissions as a workaround, but that’s not really a file lock.
To prove this, you will see that our fs.openSync call executes CreateFile with the read/write shared flags to comply with the POSIX standard.
This means on Windows you cannot prevent another application from opening and modifying your file if you don't use CreateFile.
What does this have to do with Snowtrack?
Imagine a user saving a big file in a graphic application and while the file is still being written to disk, the user attempts to commit the file change. How does Snowtrack deal with this?
As we learned, open(..) has no file locking and most applications don't even follow the file protocol and Snowtrack cannot control how Photoshop, Blender, and co. open and write their files.
This means the only reliable chance of detecting if a file is still being written by another process is to check prior to a commit if any process on the system has a write handle on that file.
On Windows, I solved this with a custom helper process and the Windows API of Restart Manager which is mainly used for installers to ensure the files it is about to replace are not open anymore.
On MacOS I invoke the system process /usr/sbin/lsof (list open files) with an inclusion of the working-directory to speed up the execution of this command.
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
-
Summary
Graphic projects can differ in size tremendously. From a single Photoshop file up to a 50 GB file set of 3D scenes, textures, and assets. These project types come with their own set of problems. In the following, I want to clear up some misconceptions about the topic around file locking.
File Locking
Take a look at the code snippet below.
Imagine more than one process wants to open the same file at the same time. What do you think will happen?
Answer: It depends on the OS and if you're the maintainer of all processes.
When you call
fs.openSync
NodeJS will forward the call behind the scenes to an OS function as you can see from this C codeThe function
open(..)
is an OS function and available in all operating systems. But the internals of this function differs between Windows, Linux and macOS so I will cover them separately.macOS/Linux
Technically, neither macOS nor Linux have true file-locking mechanisms. Although you can read or write-lock a file using a function called
fcntl
, only programs that use this function regard and respect the file lock. This means, any other process which doesn't usefcntl
and directly wants to open a file can acquire a file handle and manipulate the content as long as the file permissions allow it. What a bummer.That's why file locking on macOS and Linux is also called "advisory file locking".
Windows
Windows is more complicated in that matter. Windows offers two functions to open a file. Either through the Windows API function called CreateFile (yes, that's really the name to open files),...
...or through
open(..)
. But theopen(..)
function on Windows is a POSIX extension and usesCreateFile
internally as well.As we've seen above NodeJS uses
open(..)
, but since we know that this is just a wrapper forCreateFile
, let's check out that function:CreateFile
has a parameter calleddwShareMode
. A file that is opened withdwShareMode=0
cannot be opened again until its handle has been closed.So if you use
open(..)
on a file that was already open by another process withCreateFile(…, dwShareMode=0)
you receive this error message:On the other hand, if you use
fs.openSync
in NodeJS, oropen(..)
in C/C++, to open a file that hasn't been opened yet, you cannot prevent another application from modifying it*.* Unless you use file permissions as a workaround, but that’s not really a file lock.
To prove this, you will see that our
fs.openSync
call executesCreateFile
with the read/write shared flags to comply with the POSIX standard.This means on Windows you cannot prevent another application from opening and modifying your file if you don't use
CreateFile
.What does this have to do with Snowtrack?
Imagine a user saving a big file in a graphic application and while the file is still being written to disk, the user attempts to commit the file change. How does Snowtrack deal with this?
As we learned,
open(..)
has no file locking and most applications don't even follow the file protocol and Snowtrack cannot control how Photoshop, Blender, and co. open and write their files.This means the only reliable chance of detecting if a file is still being written by another process is to check prior to a commit if any process on the system has a write handle on that file.
On Windows, I solved this with a custom helper process and the Windows API of Restart Manager which is mainly used for installers to ensure the files it is about to replace are not open anymore.
On MacOS I invoke the system process
/usr/sbin/lsof
(list open files) with an inclusion of the working-directory to speed up the execution of this command.Additional resources
Beta Was this translation helpful? Give feedback.
All reactions