history merge has mmap issues with certain remote filesystems

[HAOCHENYE]

Fish version

fish, version 3.7.0

System information

> uname -a
Linux xxx #1 SMP Thu Nov 8 23:39:32 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

> echo $TERM
xterm-256color

My problem

I seem to have encountered the issue mentioned in this GitHub issue. I'm a user of Slurm, and my daily working directory is mounted on the Slurm's management node in a similar way to NFS. Similarly, my fish history file is also stored in the mounted directory.

Recently, while experimenting with fish, I noticed that fish's history doesn't seem to work properly. Since I'm just a beginner with fish and my understanding of Linux isn't as deep as some others, please allow me to present my issue through a video.

The process in the video roughly goes like this:

1. I created two terminals, a and b, and both entered fish shell.
2. Checked history and typed hello fish in terminal a.
3. Checked the history file and found that hello fish was properly recorded.
4. Checked history in terminal b and confirmed that hello fish was not there.
5. Called history merge in terminal b, but hello fish still didn't appear in the history.
6. Manually edited the history file and saved it. Then, called history merge again in terminal b and found that hello fish appeared.

20240410-033016.mp4

According to my understanding, I should have been able to find hello fish in terminal b's history in step 5, but it didn't happen (didn't notice the update of the history file? Not sure of the exact reason). When I manually edited the history file and saved it, then called history merge, the result was normal.

I'm not sure if the cause of this phenomenon is the same as mentioned in this issue. Looking for an answer.

[HAOCHENYE]

Um, I think I know where the problem lies now. This switch-case:

fish-shell/src/path.cpp

Line 120 in 80394ea

switch (static_cast<unsigned int>(buf.f_type)) {

does not correctly handle my storage type, which leads to subsequent use of mmap. Our storage returns a special f_type, which is not included in the enumerated data.

[krobelus]

great work, can you share the file type magic number? Or the relevant line from /proc/mounts?
Perhaps we can find a better approach.
Does df -l path/to/your/file work? Their heuristic is here: https://github.com/coreutils/gnulib/blob/master/lib/mountlist.c#L232-L236

[HAOCHENYE]

great work, can you share the file type magic number? Or the relevant line from /proc/mounts? Perhaps we can find a better approach. Does df -l path/to/your/file work? Their heuristic is here: https://github.com/coreutils/gnulib/blob/master/lib/mountlist.c#L232-L236

The type of the magic num is fea36969, which is extracted by:

#include <iostream>
#include <sys/statfs.h>

int main() {
    const char* path = "/mnt/hwfile/"; 

    struct statfs fs_info;

    statfs(path, &fs_info);
    unsigned int fs = static_cast<unsigned int>(fs_info.f_type);
    std::cout << std::hex << fs << std::endl;

    return 0;
}

The result of df -lh path/to/your/file is:

Filesystem      Size  Used Avail Use% Mounted on
/hwfile01       850T  755T   96T  89% /mnt/hwfile

Hmm, it seems quite challenging to differentiate remote file systems through simple operations, as various providers' file systems don't adhere to strict standards (perhaps innovation?). I think it would be simpler if Fish could offer a toggle to control whether to enable mmap, or allow users to configure whether it belongs to a remote FS.

[krobelus]

It seems we should do

diff --git a/src/path.rs b/src/path.rs
index 92ecefd1a..97a1b8ff8 100644
--- a/src/path.rs
+++ b/src/path.rs
@@ -660,7 +660,7 @@ fn path_remoteness(path: &wstr) -> DirRemoteness {
                 => DirRemoteness::remote,
             _ => {
                 // Other FSes are assumed local.
-                DirRemoteness::local
+                DirRemoteness::unknown
             }
         }
     }

We'll still use flock for unknown remoteness but won't mmap the file.

We can still return DirRemoteness::local for some most file systems.
Or maybe we can check if mmap fails somehow.. I guess no.

[krobelus]

But still, if there are meaningful difference between remote and local filesystems, I don't think applications should be required to list all possible local filesystems.

I'm not sure if we want to stop using mmap.
(I'm also not sure what's the problem with using it on NFS.)

Maybe we can detect that mmap is failing and then fall back to reading the file?
How does the problem manifest for you? Any reproducer? If mmap returns MAP_FAILED, we could simply do

diff --git a/src/history/file.rs b/src/history/file.rs
index 489bc83cc..f63783277 100644
--- a/src/history/file.rs
+++ b/src/history/file.rs
@@ -121,7 +121,7 @@ impl HistoryFileContents {
         let len: usize = file.seek(SeekFrom::End(0)).ok()?.try_into().ok()?;
         let mmap_file_directly = should_mmap();
         let mut region = if mmap_file_directly {
-            MmapRegion::map_file(file.as_raw_fd(), len)
+            MmapRegion::map_file(file.as_raw_fd(), len).or_else(|| MmapRegion::map_anon(len))
         } else {
             MmapRegion::map_anon(len)
         }?;

I don't know how else it could fail, you don't get garbage data, right?

[HAOCHENYE]

But still, if there are meaningful difference between remote and local filesystems, I don't think applications should be required to list all possible local filesystems.

I'm not sure if we want to stop using mmap. (I'm also not sure what's the problem with using it on NFS.)

Maybe we can detect that mmap is failing and then fall back to reading the file? How does the problem manifest for you? Any reproducer? If mmap returns MAP_FAILED, we could simply do

diff --git a/src/history/file.rs b/src/history/file.rs
index 489bc83cc..f63783277 100644
--- a/src/history/file.rs
+++ b/src/history/file.rs
@@ -121,7 +121,7 @@ impl HistoryFileContents {
         let len: usize = file.seek(SeekFrom::End(0)).ok()?.try_into().ok()?;
         let mmap_file_directly = should_mmap();
         let mut region = if mmap_file_directly {
-            MmapRegion::map_file(file.as_raw_fd(), len)
+            MmapRegion::map_file(file.as_raw_fd(), len).or_else(|| MmapRegion::map_anon(len))
         } else {
             MmapRegion::map_anon(len)
         }?;

I don't know how else it could fail, you don't get garbage data, right?

Let me try this out to see if it works. I'm not really familiar with Rust. Can you give me some simple instructions on how to build from source using Rust?

[krobelus]

you can use something like

mkdir build
cd build
cmake .. -DCMAKE_INSTALL_PREFIX="$HOME"/.local &&
make &&
make install

[HAOCHENYE]

@krobelus, I've updated this line and built fish from source, but the issue persists. It seems like the mmap function isn't working as expected, rather than just 'Failed'.

Here are some discussions related to nfs and mmap:
"Hey @krobelus, I've updated this line and built fish from source, but the issue persists. It seems like the mmap function isn't working as expected, rather than just showing a 'Failed' message.

Here are some discussions related to nfs and mmap:

• [SOLVED] msync() and NFS isn't working as thought (linuxquestions.org)
• 151284 – mmap of file over NFS corrupts data (redhat.com)

Perhaps the problem lies in the improper configuration of the nfs filesystem."

Perhaps this problem lies in the improper configuration of the nfs filesystem.