Fish autocomplete sometimes forgets history and goes back in time

[jacobsa]

I'm using fish 3.6.4 with this configuration. Typically I run fish within tmux, with a few different tmux windows open at a time.

I find that a few times per day, fish seems to forget history for commands I run all of the time. For example, I run a command of this form many times per day:

foo; and bar; and baz

Normally when I typ fo, fish will automatically complete the rest of it and I can press right, then enter, to run it. But a few times per day one instance of fish will totally forget it, and auto-complete a broken typo version I ran months ago:

foo; and basdfg; and baz

If I switch to other instances of fish running in other tmux windows, they don't have this problem: they still autocomplete correctly. If I close the tmux window and open a new one, the new instance of fish is fine. Even if I run fish to get a new sub-shell within the broken instance, it also uses the correct history. It's only the one busted instance of fish.

To be clear, it's not just one command. This issue happens with several different commands that I run frequently, and it seems like when fish gets into this state it's all recent history that is lost, not just one command.

It's almost as if there is some in-memory circular buffer of history, and it eventually overflows when I run enough commands, and then fish is confused and thinks old history is current.

How can I debug this issue? Is there anything I can collect to help you understand it better?

---

> fish --version
fish, version 3.6.4

> uname -a
Linux <redacted>.com 6.5.13-1rodete2-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.5.13-1rodete2 (2024-01-11) x86_64 GNU/Linux

> echo $TERM
screen-256color

> tmux -V
tmux 3.3a

[krobelus]

Is this on NFS?
When it happens, is there a file like ~/.local/share/fish/fish_history.2NsTOE that matches the history of the broken session? To compare, maybe dump the entire history with history.
Does history merge unbreak it?

[jacobsa]

It's not NFS, no. There is currently no file in ~/.local/share/fish/ but fish_history itself, but I'm not sure if it's expected to persist after the broken session.

What does history merge do? I can try it next time I see the problem.

[krobelus]

What does history merge do? I can try it next time I see the problem.

It incorporates new history from other, newer shells; the crucial side effect is that it will reload the history file.
If it doesn't, you might need to run some command in another shell first, so history merge will actually do something.

[jacobsa]

Okay two updates:

• If I run history | grep foo | head when a shell is in this state, it also agrees that the history is busted. It shows only the old typo version, or at least the more recent version doesn't show up in the first ten entries. Trying this in a new tmux window shows the correct history.

• history merge does seem to restore the shell to a working state.

Any ideas what is causing the shell to get into this state in the first place? There is no network file system involved, and it's not like it starts with bad history; it just becomes bad eventually.

[faho]

What does your system clock say, i.e. date?

[krobelus]

The missing commands are from the current shell, right? Or from previous shells? (Probably not both)

[jacobsa]

Definitely from previous shells; these are commands I run every day and have done so for years. I think they usually were probably also run in the current shell before they were forgotten, but I'm less sure about that.

date shows the correct current local time, in both the busted shell and a new one.

[krobelus]

This is a really mysterious bug, thanks for taking the effort to report. Maybe there's a simple explanation like you set fish_history or fish_private_mode.. but those can't be it because history merge works.

When we create a history, or run history merge, we record the current system time.
Every command from the history file that has a later timestamp will be excluded, because those commands are assumed to come from other shell instances.

Maybe there is some memory corruption that stomps the timestamp, setting it to some time in the past. That would explain why fish suddendly starts ignoring those.

We probably need to add some logging.

Can't hurt to try latest Git master as well, although I don't expect a change.

[jacobsa]

Thanks for being attentive to my weird bug. :-) If you want to add some logging and tell me how to check it, I'd be more than happy to report back.

---

There was one other theory I had, as someone who doesn't know fish's internals. How well is the plain text history format able to "re-synchronize" after corruption? Like if a line gets deleted or messed up or duplicated or something, are all of the entries after that liable to not work correctly? Or does it re-synchronize by looking for some marker afterward? I have in the past manually deleted entries from the file, and I worry if the format isn't robust it may have left me with a persistent problem if I made any editing mistakes.

[zanchey]

Running with fish --debug history,history-file might help.

[jacobsa]

Thanks, can you help me understand what that does? Will there be debug output somewhere that I can examine, and what am I looking for?

[faho]

Will there be debug output somewhere that I can examine, and what am I looking for?

Yes, there will be debug output. By default it'll go to the terminal, add --debug-output /path/to/file to write it to a file.

I don't think we currently know anything specific to be looking for.

[magnetophon]

I just wanted to mention I have the exact same symptoms.
Let me know if there's anything I can do to help.

[faho]

Alright, some non-exhaustive ideas:

1. What is the output of history when this happens? Do the offending entries turn up there?
2. Is your system clock okay? Does it move?
3. Do you have any configuration? Especially anything history-related, but also e.g. the fzf integration?
4. Does this happen with 3.7.0?
5. Do you have any network mounts?
6. Do you suspend your system?
7. Does fish --debug history'*' --debug-output /tmp/history.txt show anything sensible if this happened? Unfortunately you'll have to run fish with that explicitly.

[ybc37]

I'm also affected and can add that I relatively often delete items from the history (history delete ...). @jacobsa also mentioned that he deleted items (though directly in the file). Maybe that's something to look at?

Edit: Also, I use tmux, so I do have multiple fish processes running. Maybe that in combination with deleting items?

---

What is the output of history when this happens? Do the offending entries turn up there?

I'll try to think about that the next times it happens.

Is your system clock okay? Does it move?

Never noticed issues with my system clock.

Do you have any configuration? Especially anything history-related, but also e.g. the fzf integration?

I do use fzf integration and have two aliases to work with the history.

Does this happen with 3.7.0?

Yes

Do you have any network mounts?
Do you suspend your system?

Both no

Does fish --debug history'*' --debug-output /tmp/history.txt show anything sensible if this happened? Unfortunately you'll have to run fish with that explicitly.

Didn't try that, would have to change my config.

[magnetophon]

I also sometimes delete stuff from my history, just thought i'd mention it.

What is the output of history when this happens? Do the offending entries turn up there?

I'll report back the next time it happens.

Is your system clock okay? Does it move?

Never noticed any issues.

Do you have any configuration?

Yes:
https://github.com/magnetophon/.dot/tree/master/common/.config/fish

Especially anything history-related, but also e.g. the fzf integration?

I use a couple of plugins:

jorgebucaran/fisher
ilancosman/tide@v6
franciscolourenco/done
patrickf1/fzf.fish
meaningful-ooo/sponge
decors/fish-colored-man
jorgebucaran/autopair.fish
lewisacidic/fish-git-abbr

sponge and fzf.fish are the history related ones.

Does this happen with 3.7.0?

Yes it does.

Do you have any network mounts?

No.

Do you suspend your system?

Yes I do.

Does fish --debug history'*' --debug-output /tmp/history.txt show anything sensible if this happened? Unfortunately you'll have to run fish with that explicitly.

I have fish as my system shell in NixOS.
I'd have to figure out how to make the above command my default shell.

[retrotails]

I have this happen as well, it seems to take quite a while. I opened a terminal with debug history, and just let it sit there. The next day, I checked the history in that terminal and it forgot several commands, all of which are commands I ran in other terminals while that one was open. Checking the fish_history file, there's only one entry for each of the missing commands, and they have timestamps of when when I most recently ran them. history --merge always fixes it.
I have deleted history entries before, but the last time was quite a while ago.

1. What is the output of history when this happens? Do the offending entries turn up there?

history is also missing the commands

2. Is your system clock okay? Does it move?

My system time is correct

3. Do you have any configuration? Especially anything history-related, but also e.g. the fzf integration?

No configuration at all, completely default

4. Does this happen with 3.7.0?

I'm on 3.7.0 now, and I've experienced this for a long time.

5. Do you have any network mounts?

Several, none of my commands were run while cd'd into one though.

6. Do you suspend your system?

Never

7. Does fish --debug history'*' --debug-output /tmp/history.txt show anything sensible if this happened?

Not really.

history: Loaded 7181 old items
history: Saving 1 items via appending
history: Loaded 7163 old items
history: Saving 1 items via appending

[faho]

it forgot several commands, all of which are commands I ran in other terminals while that one was open

Fish does not read the history from other sessions after startup unless you run history merge, so that is as expected.

[retrotails]

Fish does not read the history from other sessions after startup unless you run history merge, so that is as expected.

I'm talking about old commands that were in the history, but I ran them again somewhere else after opening the test terminal

[0ion9]

I think I mentioned this bug to krobelus at some point (few years ago?) but did not report. FWIW it is also still happening for me, and I can confirm that the commands in question are 'old commands that were in the history' rather than items recently added by work in other presently-running instances of fish.

List of details relating to questions asked thus far:

• fish-git 3.7.1.g3af849d73

• terminal: Kitty 0.33.0-1

• OS: Arch Linux x86_64

• not using tmux or any other multiplexer

• I do not use suspend or NFS mounts (but do rarely use FUSE mounts, eg archivemount); in any case the history file does not have any relation to these in my case

• I do use ntpd, but as far as I can tell my system clock keeps reliable time, so it shouldn't be incurring large adjustments.

• history is afflicted with forgetfulness in an identical way to autocomplete.

• I do (rarely) delete history items, but only using the history delete interface

• I don't set fish_history or use private mode

• IIRC I have the fzf integration enabled, but don't use it to interact with history

• Filesystem of /home/: ext4, in case that makes any difference to obscure timestamp interaction type things.

• "It's almost as if there is some in-memory circular buffer of history, and it eventually overflows when I run enough commands, and then fish is confused and thinks old history is current." -- I think this is incorrect, at least regarding the problem as it manifests on my system. Often, the problem will occur in a nominally 'fresh' shell (kitty will automatically start several shell instances, and when I eventually switch to the next instance, nothing has been done in that instance at that point but it is clearly afflicted with forgetfulness)

• I have not so far checked the output of date in an afflicted shell instance, but I can do that

• Will try to get logging setup.

[krobelus]

someone needs to look at the code and add relevant logs

[planet36]

I've been increasingly encountering a problem like this one for the past few months.
If manifests after I first log in. When I type history, no history is printed, even though fish_history still exists and contains earlier history.

The last time it happened, I noticed there were embedded NUL bytes in fish_history. (screenshot included)
When I removed the entry containing the NUL bytes, and logged out & in, things worked normally.

$ fish --version

fish, version 3.7.1

$ uname -srm

Linux 6.8.9-arch1-1 x86_64

$ fisher list

jorgebucaran/fisher
jethrokuan/z

$ tmux -V

tmux 3.4

/ is ext4

$ wc --lines fish_history.BAD

141748 fish_history.BAD

[jacobsa]

@planet36 Maybe best to file a different bug for that, so as to avoid derailing this one? It seems like different symptoms (of course the cause could be related in the end).

[planet36]

@planet36 Maybe best to file a different bug for that, so as to avoid derailing this one? It seems like different symptoms (of course the cause could be related in the end).

I just created this issue 10483