Help interpreting seemingly infinite loop and out-of-memory error on fuzzer input

[ahelwer]

I set up fuzzing for the tree-sitter-tlaplus grammar and it quickly found an input where the parser seemingly enters an infinite error recovery loop before eventually running out of memory. Here is the input (I will avoid marking it as a tla code block to keep github happy with me):

TA~@(AA+/\

Here's a 3MB file full of tree-sitter debug parse output on this input: out.txt

Representative sample:

process version:0, version_count:2, state:2213, row:0, col:12
lex_external state:4, row:0, column:12
  consume character:'T'
lexed_lookahead sym:_dedent, size:4294967284
detect_error
process version:1, version_count:2, state:2209, row:0, col:12
lex_external state:4, row:0, column:12
  consume character:'T'
lex_internal state:195, row:0, column:12
lex_external state:1, row:0, column:12
lex_internal state:0, row:0, column:12
lexed_lookahead sym:end, size:0
detect_error
resume version:0
recover_with_missing symbol:identifier, state:4282
skip_token symbol:_dedent
process version:0, version_count:2, state:0, row:0, col:0
lex_external state:1, row:0, column:0
lex_internal state:0, row:0, column:0
  consume character:'T'
  consume character:'A'
  consume character:'T'
  consume character:'A'
lexed_lookahead sym:identifier, size:2
recover_to_previous state:2213, depth:2

It seems like it's perpetually recovering to the very start of the text and never makes any progress. Does this seem actionable? As in, is there any external scanner behavior that could cause this? Or is it more likely to be a bug in tree-sitter's error recovery process?

Possibly relevant:

• One of my external scanner tokens is enabled at the very start of the file (which might be rare among tree-sitter grammars).
• Related? Stack overflow with tree-sitter 0.20.8 and 0.20.9 (but works with 0.20.7) #2071
• The external scanner does emit an empty token upon seeing /\, and also modifies its state, so it is allowed during error recovery by Allow empty external tokens during error recovery, if they change the scanner's state #1783; however, I look for whether all tokens are marked valid in my external scanner (and thus we are in error recovery mode) and this does not seem to be the case at that time

[amaanq]

I think it's the same as this

tree-sitter/tree-sitter-python#207