Error nodes produced even though other match seems possible

[bbannier]

I am trying to parse a language with identifiers and also literals.

b
b""

Here b is an indentifier, and b"" is a literal (also possible would be e.g., b"foo").

I have the following simplified grammar:

module.exports = grammar({
    name: 'foo',

    rules: {
        file: $ => repeat(choice(
            $.ident,
            $.literal,
        )),

        ident: _ => /[a-z]/,
        literal: _ => seq('b', /\"[a-z]*\"/),
    },
});

This produces the following match state:

(file [0, 0] - [2, 0]
  (ERROR [0, 0] - [0, 1])
  (literal [1, 0] - [1, 3]))
foo.bar	0 ms	(ERROR [0, 0] - [0, 1])

I.e., I am able to parse constructs like b"", but the matcher for ident goes on an error branch and does not recover.

I have experimented with token and prec without success. What is missing here?

---

I had previously asked this question on Stackoverflow, but here seems to be a better place.

[maxbrunsfeld]

I think you may want the ‘b’ prefix to be part of the same token as the string, instead of being a separate token. Because if it’s a separate token, the lexer doesn’t have a way to distinguish it from an identifier.

[bbannier]

Thanks for nudging me in the right direction @maxbrunsfeld. With the following change this toy example does indeed work:

-         literal: _ => seq('b', /\"[a-z]*\"/),
+         literal: _ => token(seq('b', /\"[a-z]*\"/)),

(file [0, 0] - [2, 0]
  (ident [0, 0] - [0, 1])
  (literal [1, 0] - [1, 3]))

If the matcher for the quoted part of literal becomes more complicated, this like expected runs into the usual issues around non-terminals, but this is a feature of token, not the example I posted, e.g.,

        ident: _ => /[a-z]/,
        literal: $ => token(seq('b', $.ident)), // WRONG.

Error processing rule literal
Details:
  Grammar error: Unexpected rule Symbol(Symbol { kind: NonTerminal, index: 1 })