Exclusive literal tokens for external scanners

[ahlinc]

Current implementation has behavior that if an external scanner doesn't return a token and the token defined in the grammar.js externals as a string or a regexp than there works a fallback mechanism and an internal scanner of tree-sitter's lexer will rescan a piece of input again and can match the token.

In some situations the above may be an undesired behavior:

1. Internal lexer's scanner don't know for what reasons the external scanner may recognize the token but decided to reject it due to unmatched other conditions like incomplete surrounding token context or own internal state managed through serialize/deserialize.
2. It is better to fail fast what leads to the lexer will just drop unexpected small piece of input and this will make smaller error in input stream.
3. This may be helpful in developing and debugging process of an external scanner when there is one source of decision about tokens.
4. No need to use a workaround to replace such tokens to a named tokens analog in the externals list and through all the grammar, what leads to more compact and visually clear and understandable grammar rules.

Examples

In the next grammar example interpolationStart may be matched only by the external scanner and the internal lexer's scanner will don't do anything from it's side and only see a fact that the token are missing or recognized by the external scanner.

module.exports = grammar({
  externals: $ => [
    $.interpolationStart,  // "${"
    "}",
  ],
  rules: {
    interpolation: $ => seq($.interpolationStart, $.body, "}"),
    body: $ => ...,
  }
});

In the next situation if the external scanner returns false and there is ${ at a current input stream position (or nearly if there is
some wrong unexpected input that may be skipped):

module.exports = grammar({
  externals: $ => [
    "${",
    "}",
  ],
  rules: {
    interpolation: $ => seq("${", $.body, "}"),
    body: $ => ...,
  }
});

Such behavior helps to write less code in the external scanners if some conditions don't meet.

---

What I suggest is to introduce a special marker on the grammar that will mark such simple tokens defined like "${" that they are exclusive for external scanner and fallback to detection through internal lexer's scanner shouldn't happen for them:

module.exports = grammar({
  externals: $ => [
    exclusive("${"),   // The exclusive(token) is proposed DSL function that will mark tokens
                       // by setting an additional flag in the json representation of the grammar.
    "}",
  ],
  rules: {
    interpolation: $ => seq("${", $.body, "}"),
    body: $ => ...,
  }
});

This will allow to continue to use literal representation of such tokens through the grammar rules, what will keeps it simpler for understanding.

[ahlinc]

@maxbrunsfeld could you, please, tell what do you think about above suggestion?

[maxbrunsfeld]

To me, it does not seem worthwhile to add a new API for this purpose. As you said, you can already express this behavior using your first example. I think each API function that we add increases the potential for confusion, so I have tried to avoid any functions that aren't strictly required to implement a certain parsing behavior.

[ahlinc]

@maxbrunsfeld okay, I understand your point but would it be acceptable at least to add a json level flag to a symbol that will mark symbol as exclusive, so such action may be easily added by end users if them know what are they doing?

[ahlinc]

After half of the year with Tree-sitter I can add that:

• There is an ability to rename an external token to named or anonymous token with an alias(..., $.new_name) command like:

  alias($.interpolationStart, "${")