How to deal with member access "." preceding whitespace

[Pyromuffin]

I have a problem with my grammar where if you were to type

name : string;
name.


other_thing = 1;

it will parse that as a valid member access expression: name.other_thing. Which it is. And it would be if I were to leave the code in this state, however, when I press "." after name, I'm generally looking to get the member list, not screw up my entire program by potentially creating a member access between two really weird things.

for instance:

name : string;
name.


Person :: struct
{
 // etc
}

now makes this try to do member access between name and Person, which stops Person from being a valid struct syntactically, causing a cascade of errors everywhere i've used Person.

the rule is what you would expect:
member_access: $ => prec.left(seq($.expression, ".", $.expression)),
I've tried making member access use token.immediate(".") but it seems to only matter for the preceding whitespace, not the whitespace following the rule.

ok: expression. expression
not ok: expression .expression

So, I am wondering if there is something I can do to get tree sitter to error out when whitespace follows the "."

I'd appreciate your thoughts on this problem.

Thanks,
Kelly

[issue-label-bot[bot]]

Issue Label Bot is not confident enough to auto-label this issue. See dashboard for more details.

[maxbrunsfeld]

We've introduced the Discussion board as a place for these types of questions, so I transferred this issue there.

[maxbrunsfeld]

Yeah, as you said, token.immediate affects the preceding whitespace, not the following whitespace. Some thoughts:

• Can it really be an aribitrary expression after the dot? I would think you would want something more specific there, like an identifier.
• Assuming it is something more specific, you could have a special "member name" rule for the part after the ., which uses token.immediate. If you want, you could still alias it so that it matches the old rule's name.
• More generally, you could add an external token called somethign like $.no_whitespace, and use it after the .. You'd have to add some code in an external scanner that checks that there's no whitespace, and returns a zero-length token. We do something like this in the bash grammar. See here and here.

[Pyromuffin]

thanks for the ideas. member access can probably be between two identifiers/subscript_expressions, but this programming language is really uh, flexible and you can use arbitrary expressions in a lot of places you wouldn’t expect, so i generally default to using expression instead of more specific things because i have been bitten by that in the past. unfortunately the language

1. doesn’t have an official grammar spec
2. is still in development (not by me)

so it’s hard to say what is exactly right. if i can express the rhs as a terminal, then yes, i could use token.immediate() on that. but i think that is probably not going to work out because of at least the possibility of subscripting. i already have an external scanner for herestrings, so i’ll look into what the bash grammar does.

it’s really a shame that i need to hack the parser up for something that is really a UX related thing. it’s possible that i could detect this white space while iterating through the syntax tree (by finding the “.” token in the text buffer and then checking the adjacent characters), but that still leaves the post-whitespace mangled syntax tree as a problem for the rest of the document.

[maxbrunsfeld]

Yeah, ideally Tree-sitter would just recover from the error in the most intuitive way possible without you having to change anything. But it's tricky because, with the current grammar, we can't detect the error until we see the :: token after Person. Maybe in the future we will work more on this, possibly even adding functions in the DSL to let you control error recovery preferences. That's a bit far off though.

[Pyromuffin]

I think you're right. it comes down to controlling error behavior. A similar thing happens with a dangling binary expression.

a +

Person :: struct
{

}

if the error occurred on the first line and left the rest of the program parsing correctly, then this wouldn't be a problem. I've solved the issue (poorly) for the moment by implementing a "member_access_nothing" expression which is just seq($.expression, ".", choice(" ", "\n")) , and it seems to be holding up, but I am not extremely confident about this.

the dangling binary expression could be solved similarly, but it is less urgently in need of solving because a generic binary expression doesn't typically require the immediate result of an ideally parsed syntax tree.

[Pyromuffin]

it also might be due to the error metric, ``a +``` is two errors, missing an expression and missing ";" for the end of the expression_statement. Where a + struct_decl is maybe less errors total, and therefore a more correct parse?

[Pyromuffin]

I am now running into a worse problem with function-calls-in-progress.
my hack of checking for "\n" and emitting a "function_call_in_progress" node is not working because multiline function calls are not that uncommon. If you have suggestions on how to redesign the grammar so that I can error out in the function call instead of way after it trashes the entire file, i would appreciate that.

here's a fragment of grammar for the function call. I'm not really an expert at parser grammars, so i've done it in the most obvious way.

      argument_name: $ => seq(
        $.identifier,
        "="
      ),

      argument: $ => prec(1, seq(
        optional($.argument_name),
        $._expression,
      )),

      _argument_list: $ => prec(1, seq(
        "(",
        CommaSep($.argument),
        ")",
      )),

      func_call: $ =>seq(
        $._expression,
        $._argument_list,
      ),

Do you know of any examples using tree-sitter for as-you-type parsing? I'm starting to think I might have to use a technique outside the parser to account for these intermediate states. like keeping a history of most recent error-free parses, or using hysteresis.

[Pyromuffin]

it seems that atom (which presumably uses tree-sitter) also has this problem, at least with its C grammar. To get around around this problem, atom and most other IDEs i have tested automatically inserts closing parens on typing a function call.

this fixes the problem on editors where this feature is available.