Trouble defining comments in Liquid templating language

[hankthetank27]

I'm new to tree sitter, and have been working on implementing a forked grammar for the Liquid HTML templating language. I've had reasonable success defining many of the language constructs, but have encountered a lot of trouble with the comments.

In the language you declare a comment if the following way:

{% comment %}
    comment content goes here
{% endcomment %}

I'd like to implement it by consuming all charters between the tags, something like this...

  _tag_delimiter_open: (_) => choice("{%", "{%-"),
  _tag_delimiter_close: (_) => choice("%}", "-%}"),

  _paired_comment: ($) =>
    seq(
      $._tag_delimiter_open,
      "comment", 
      $._tag_delimiter_close,

      repeat(/./),

      $._tag_delimiter_open,
      "endcomment", 
      $._tag_delimiter_close,
    ),

...which clearly does not work correctly because once tree sitter encounters any _tag_delimiter_open token after repeat(/./), it consumes it and subsequently errors on identifying any non "endcomment" token.

For example:

    {% comment %}
      {{ data }}
    {% endcomment %}

    (template
      (comment)
      (content))

    {% comment %}
      {% render "file" %}
    {% endcomment %}

  (template
      (comment
        (ERROR
          (identifier)
          (identifier)))
      (content))

I've tried several approaches regarding fixes, including wrapping the "endcomment" token and its delimiters as a token with a higher precedence, and trying do replicate something similar to how the Ruby tree sitter grammar handles multi-line comments (which quickly became very unwieldy with trying to handle the delimiter options and spaces) to no avail.

Would implementing a scanner be the best approach to cracking this? I'm open to any suggestions or ideas as I don't really know what I'm doing too well :p

[mingodad]

What about using a regex like this /{% comment %}.+?{% endcomment %}/ ?

[hankthetank27]

Thanks for the suggestion! Unfortunately I could not get a regex such as this one to perform as desired. Having to handle an arbitrary amount of whitespace and multiple delimiter patterns complicated things too much. I ended up going with an external scanner which seems to be working well. Cheers.