Help with assembly grammar

[Timmmm]

Hi, I was wondering if you could help me figure out how to make a grammar for assembly, specifically this assembly. Unfortunately it uses LLVM's assembly parser which supports a dozen different variants and is very ad-hoc. In other words there's no formal grammar. Here's the gist though:

// C-style comments
# Also these comments
; And possibly these

// Labels are C-style
VectorInnerAdd_core_half:
// But you also can make local labels starting them with .L
.Laddend_scalar:
// And there are special integer labels 
0:

// Instructions are terminated by newlines.
nop
shr $stride, $addend_len, 2
// Instruction parameters can be expressions
rpt $acts_block_count, (2f - 1f)/8-1

// There are also directives, which start with a .
.directive
.directive with parameters

// There are "bundles" which are 2 instructions surrounded by braces
  { nop
    fnop }
  {
    ldst64pace $tmp1, $tmp1, $packed_ldst_addrs+=, $stride, 0x05
    f16v4mix $tmp0, $current_addend0, $tmp0
  }

// Finally there is all the C preprocessor nonsense.
EXPORT_FN   CPP_FUNCNAME(Supervisor,SCALED___ADD)

So the issues I'm having are:

1. Newlines are significant, but I'm not sure how to handle that. I had a look at the Python grammar but it uses an external C scanner and it looks pretty complicated and undocumented. Is there any way to handle newlines explicitly in grammar.js?

2. How on earth do you deal with the C preprocessor, given that it means basically anything can appear anywhere?

3. I can't figure out how to distinguish labels, local labels and directives:

foo:
.Lfoo:
.foo

Thanks! Sorry if this is the wrong place to ask!

[maxbrunsfeld]

Newlines are significant, but I'm not sure how to handle that. I had a look at the Python grammar but it uses an external C scanner and it looks pretty complicated and undocumented. Is there any way to handle newlines explicitly in grammar.js?

You can handle newlines in grammar.js too. The Go grammar does this for example. It can work even though newlines can also function as whitespace. The lexer will only produce a \n token in states where that token is valid.

How on earth do you deal with the C preprocessor, given that it means basically anything can appear anywhere?

The C preprocessor is tricky. I think you'll need to approximate it as if it were part of the same language by explicitly modeling certain preprocessor constructs and usage patterns. This is a fundamental limitation of parsing arbitrary source files without knowing the values of compiler flags. Of course, you could also run the preprocessor before parsing your file with Tree-sitter, but usually when people are using Tree-sitter, it's because they want their grammar to be applicable within the limitations of lightweight text editors and other programming tools that operate on raw files.

I can't figure out how to distinguish labels, local labels and directives:

I have given this a super careful read right now, but this seems doable with normal regexes, or normal grammar constructs. Let me know if you continue to have trouble with this, and you are able to narrow it down to a tree-sitter-specific issue or limitation.

[Timmmm]

It can work even though newlines can also function as whitespace. The lexer will only produce a \n token in states where that token is valid.

Ah... very clever. It might be worth explaining this in the docs since it is pretty confusing! Something like this, if I'm understanding it correctly (feel free to copy/pasted):

extras - an array of tokens that may appear anywhere in the language. This is often used for whitespace and comments. The default value of extras is to accept whitespace. To control whitespace explicitly, specify extras: $ => [] in your grammar. Even if you use extras to ignore whitespace you can still have rules that match whitespace (for example newlines). The lexer will match whatever is in your rule first, and only fall back to ignoring whitespace if that rule does not match. See the Go grammar for an example of this.

I think you'll need to approximate it as if it were part of the same language by explicitly modeling certain preprocessor constructs and usage patterns.

Ah I figured as much. I guess it might be worth pointing this limitation out in the introduction?

Of course, you could also run the preprocessor before parsing your file with Tree-sitter, but usually when people are using Tree-sitter, it's because they want their grammar to be applicable within the limitations of lightweight text editors and other programming tools that operate on raw files.

Yeah exactly, and I imagine tracking spans/line numbers would get very complicated!

I can't figure out how to distinguish labels, local labels and directives

I think I figured this out actually. Thanks for all the help!