Are highlighting queries in tree-sitter-XXX repositories considered canonical?

[guillaumebrunerie]

For every language, there are highlighting queries being maintained in the tree-sitter-XXX repositories. Is the idea that those queries should be considered "canonical", and that everyone using tree-sitter for highlighting should use them?

I'm asking because for instance there is a repository maintaining highlighting queries for neovim (https://github.com/nvim-treesitter/nvim-treesitter/tree/master/queries), and another one for Emacs (https://github.com/emacs-tree-sitter/tree-sitter-langs/tree/master/queries). It feels like a lot of duplicated work to maintain (presumably very similar) highlighting queries in several different places.

In particular, the Emacs repository justifies it in the following way:

Highlighting query patterns for a language are in the file queries/<lang>/highlights.scm. Most of them are intentionally different from those from upstream repositories, which are more geared towards GitHub’s use cases. We try to be more consistent with Emacs’s existing conventions. (For some languages, this is WIP, so their patterns may look similar to upstream’s.)

In general, try to follow what the docstrings of tree-sitter-hl-face: faces say. Most importantly:

• Definitions and uses should be differentiated:
  • @function vs. @function.call.
  • @method vs. @method.call.
  • @type.parameter vs. @type.argument.
• @variable and @variable.parameter should be applied only to declarations/definitions/bindings/mutations (writes), not usage (reads).
• Special faces should have high priority (placed earlier in the pattern list): @function.macro, @type.builtin, @variable.special.
• Patterns whose internals may be highlighted should have low priority (placed towards the end). Example: strings with interpolation.

What do people working on tree-sitter think?

I'm using Emacs and I would like to contribute fixes to highlighting queries upstream, but I'm not sure where upstream is…

(tagging @amaanq, as we interacted recently in the Javascript/Typescript grammars, and I see that you recently contributed to the neovim repository)

[ahelwer]

Queries are how you integrate a client with a specific tree-sitter grammar, so each client (neovim, emacs, etc.) owning the queries they use to integrate with each grammar makes sense. Different clients support different capture names and predicates. The queries you see living in each tree-sitter repo are only consumed by the tree-sitter highlight command and github itself (for the few languages it highlights with tree-sitter), as far as I know. They also serve as useful examples to adapt to different clients.

Highlighting query files are not a very large amount of work to make and might take a day or two at most for a very complicated grammar. If new clients want to reuse existing work they can choose to support the same capture names and predicates as for example neovim.

It's not really accurate to call the queries living in each tree-sitter repo canonical, but for my own grammar I do maintain a sort of correspondence between them and queries in the various supported clients.

[amaanq]

For every language, there are highlighting queries being maintained in the tree-sitter-XXX repositories. Is the idea that those queries should be considered "canonical", and that everyone using tree-sitter for highlighting should use them?

Those that are owned by tree-sitter are meant for usage w/ the highlight subcommand in the cli iirc, and the captures are a bit simpler.

Now for Neovim, one of the main differences between nvim-treesitter/neovim core and tree-sitter is that our queries are last-wins, or basically, all captures are applied and the latest + most inner capture is shown, e.g. if we have a tree like (field (identifier)), where the entire field is captured @field and identifier @variable, @variable would always win. But if we had (field (identifier) @field), then the order of the captures dictate who wins (last). Our captures are also a bit more fine-tuned than upstream, though Neovim has expressed interest in collaborating & sharing queries. More relevant info is in this thread: nvim-treesitter/nvim-treesitter#3944. Fwiw, all my own grammars for less-common languages use the nvim-treesitter query files, as it's easier for me to work with for nvim-ts/neovim, and has a lot more coverage w/ the captures

Emacs is also a wildcard which we know very little about in terms of captures, since they seem to embed the query-like syntax in Emacs-lisp code itself (would be easy enough and sort-of makes sense given the lisp-likeness, I guess)

I for one would love to standardize a broader set of captures that make sense for all editors, so that a majority of queries can be shared - Max himself has agreed that last-wins is a better system for queries while working on the Zed editor, but currently upstream and Helix use first-wins/a short circuit logic. I plan to add a flag for the highlight crate to allow a "last-wins" precedence soon, something like "--apply-all-captures" probably.

Upstream for you is probably Emacs, if you think tree-sitter would also benefit from lacking captures that's welcome as well (just peruse some of the repos queries for the style/standard)

[sogaiu]

I think there are actually (at least) 2 Emacs situations.

There is the Rust-based effort for Emacs 28.x and below -- in this case, IIUC, usually highlight query files live as separate files within subdirectories here.

For Emacss 29.x (still to be released I believe) and above where there is native support for tree-sitter, the convention ATM appears to be one of embedding query content within major mode <lang>-ts-mode.el files (as mentioned by @amaanq) like this.

[guillaumebrunerie]

Thank you for the link @amaanq, interesting discussion!

I think there are actually (at least) 2 Emacs situations.

I see, I haven’t looked at Emacs 29 at all yet, so I only know the Emacs 28 situation which uses regular query files. Although some languages don’t seem to be given much love there, for instance the last update to the Typescript grammar was 2 years ago (!). I've actually made my own JS/JSX/TS/TSX mode (https://github.com/guillaumebrunerie/ultimate-js-mode) with my own queries, as there was nothing satisfactory at the time. Hopefully I'll be able to use the built-in mode in Emacs 29 when it will be released.

I'm a bit surprised by the embedding of queries in the major modes in Emacs 29, though, it seems like it would make it harder to share/reuse queries.

[sogaiu]

Although some languages don’t seem to be given much love there, for instance the last update to the Typescript grammar was 2 years ago (!).

I don't know specifically about the reasons behind the support status for that grammar, but there is a kind of blocking situation which IIUC stalled things a bit for elisp-tree-sitter (the Rust-based one): #1294

At least in my case, this as well as the development of the newer Emacs 29+ integration gradually reduced the amount of time and attentiom I paid to integrating with the Rust-based effort.

I'm a bit surprised by the embedding of queries in the major modes in Emacs 29, though, it seems like it would make it harder to share/reuse queries.

That may be so. It's not all downsides though because one can programmatically manipulate things. A trivial example is this where something defined elsewhere using Emacs Lisp is involved in construction of the overall highligting query [1].

Note as @ahelwer mentioned:

Different clients support different capture names and predicates.

so the idea of complete sharing of queries is not really practical, though it might still be a worthwhile endeavor for some grammars and client combinations.

---

[1] See the <lang>-ts-mode.el files here for more examples if interested.

[guillaumebrunerie]

Thanks! I didn't know about this issue, I managed to use the latest versions of the grammars with elisp-tree-sitter, but I had to rebuild the parsers with tree-sitter generate --abi 13, I guess that's because elisp-tree-sitter is stuck with an older version of tree-sitter.

I hope it will still be possible to customize the queries in Emacs 29? I could already see several things in the Typescript highlighting queries that I will want to change.

[sogaiu]

I hope it will still be possible to customize the queries in Emacs 29?

I don't know what the status is regarding this (haven't checked the GNU Emacs development mailing list recently -- may be it's worth asking there?), but there is some related discussion here for a case of a mode that is being developed outside of the Emacs development tree.

[Drozdddd1]

Thanks you