Using tree-sitter to assist LSP completion

[Ce11an]

Hey 👋

I hope it's ok to ask questions here!

I am working on a creating a tree-sitter for SurrealQL the query language of SurrealDB. I have started the grammar. It is very much a work in progress. While working on the grammar, I wanted to see if I could use my newly created tree-sitter as a parser for an LSP I am building. I am trying to implement completion and failing miserably (this is my first time with tree-sitter and LSPs 😆 ). Does anyone have any advice? I have some examples of what I am trying to achieve:

"X"  is a place holder for where the cursor will be:

1. SELECT X

Completion list would be ["VALUE"]

2. SELECT amount FROM payments X

Completion list should be ["ORDER BY", "WHERE", "SPLIT", "GROUP BY"]

3. SELECT amount FROM payments:cellan X

Completion list should be ["ORDER BY", "WHERE", "SPLIT", "GROUP BY"]

Essentially, I want to get the closest previous node to the cursor. I can map the node to the values I want to populate for completion. However, when I parse the code the node kind for value X "source_file". Do I need to walk the cursor, should I query the tree, or should I take a completely different approach? I am not sure if it is the way my grammar is set up or I am going about this the wrong way.

Any advice would be appreciated! 🙏🏻

[WillLillis]

I added some rudimentary completion capabilities to asm-lsp using tree-sitter's rust bindings. The basic building blocks for this were:

• Maintain an in-memory copy of the source file being edited. This copy can be updated in step with edits made to the source file in two ways:
  • Incremental -- The LSP client sends edits over that you then apply to your copy. If you're writing the LSP in Rust the lsp-textdocument crate makes this fairly straightforward
  • Full -- The client sends a full copy of the document each time it's edited. This is much simpler but also comes at a performance cost, especially as the documents grow larger.
• Alongside the in-memory copy of the source file, maintain a tree-sitter tree. If you're going with incremental document synchronization, you can also take advantage of tree-sitter's incremental parsing by applying edits to the tree and reparsing, rather than reparsing the entire tree each time an edit is made.
• Each time a Completion request is issued to the LSP server, the program would then issue a query checking for nodes under/nearby the current cursor position that the server is able to provide autocomplete suggestions for (in the case of asm-lsp, this is currently just registers and instructions). Depending on how you write your grammar, one way you could approach this is to make all of the X's from your example as children of the parent nodes (e.g. SELECT, FROM, etc.). If your cursor overlaps with a child field's location, you can offer suggestions based on the node type or textual content of the corresponding parent.

To give some more concrete info, in asm-lsp:

• The completion logic is all computed here
• The update logic is handled here

Hope this is helpful :)

[Ce11an]

suggestions

Wow, that's a detailed response - thank you! I've been trying to walk the cursor find the relevant matches, but I'll try to use a query now 🙌🏻

In your opinion, would an LSP for a SQL like language be primarily trigger based, i.e. if certain keywords are written followed by a whitespace? I'll definitely go through your examples for inspiration ✨

[WillLillis]

I've been trying to walk the cursor find the relevant matches, but I'll try to use a query now 🙌🏻

Yeah, when I started I initially tried manually traversing the tree too, but found the query API to be significantly easier. I would also highly recommend the cli tool's playground feature for testing out queries for your grammar.

In your opinion, would an LSP for a SQL like language be primarily trigger based, i.e. if certain keywords are written followed by a whitespace?

When I was working on completion, I worried less about how the requests were triggered and more of just shaping the program around how it would respond to Requests and Notifications from the LSP spec. It was a lot easier to rely on tree-sitter to determine when/how to provide completions based on the source file at a syntactic level, rather than a textual (with the small exception of trigger characters). There's definitely a lot of ways you could do this, but I would recommend starting by trying to match against certain node patterns with a tree-sitter query and then offering completions from that point. For a quick example pulling from your grammar, a simple query could be

(source_file
  (select_statement
    (select_clause)
    (from_clause) . (_)?))

If the query matches and the cursor column position is past the end of the (from_clause) node, you could return a completion list for all of your optional clauses (e.g. WITH, WHERE, etc.).

[Ce11an]

Thanks for the advice 😮 I managed to get the playground working via the CLI, should have done this from the start - so nice! I will play around with this query more when implementing it into the LSP 💯