How to customize sorting of candidates sifted by company-capf?

[kings2u]

Despite studying the code of company-capf, I cannot determine how to write a custom completion-at-point-function that contains the necessary metadata for the company-capf function to take advantage of the company-capf--sorted variable offered in company-capf.el. There is no keyword like :company-docsig or :company-kind as there is with other parts of company-capf, though customizing sorting is clearly possible from what I can see in the company-capf--candidates function. Any advice on how to construct my CAPF would be great!

[dgutov]

There is no :company-* for it because sorting is an original feature of a completion table, as defined in the core.

See the docstring of completion-metadata and display-sort-function in particular. You'll probably want to ask how to assign this metadata field to a completion table -- hopefully the description at the top of lisp/minibuffer.el can answer that. Do you know how to find it?

And yes, it seems unnecessarily complex, but we have what we have.

[kings2u]

Thanks for the pointers! And yes, the critical issue I can’t answer is how to assign the metadata to a completion table in the first place for use in a CAPF. I’m unable to locate medatata.el with find-library or locate. Did you mean minibuffer.el?

The best resource I’ve found on the topic of adding metadata to a table is https://with-emacs.com/posts/tutorials/customize-completion-at-point/ but its example deals with the context of completing-read, not completion-at-point-functions. Another similar resource is here. FYI, I’m trying to combine my own list of possible completion candidates with all those in obarray and sort them from shortest length to longest.

Do you have any suggestions?

[dgutov]

I’m unable to locate medatata.el with find-library or locate. Did you mean minibuffer.el?

Yes. Sorry! I'll edit the answer above.

[dgutov]

Do you have any suggestions?

Here's a concrete example: project--file-completion-table in lisp/progmodes/project.el. It's a function that takes a flat list of strings and returns a closure which works as a completion table, and which reports ((category . project-file)) as its metadata.

[dgutov]

Ah, and the example at with-emacs.com also creates such completion table.

The difference with c-a-p-f is that a completion-at-point-functions element is another function that needs to return a list like:

(START END TABLE ...EXTRA-PROPS)

Where TABLE is a completion table (often a function) which can be defined like described above.

[kings2u]

Thank you for clarifying. With your pointers, and after examining the code for eglot.el, I was able to narrow down the basic template to sort candidates like I wanted. :)

;; -*- lexical-binding: t -*-
(setq test-candidates '("aaaaa" "aaaaaaaaaa" "aaa" "aaaaaaaa"))
(defun test-capf ()
  (let* ((start (save-excursion (skip-syntax-backward "w_")
                               (point)))
         (end (point))
         (sort-completions (lambda (completions)
                            (sort completions
                                  (lambda (x y)
                                       (or (< (length x) (length y))
                                           (and (= (length x) (length y))
                                                (string< x y)))))))
         (metadata `(metadata . ((display-sort-function . ,sort-completions)))))
    (when (and start end)
      (list start end
            test-candidates 
            :company-kind (lambda (cand) 'variable)))))
(setq-local completion-at-point-functions '(test-capf t))

[kings2u]

Perhaps your personal config also adds something

You were right about this. I had this flex-styling snippet in my config, and removing it made the sorting function take effect:

(define-advice company-capf 
    (:around (orig-fun &rest args) set-completion-styles)
  (let ((completion-styles '(flex)))
    (apply orig-fun args)))

However, I’m not sure that tells the whole story since the completion-metadata-get method I described does seem to allow my advice to coexist with my custom sort-completions function:

Aren't those the same? See completion-metadata-get's definition: it just does (cdr (assq prop metadata)). I don't see any difference in behavior after applying this.

Doing the debugging you suggested does suggest a subtle difference: the (sortfun (cdr (assq 'display-sort-function meta))) line produces a sortfun with the byte-coded function value of my sort-completions function, whereas the alternative completion-metadata-get method produces a sortfun value of the plain symbol sort-completions.

Does this information raise a solution as to how I might tailor my sort-completions function while keeping my flex-styling advice to company-capf? Perhaps I should just use the rewritten company-capf--candidates in my personal config? It would be nice to have the best of both worlds: flex completion styles and custom sorting.

[dgutov]

Does this information raise a solution as to how I might tailor my sort-completions function while keeping my flex-styling advice to company-capf?

Not yet. I don't see the difference in those implementations.

(sortfun (cdr (assq 'display-sort-function meta))) line produces a sortfun with the byte-coded function value of my sort-completions function, whereas the alternative completion-metadata-get method produces a sortfun value of the plain symbol sort-completions

Does this mean that to reproduce this I need to byte-compile the file with your custom code first?

[kings2u]

Does this mean that to reproduce this I need to byte-compile the file with your custom code first?

I meant that (sortfun (cdr (assq 'display-sort-function meta))) sets sortfun with the value of a byte-coded function object as described here. So when the critical line (setq candidates (funcall sortfun candidates)) is executed, sortfun has the value:

#[257 \300\203
\0\300�!\202�\0\211\301 \203�\0d\302 V\203�\0\303�\304"\202�\0\211\207 [sort-completions window-minibuffer-p minibuffer-prompt-end sort #[514 \300\301\302�#\300\301\302�#�\206�\0\301�\206�\0\301V\207 [get-text-property 0 completion-score] 7 
(fn C1 C2)]] 5 
(fn COMPLETIONS)] 

whereas with the completion-metadata-get method, sortfun just has the value sort-completions.

I have debugged the current version of company-capf--candidates with the following changes

(setq company-capf--sorted (functionp sortfun))
- (when sortfun
-            (setq candidates (funcall sortfun candidates)))
+ (when sortfun
+          (let ((inhibit-redisplay nil))
+            (debug)
+            (setq candidates (funcall sortfun candidates))))

and observed how my sort-completions function initially works to reorder the candidates, but then another unnamed function (whose source code is not available to me) jumps in to re-sort them back into the default order.

Anyways, thanks for your patient help as always :) I can mark this as solved if you like since it seems you’re content with the current behavior.

[kings2u]

Also interesting is that changing my advice to company-capf to use orderless-flex instead of the built-in emacs flex resolves the problem without needing to change any part of company-capf--candidates:

(define-advice company-capf
    (:around (orig-fun &rest args) set-completion-styles)
  (let ((completion-styles '(orderless))
        (orderless-matching-styles '(orderless-flex)))
    (apply orig-fun args)))

[dgutov]

I meant that (sortfun (cdr (assq 'display-sort-function meta))) sets sortfun with the value of a byte-coded function object as described here.

Hmm.

I guess it's a sorting function added by the flex style. It's unconditionally added in completion--flex-adjust-metadata. It's probably "the right thing" for us to honor this.

The completion-metadata-get might actually get the wrong result, leading to no flex sorting in the general scenario as well. I wonder how, though. It's as if input and (buffer-substring (nth 1 res) (nth 2 res)) are different, but I can't reproduce that with print-debugging.