Developing xref-ctags.el,a client tool running on GNU/Emacs

[masatake]

@rrthomas has an interest in this topic.

@AmaiKinono has some good code in @AmaiKinono's private repo.
I think we can it as a base. It is GPL'ed. So, in my understanding, referring to it never becomes an issue for our future contribution for GNU/Emacs. See https://www.emacswiki.org/emacs/Copyright_Assignment to know what I care about.

@AmaiKinono, are you still be in busy?
I would like you to add @rrthomas to your private repo.

[rrthomas]

@masatake, as you may have noticed, I am already an Emacs contributor with an active copyright assignment.

If you're suggesting that I can refer to @AmalKinono's code, then it doesn't really matter what its license is: if I can be considered to have copied it, then @AmalKinono will also need to do a copyright assignment. It would be easier for me simply to do a clean-room implementation if a copyright assignment is not possible. Of course, it would be best if I could use @AmalKinono's code with a copyright assignment in place.

[masatake]

As far as I know, xref-etags.el loads whole TAGS file into a buffer?
Searching is done by the regex engine of Emacs.
I think this becomes an issue if the TAGS file becomes larger.

@AmaiKinono's elisp doesn't load tags file into the buffer.
Instead, it utilizes the readtags command shipped as part of u-ctags.
readtags command does "binary searching". It minimizes the impact of the size of tags in cost of searching.

[rrthomas]

@masatake, that sounds like a useful optimization. However, especially in order to increase the chances of getting the code accepted into Emacs itself, it would be good to have a pure Elisp implementation that reads the ctags file itself. I was thinking I would start by looking at etags.el and seeing what changes are necessary to support ctags, and then in a second pass seeing what refactoring is necessary to merge the support for ctags into etags.el.

[masatake]

I don't think extending etags.el is better.
Could you tell me the largest TAGS file have you ever loaded?
My tags file for linux kernel is more than 1GB.
Generally I also refer tags file for glibc and systemd. So the tags file becomes larger.

The readtags is part of u-ctags. So we can expect the readtags command is avaiable where tags file is.
I would like you to decide the direction after evaluating @AmaiKinono's work.

I would like to see https://docs.ctags.io/en/latest/man/readtags.1.html#expression, too.

[rrthomas]

I agree that supporting the use cases you mention is a good idea.

However, as far as I can see, universal-ctags is the only implementation that supports readtags, and it will be easier to get Emacs to include ctags support if the new package works with any ctags program.

[AmaiKinono]

Sorry for the late reply. I'll be busy till the end of this year, but I'll keep an eye on this topic.

I would like you to add @rrthomas to your private repo.

Actually I think I could just make it public. It's not yet finished, but it already has a comprehensive readtags interface and xref integration. I think it's a good idea to make it public so people could learn/copy from it.

About copyright assignment...

Is it possible to do it online? I don't have experience sending international mail. But I'll be happy to assign one anyway.

About whether use readtags or not

My code is completely based on readtags, so maybe it's not a valuable reference if xref-ctags.el doesn't use readtags.

I also vote for using it. As far as I know, Exuberant Ctags also comes with a readtags implementation, which is not as feature-rich as the one in u-ctags, but it can do binary search, can filter tags by prefix, so it should be enough for xref-ctags. I heard there are some other ctags implementations but I never saw them in real life, so it should be safe to assume that ctags users will have readtags on their machines.

Even if xref-ctags are going to load the tags file (don't know if the RAM usage would be a problem though...), I think it's a good idea to implement binary search on our own. The TAG_FILE_SORTED pseudo tag can tell you whether the tags are sorted alphabetically, and we could fallback to regex searching if it's not.

[AmaiKinono]

Just made it public. Here: https://github.com/AmaiKinono/citre

I wrote a bit about the design in docs/core-layer.md. The code is divided into 3 layers, and you could read it clearly in the outline of citre.el. These may help you read the code, and feel free to ask me if you have any questions.

[AmaiKinono]

@masatake once proposed that Emacs itself could include a readtags interface (citre-readtags.el in my repo). If this actually happens, people could build all kinds of tools on top of it, including something like xref-ctags. I don't know which idea is better.

[masatake]

@AmaiKinono, thank you.
@rrthomas may decide the direction :-)

[rrthomas]

Thanks, @AmaiKinono, I shall look at your code as soon as I can.

It would be a good idea to put a copyright assignment in place if you can start that now, as it can take some time. It should be possible to do it paperlessly: please write to [email protected] as described in https://www.gnu.org/software/emacs/manual/html_node/emacs/Copyright-Assignment.html

[masatake]

I write what I know about the availablity of readtags command.

Exuberant-ctags is the most popular ctags implementation. However, "make install" target of Exuberant-ctags doesn't install its readtags command.
Universal-ctags is not so popular as Exuberant-ctags. However, "make install" target of Universal-ctags installs its readtags command.

[rrthomas]

Thanks, @masatake, I didn't know there was a (non-installed) version of readtags with exuberant-ctags. We should also consider, from the point of view of code specifically for Emacs, that Emacs's ctags has no readtags.

But I think I will start with @AmaiKinono's code: it seems like it may be possible already to use readtags conveniently in a nice 3rd-party package, and that already would be good.

[masatake]

Thanks, @masatake, I didn't know there was a (non-installed) version of readtags with exuberant-ctags. We should also consider, from the point of view of code specifically for Emacs, that Emacs's ctags has no readtags.

Oh, it is not my intention. What I would like to write is that I recommend you to assume it is u-ctags' readtags if readtags is there.
Ignoring readtags of e-ctags may be better.

[masatake]

http://emacs.1067599.n8.nabble.com/gtags-ctags-etags-cscope-support-td478663.html#a478669

[rrthomas]

I have found some time to look at citre. Unfortunately there is no README at present, but I managed to install it and get citre-jump to work. I will ask the author one or two questions.

I cannot find a Vala parser or branch in universal-ctags at the moment; am I missing something? So instead, I used anjuta-ctags (which we've discussed before), since its tags file output should be OK (you may remember I tried using its etags-style output and that did not work well).

As far as I can see, all that citre should need is to be packaged for easy installation (e.g. via MELPA), and with some top-level documentation to make it easier to start using. Otherwise, it looks very promising.

[AmaiKinono]

I think @masatake's main concern is to have a built-in client tool in Emacs, rather than a package that could be "easily installed" (though the latter is also important and needed).

[rrthomas]

That will be much harder to persuade the developers of: why would the Emacs developers want to take on the burden of several thousand lines more elisp? It cannot be because of the readtags functionality, because for that they would have to take universal-ctags itself upstream into Emacs. (That would be nice!) The obvious question would be "why does Emacs need ctags support when it already has etags support, and universal-ctags can already output in etags format?"

[AmaiKinono]

Umm, first I didn't meant to merge Citre into GNU Emacs, in case you may misunderstood it. xref-ctags.el should be a separate, small package that aims at basic functionalities.

It cannot be because of the readtags functionality

We could make xref-ctags.el not depend on readtags.

why does Emacs need ctags support when it already has etags support, and universal-ctags can already output in etags format?

Ah, this is a real question... maybe @masatake have an answer.

I'd say the ctags format is simply better. If you edit the source file, then use Citre to find a definition in it, it could still find the right line most of the time. This is because ctags uses a search pattern to locate a line, so it's still useable after editing. This can't be done using etags format.

Also, u-ctags offers many useful fields that could offer additional annotations in the xref UI. Citre uses the kind field in xref, and it could further use typeref field. This makes it easy to filter out some definitions without goto it.

But yes, I understand that things are not accepted by Emacs just because they are cool. The reasons I gave looks insufficient even to myself.

[rrthomas]

@AmaiKinono I think we're going round in circles here. I suggested a straightforward ctags interface, but @masatake said they preferred support for readtags. :)

[AmaiKinono]

We can discuss on these things. But, maybe it's better to hear how Emacs people think of it (if we could) than we three trying to decide on a design here.

[rrthomas]

@AmaiKinono, it depends what "it" is. What's the suggestion? I can certainly write to the Emacs developers' list with a concrete proposal. Of course, it's better if it's for something for which code already exists :) As I said, I think the best thing would be ctags support that mirrors the current etags support. However, it's not very useful in the short term. In the long term it would be great if Emacs could drop etags support altogether (from both Emacs itself, and from Emacs's implementation of ctags).

[masatake]

@rrthomas, the branch is at #2677 .

I wonder it is possible to write a useful tags file frontend without readtags.

I run ctags and etags on the source tree of Linux kernel.

  -rw-r--r--.   1 jet jet 631086996 Dec  3 07:53 tags
  -rw-r--r--.   1 jet jet 230810237 Dec  3 07:16 TAGS

tags output is larger than 600MB. However, this tags file is optimized in size.
Linux kernel supports multiple architectures. The tags/TAGS files are only for x86.
If tagging whole the source tree, the sizes become larger. I'm not surprised if it is
larger than 1GB. For making M-. enough fast, you will load a tags file into a buffer.
Do you want to do so?

An alternative approach for avoiding to load tags file into a buffer is integrating libreadtags,
the engine of readtags command, into Emacs' C code side.

[AmaiKinono]

Supporting huge tags files comes with the price of introducing external dependencies. Considering etags.el itself can't work well with huge TAGS files, I think it's acceptable to use a similar implementation, and let the user seek for a better tool if they need to work with huge tags files.

An alternative approach for avoiding to load tags file into a buffer is integrating libreadtags

I actually have some alternative ideas:

• Use grep, like what Citre originally did. If I remember it correctly, using this approach with GNU grep on linux kernel is not very fast, but ripgrep is fast. We could offer a user option to use different grep programs.

• Use sed to implement binary search on sorted tags files. It can read a specific line from the file, so we could just count the total lines, and read the middle line, compare, than read the middle of a half of the file...

[rrthomas]

Just a note to say that I found the vala branch of @masatake's ctags repo, so I will be using universal-ctags in future testing.

[masatake]

@rrthomas, please note that the branch will be rebased frequently.

[rrthomas]

@masatake noted, thanks!