Erroneous newlines using efm in neovim

[farbodsz]

Problem

Sometimes when running formatters through EFM in neovim, extra newlines are inserted that would not appear if the formatter was run directly from the command line.

One example of this is running prettier to format a table, but this problem seems to be present regardless of which formatter is being run through EFM.

Below I'll detail one specific example of this when I used prettier.

Configuration

Using efm within nvim with this config:

  prettier = {
    formatCommand = "./node_modules/.bin/prettier --stdin-filepath ${INPUT}",
    formatStdin = true,
  },

My full configuration can be found here.

NVIM version

NVIM v0.5.1
Build type: RelWithDebInfo
LuaJIT 2.1.0-beta3
Compilation: /usr/bin/gcc-11 -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=1 -DNVIM_TS_HAS_SET_MATCH_LIMIT -O2 -g -Og -g -Wall -Wextra -pedantic -Wno-unused-parameter -Wstrict-prototypes -std=gnu99 -Wshadow -Wconversion -Wmissing-prototypes -Wimplicit-fallthrough -Wvla -fstack-protector-strong -fno-common -fdiagnostics-color=always -DINCLUDE_GENERATED_DECLARATIONS -D_GNU_SOURCE -DNVIM_MSGPACK_HAS_FLOAT32 -DNVIM_UNIBI_HAS_VAR_FROM -DMIN_LOG_LEVEL=3 -I/home/runner/work/neovim/neovim/build/config -I/home/runner/work/neovim/neovim/src -I/home/runner/work/neovim/neovim/.deps/usr/include -I/usr/include -I/home/runner/work/neovim/neovim/build/src/nvim/auto -I/home/runner/work/neovim/neovim/build/include
Compiled by runner@fv-az87-829
Features: +acl +iconv +tui

See ":help feature-compile"
system vimrc file: "$VIM/sysinit.vim"

fall-back for $VIM: "

/home/runner/work/neovim/neovim/build/nvim.AppDir/usr/share/nvim"
Run :checkhealth for more info

EFM version: efm-langserver 0.0.37 (rev: 9c36ed8/go1.17.1)

Steps to reproduce

Create an empty project directory and install prettier in it:

$ mkdir example
$ yarn add prettier

Add the following file (e.g. example.md):

# Example

| First               | Second                              |
| ----- | ------------------------------------- |
| Lorem ipsum text      | More dummy text                  |

Output of vim.lsp.buf.formatting():

# Example


| First            | Second          |

| ---------------- | --------------- |

| Lorem ipsum text | More dummy text |

Expected result

Running ./node_modules/.bin/prettier example.md:

# Example

| First            | Second          |
| ---------------- | --------------- |
| Lorem ipsum text | More dummy text |

[gjeusel]

Same here, happen with prettier for yaml, but also black for python.

NVIM version

NVIM v0.5.1
Build type: RelWithDebInfo
LuaJIT 2.1.0-beta3
Compilation: /Library/Developer/CommandLineTools/usr/bin/cc -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=1 -DNVIM_TS_HAS_SET
_MATCH_LIMIT -O2 -g -Wall -Wextra -pedantic -Wno-unused-parameter -Wstrict-prototypes -std=gnu99 -Wshadow -Wconvers
ion -Wmissing-prototypes -Wimplicit-fallthrough -Wvla -fstack-protector-strong -fno-common -fdiagnostics-color=alwa
ys -DINCLUDE_GENERATED_DECLARATIONS -D_GNU_SOURCE -DNVIM_MSGPACK_HAS_FLOAT32 -DNVIM_UNIBI_HAS_VAR_FROM -DMIN_LOG_LE
VEL=3 -I/Users/gjeusel/src/neovim/build/config -I/Users/gjeusel/src/neovim/src -I/Users/gjeusel/src/neovim/.deps/us
r/include -I/Library/Developer/CommandLineTools/SDKs/MacOSX11.3.sdk/usr/include -I/opt/homebrew/opt/gettext/include
 -I/Users/gjeusel/src/neovim/build/src/nvim/auto -I/Users/gjeusel/src/neovim/build/include
Compiled by [email protected]

Features: +acl +iconv +tui
See ":help feature-compile"

   system vimrc file: "$VIM/sysinit.vim"
  fall-back for $VIM: "/usr/local/share/nvim"

• EFM version: efm-langserver 0.0.37 (rev: HEAD/go1.16.5)
• Prettier version: 2.4.1
• Black version: 19.10b0

Notes:

• It was not happening with those same prettier + black version before neovim 0.5.1
• It is not happening calling directly > prettier myfile.yaml

[farbodsz]

Just noticed I get this problem with clang-format too, in the latest neovim-0.6.0, so I'll update the issue title.

The details of this clang-format issue are:

#include "ast.hpp"
#include <string>

std::string print_ast(const AstNode &node, const std::string &prefix, const std::string child_prefix) {

}

Output:

#include "ast.hpp"
#include <string>


std::string print_ast(const AstNode &node, const std::string &prefix,

                      const std::string child_prefix) {}

Expected (i.e. what I get from running clang-format from the command line):

#include "ast.hpp"
#include <string>

std::string print_ast(const AstNode &node, const std::string &prefix,
                      const std::string child_prefix) {}

[smichaud]

I confirm that I have the same problem with pandoc/yaml, black/python, prettier/json,ts...
I am using efm to format my files for a while and didn't noticed it before, so I guess it was introduced recently.

[nmrtv]

I have two machines. One at home - Linux, another at work - Windows (using powershell). Neovim configurations are the same. And I have a similar issue, but only on a Windows machine and only with files that have windows (\r\n) line endings. If I change line endings to Linux style (\n), formatting works correctly without extra blank lines. On Linux, formatting works regardless eol type.

Edit:
Using Neovim 6.0 with native lsp.

[fsouza]

Spent some time looking into this issue and wrote my findings in a neovim issue: neovim/neovim#16842 (comment)

cc @mattn @lukas-reineke as you two may be interested

[mattn]

diff --git a/langserver/diff.go b/langserver/diff.go
index 01d1df3..5bec701 100644
--- a/langserver/diff.go
+++ b/langserver/diff.go
@@ -218,7 +218,7 @@ func shortestEditSequence(a, b []string) ([][]int, int) {
 }
 
 func splitLines(text string) []string {
-	lines := strings.SplitAfter(text, "\n")
+	lines := strings.Split(text, "\n")
 	if lines[len(lines)-1] == "" {
 		lines = lines[:len(lines)-1]
 	}

I'm not sure but does this works good?

[raviqqe]

@mattn Unfortunately, it doesn't seem to fix the bug...

I haven't gone deep into the diff algorithm. But I can reproduce the problem with a sub-minimal example of sed like below.

version: 2
tools:
  sed: &sed
    format-command: sed s/[[:blank:]]*$//g
    format-stdin: true
languages:
  text:
    - <<: *sed

With the configuration above, the following text file ($ is a newline character)

foo $

is formatted into

$
foo$

with the log of

2022/01/05 03:34:53 jsonrpc2: <-- result #9: textDocument/formatting: [{"range":{"start":{"line":0,"character":0},"end":{"line":1,"character":0}},"newText":""},{"range":{"start":{"line":1,"character":0},"end":{"line":1,"character":0}},"newText":"foo"}]

It looks like the diff code generates an extra delete op? although I'm not sure if it's normal or not.

[fsouza]

Looks like this was fixed in neovim: neovim/neovim#17037