java.lang.OutOfMemoryError: Java heap space

[tapegram]

Hello! I've been trying to get the LSP server working for me in vscode and spacemacs on and off for a bit, but I consistently run into the same OOM error when trying to work in a medium/large-ish repo

I've been following different guides and trying random things like

• implementing the performance improvements from here for emacs https://emacs-lsp.github.io/lsp-mode/page/performance/
• just trying to randomly pass in values to increase memory in jvm/gradle
• tweaking what garbage collector settings I can find, etc.

and was just wondering if there is any actual solution to this?

Some other information:

Any help is appreciated, thanks!

[themkat]

#403 had the same error. I use Emacs myself, but have never experienced this. (even on large repos). I use Temurin/Adoptium though. I guess you are using Amazon Coretto? Are there any global settings for Java you are using? Any JAVA_OPTIONS environment variable that might affect the behaviour of the JVM?

How much RAM does your machine have? The symbol index is currently very RAM hungry, so you should have some RAM available (I have 16GB on my Macs and it never needs that much. 8GB might be too little for very big projects).

You are using an M1 Mac, yes? Are the JDK an ARM build? Or is it an x86 build running through Rosetta?

[tapegram]

Thanks for the response!

JAVA_OPTIONS - nothing set for me

JDK - I am using coretto which I manage through sdk man. In a bit I can experiment with temurin.

Machine specs:

I believe based on the sdkman sdk list java (image above) that I am using an ARM build.

[tapegram]

I do see this error when the lsp server starts though i don't really know what it means and some light googling didn't help.

[themkat]

@tapegram , that is not really important. Happens on all machines and uses. We are not using a Notebook document service, and it does not seem to cause any issues.

I'm a little shocked that a machine with your amount of RAM has heap size issues. How big of a codebase are we talking? I've never had issues with any open source projects or enterprise Java/Kotlin projects in my day-job (some projects are pretty big). I guess you have already tried setting a high heap size with –Xmx4G or something similar?

[tapegram]

Yeah I'm not really sure what's going on with this and it makes me sad :(

I just tried making the heap size like -Xmx12G with some env vars (i didnt know which to use so I used all of them lol, gradle_opts, java_opts, _java_options, sdk_java_options. And it appeared to make it not OOM, but then eventually i started getting GC specific OOM errors, so maybe I also need to learn about and then tweak GC stuff -- but it all seems weird that that is needed in the first place.

Another guess is maybe because its a multi-module repo? Is that something that the current Kotlin language server can handle well?

I'm also trying to upgrade the gradlew version in case that might improve performance and help with memory issues, but that's also just a guess (I assume it uses the gradlew cli behind the scenes?)

I appreciate the help so far! I really want to get this working in spacemacs. I also found your blog post and video about it so that's at least giving me hope that this is possible.

[themkat]

Yeah I'm not really sure what's going on with this and it makes me sad :(

I understand that completely, wish I could do more to help 🙁

Another guess is maybe because its a multi-module repo? Is that something that the current Kotlin language server can handle well?

I use multi-module projects at work every day. They are a little bit slow at indexing, but I have not experienced any issues with memory (or heap space). The language server seems to handle them okay beyond that, but I think there is a limit here to how big it can handle. I have only tried Maven multi module projects, so Gradle multi module might be very different. Have only done quite small (at least compared to the Maven projects I have been working with) multi module projects in Gradle so far.

I'm also trying to upgrade the gradlew version in case that might improve performance and help with memory issues, but that's also just a guess (I assume it uses the gradlew cli behind the scenes?)

Yes, the gradlew cli is used to fetch the dependencies of the project behind the scenes. I'm currently using at least gradle 7.5 for most of my gradle projects. Use Maven for most projects though, especially the multi module ones.

I appreciate the help so far! I really want to get this working in spacemacs. I also found your blog post and video about it so that's at least giving me hope that this is possible.

Fingers crossed 🙂 Wonder if more blogs and youtube channels than mine have covered the Kotlin language server in Emacs so far. If you find any that are not mine, feel free to post them 🙂 (mine have themkat in the username or domain). I at least love seeing the language server and alternative editors for Kotlin covered in blogs and media!

[tapegram]

Is there a way in emacs / spacemacs to pass the jvm args directly to LSP? it feels weird to be using these big gross env vars. And are there any particular settings I should be using?

This is what I have at the moment, but I still run into GC OOM errors (though its able to fully index and work for a brief moment before the GC errors start).

 # One of these has to do something, right? trying to deal with an OOM heap error in LSP server
  1 export GRADLE_OPTS="-Xmx12g -XX:+UseParallelGC -Dkotlin.daemon.jvm.options=-Xmx6g -XX:MaxMetaspaceSize=6g -XX:+HeapDumpOnOutOfMemoryError -Dfile.encoding=UTF-8"
  2 export JAVA_OPTS="-Xmx12g -XX:+UseParallelGC -Dkotlin.daemon.jvm.options=-Xmx6g -XX:MaxMetaspaceSize=6g -XX:+HeapDumpOnOutOfMemoryError -Dfile.encoding=UTF-8"
  3 export _JAVA_OPTIONS="-Xmx12g -XX:+UseParallelGC -Dkotlin.daemon.jvm.options=-Xmx6g -XX:MaxMetaspaceSize=6g -XX:+HeapDumpOnOutOfMemoryError -Dfile.encoding=UTF-8"
  4 export JDK_JAVA_OPTIONS="-Xmx12g -XX:+UseParallelGC -Dkotlin.daemon.jvm.options=-Xmx6g -XX:MaxMetaspaceSize=6g -XX:+HeapDumpOnOutOfMemoryError -Dfile.encoding=UTF-8"

[themkat]

Is there a way in emacs / spacemacs to pass the jvm args directly to LSP? it feels weird to be using these big gross env vars. And are there any particular settings I should be using?

This is what I have at the moment, but I still run into GC OOM errors (though its able to fully index and work for a brief moment before the GC errors start).

 # One of these has to do something, right? trying to deal with an OOM heap error in LSP server
  1 export GRADLE_OPTS="-Xmx12g -XX:+UseParallelGC -Dkotlin.daemon.jvm.options=-Xmx6g -XX:MaxMetaspaceSize=6g -XX:+HeapDumpOnOutOfMemoryError -Dfile.encoding=UTF-8"
  2 export JAVA_OPTS="-Xmx12g -XX:+UseParallelGC -Dkotlin.daemon.jvm.options=-Xmx6g -XX:MaxMetaspaceSize=6g -XX:+HeapDumpOnOutOfMemoryError -Dfile.encoding=UTF-8"
  3 export _JAVA_OPTIONS="-Xmx12g -XX:+UseParallelGC -Dkotlin.daemon.jvm.options=-Xmx6g -XX:MaxMetaspaceSize=6g -XX:+HeapDumpOnOutOfMemoryError -Dfile.encoding=UTF-8"
  4 export JDK_JAVA_OPTIONS="-Xmx12g -XX:+UseParallelGC -Dkotlin.daemon.jvm.options=-Xmx6g -XX:MaxMetaspaceSize=6g -XX:+HeapDumpOnOutOfMemoryError -Dfile.encoding=UTF-8"

Emacs does not always pick up environment variables from your shell. One example is PATH, where I need a package called exec-path-from-shell to make it happen.

When I have experimented with various environment variables for an Emacs use case, I have usually just set them in elisp (e.g, using the *scratch* buffer):

(setenv "JAVA_OPTS" "-Xmx12g -XX:+UseParallelGC")

There is no way beyond that which I'm aware of to send it directly to LSP. If you haven't tried using setenv yet, you should try that and see if you get a different behaviour 🙂 I have used that approach to select different versions of Java in the past. (I use homebrew to install them, and set JAVA_HOME to change the Java version).

[themkat]

@tapegram , did you try the way of setting an environment variable directly inside of Emacs? If so, did anything improve or did you get the same results? 🙂

[tapegram]

@tapegram , did you try the way of setting an environment variable directly inside of Emacs? If so, did anything improve or did you get the same results? 🙂

I got the same result, I was using spacemacs and fortunately/unfortunately I had already regenerated the env var file so it was getting the java options.

I ended up just giving it a huge amount of memory (maybe like 18g) and it stopped getting OOM errors, however, it was just generally really slow, would get confused often, and just generally seemed to be struggling with providing the kind of language support I was hoping for.

That being said, I'm still eagerly following this project! I'll probably be retrying it every now and again in the hope of escaping intellij! Thanks for all your help and for working on the language server!

[hkupty]

I just want to mention I'm experiencing the same issue using kotlin-language-server from neovim.

From my preliminary tests, it seems to be caused by a huge allocation in the debounce0 thread.
I couldn't really find if there's any leak any component being called in this thread that is allocating more than it should, but on my computer it definitely consumed all the 32 gb.
It seems to be doing some pretty intense compilation there so I wouldn't be surprised if the leak happens under the hood, in a lower level component.

Is there a way to turn off linting? Maybe that's a good short-term solution for large projects..

Below are screenshots I took of JMX and JFR on kotling-language-server (server evaluating its own code).

[hkupty]

I tried to debug further.. It might be connected to protobuf communication with the kotlin compiler daemon.
This line can be connected with the issue, since it seems to re-create the container for the same set of files multiple times, aggravating the issue since the protobuf objects will be re-created at every iteration.

I added a log where we can see ${CompilationEnvironment} -> ${sourcePath.hashCode()} to ${createContainerResult} and you can see the results here. So, no values seem to be changing in the input (thus no change in the hashCode) but still it is re-created for each iteration of async4.

[themkat]

I finally got around to investigating this a bit further. Found some information online about the KotlinCoreEnvironment.createForProduction functionality having memory leaks, most specifically this issue in jetbrains tracker. Also found a few forum threads. Unsure on how to fix this. We rely heavily on this functionality, and it will be created again and again if the language server decides that the compiler will need to be updated (i.e, when the classpath changes) 🙁 The only possible thing I can think of here is that it might be called too often early with projects with multiple modules thanks to all the addWorkspaceRoot class (which triggers an update/refresh of Compiler) in KotlinLanguageServer.kt.

Unsure if this helps us further in this issue, but writing it in case someone smarter than me can get use for it 🙂

[jpaju]

I'm also experiencing this issue. I have a ~100k loc codebase and after 9minutes 42 seconds of indexing (12801 symbols) with CPU constantly at 80-100%, the LSP crashes with an out-of-memory error. I'm using the LSP with helix and hardware is M2 Pro Mac with 32GB of RAM, if that helps.

Unfortunately this makes the LSP practically unusable which is very unfortunate since the only alternative editor that I can use (or at least I'm aware of) is IntelliJ.

I see that there have been many similar or possibly related issues over the years:

• Investigate memory usage and an on-disk index #352
• [Multiplatform + Neovim + VSCode] Performance, unresolved references, errors #376
• Slow initialization #510
• Cache Symbol Index #489
• Giant Project & VS Code Restart #160

Has there been any updates on this area lately? I can help to debug this and provide additional information if needed 😇

Here's the stacktrace

org.jetbrains.kotlin.util.KotlinFrontEndException: Front-end Internal error: Failed to analyze declaration ConcreteBatchResult\nFile being compiled: (38,1) in <path-to-file>.kt\nThe root cause java.lang.OutOfMemoryError was thrown at: java.base/java.util.Collections.unmodifiableList(Collections.java:1478)
  at org.jetbrains.kotlin.resolve.ExceptionWrappingKtVisitorVoid.visitDeclaration(ExceptionWrappingKtVisitorVoid.kt:43)
  at org.jetbrains.kotlin.psi.KtVisitorVoid.visitDeclaration(KtVisitorVoid.java:461)
  at org.jetbrains.kotlin.psi.KtVisitorVoid.visitDeclaration(KtVisitorVoid.java:21)
  at org.jetbrains.kotlin.psi.KtVisitor.visitNamedDeclaration(KtVisitor.java:406)
  at org.jetbrains.kotlin.psi.KtVisitorVoid.visitNamedDeclaration(KtVisitorVoid.java:385)
  at org.jetbrains.kotlin.psi.KtVisitorVoid.visitNamedDeclaration(KtVisitorVoid.java:973)
  at org.jetbrains.kotlin.psi.KtVisitorVoid.visitNamedDeclaration(KtVisitorVoid.java:21)
  at org.jetbrains.kotlin.psi.KtVisitor.visitClassOrObject(KtVisitor.java:41)
  at org.jetbrains.kotlin.psi.KtVisitorVoid.visitClassOrObject(KtVisitorVoid.java:37)
  at org.jetbrains.kotlin.psi.KtVisitorVoid.visitClassOrObject(KtVisitorVoid.java:473)
  at org.jetbrains.kotlin.psi.KtVisitorVoid.visitClassOrObject(KtVisitorVoid.java:21)
  at org.jetbrains.kotlin.psi.KtVisitor.visitClass(KtVisitor.java:33)
  at org.jetbrains.kotlin.psi.KtVisitorVoid.visitClass(KtVisitorVoid.java:33)
  at org.jetbrains.kotlin.psi.KtVisitorVoid.visitClass(KtVisitorVoid.java:467)
  at org.jetbrains.kotlin.psi.KtVisitorVoid.visitClass(KtVisitorVoid.java:21)
  at org.jetbrains.kotlin.psi.KtClass.accept(KtClass.kt:22)
  at org.jetbrains.kotlin.psi.KtElementImplStub.accept(KtElementImplStub.java:49)
  at org.jetbrains.kotlin.resolve.LazyTopDownAnalyzer$analyzeDeclarations$1.registerDeclarations(LazyTopDownAnalyzer.kt:80)
  at org.jetbrains.kotlin.resolve.LazyTopDownAnalyzer$analyzeDeclarations$1.visitKtFile(LazyTopDownAnalyzer.kt:98)
  at org.jetbrains.kotlin.psi.KtVisitorVoid.visitKtFile(KtVisitorVoid.java:521)
  at org.jetbrains.kotlin.psi.KtVisitorVoid.visitKtFile(KtVisitorVoid.java:21)
  at org.jetbrains.kotlin.psi.KtFile.accept(KtFile.kt:254)
  at org.jetbrains.kotlin.psi.KtFile.accept(KtFile.kt:241)
  at org.jetbrains.kotlin.resolve.ExceptionWrappingKtVisitorVoid.visitElement(ExceptionWrappingKtVisitorVoid.kt:27)
  at com.intellij.psi.PsiElementVisitor.visitFile(PsiElementVisitor.java:35)
  at org.jetbrains.kotlin.psi.KtVisitor.visitKtFile(KtVisitor.java:73)
  at org.jetbrains.kotlin.psi.KtVisitorVoid.visitKtFile(KtVisitorVoid.java:69)
  at org.jetbrains.kotlin.psi.KtVisitorVoid.visitKtFile(KtVisitorVoid.java:521)
  at org.jetbrains.kotlin.psi.KtVisitorVoid.visitKtFile(KtVisitorVoid.java:21)
  at org.jetbrains.kotlin.psi.KtFile.accept(KtFile.kt:254)
  at org.jetbrains.kotlin.psi.KtFile.accept(KtFile.kt:241)
  at org.jetbrains.kotlin.resolve.LazyTopDownAnalyzer.analyzeDeclarations(LazyTopDownAnalyzer.kt:203)
  at org.jetbrains.kotlin.resolve.LazyTopDownAnalyzer.analyzeDeclarations$default(LazyTopDownAnalyzer.kt:58)
  at org.javacs.kt.compiler.Compiler.compileKtFiles(Compiler.kt:549)
  at org.javacs.kt.SourcePath.compileFiles$compileAndUpdate(SourcePath.kt:221)
  at org.javacs.kt.SourcePath.compileFiles(SourcePath.kt:244)
  at org.javacs.kt.SourcePath.compileAllFiles(SourcePath.kt:259)
  at org.javacs.kt.KotlinTextDocumentService$lintAll$1.invoke(KotlinTextDocumentService.kt:279)
  at org.javacs.kt.KotlinTextDocumentService$lintAll$1.invoke(KotlinTextDocumentService.kt:278)
  at org.javacs.kt.util.Debouncer.submitImmediately$lambda$1(Debouncer.kt:27)
  at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
  at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
  at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
  at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
  at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
  at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.lang.OutOfMemoryError: Java heap space
  at java.base/java.util.Collections.unmodifiableList(Collections.java:1478)
  at org.jetbrains.kotlin.metadata.ProtoBuf$Package.\u003cinit\u003e(ProtoBuf.java:12910)
  at org.jetbrains.kotlin.metadata.ProtoBuf$Package.\u003cinit\u003e(ProtoBuf.java:12803)
  at org.jetbrains.kotlin.metadata.ProtoBuf$Package$1.parsePartialFrom(ProtoBuf.java:12934)
  at org.jetbrains.kotlin.metadata.ProtoBuf$Package$1.parsePartialFrom(ProtoBuf.java:12929)
  at org.jetbrains.kotlin.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:192)
  at org.jetbrains.kotlin.protobuf.AbstractParser.parseFrom(AbstractParser.java:209)
  at org.jetbrains.kotlin.protobuf.AbstractParser.parseFrom(AbstractParser.java:49)
  at org.jetbrains.kotlin.metadata.ProtoBuf$Package.parseFrom(ProtoBuf.java:13217)
  at org.jetbrains.kotlin.metadata.jvm.deserialization.JvmProtoBufUtil.readPackageDataFrom(JvmProtoBufUtil.kt:42)
  at org.jetbrains.kotlin.metadata.jvm.deserialization.JvmProtoBufUtil.readPackageDataFrom(JvmProtoBufUtil.kt:37)
  at org.jetbrains.kotlin.load.kotlin.DeserializedDescriptorResolver.createKotlinPackagePartScope(DeserializedDescriptorResolver.kt:69)
  at org.jetbrains.kotlin.load.java.lazy.descriptors.JvmPackageScope$kotlinScopes$2.invoke(JvmPackageScope.kt:47)
  at org.jetbrains.kotlin.load.java.lazy.descriptors.JvmPackageScope$kotlinScopes$2.invoke(JvmPackageScope.kt:44)
  at org.jetbrains.kotlin.storage.LockBasedStorageManager$LockBasedLazyValue.invoke(LockBasedStorageManager.java:408)
  at org.jetbrains.kotlin.storage.LockBasedStorageManager$LockBasedNotNullLazyValue.invoke(LockBasedStorageManager.java:527)
  at org.jetbrains.kotlin.storage.StorageKt.getValue(storage.kt:42)
  at org.jetbrains.kotlin.load.java.lazy.descriptors.JvmPackageScope.getKotlinScopes(JvmPackageScope.kt:44)
  at org.jetbrains.kotlin.load.java.lazy.descriptors.JvmPackageScope.getContributedClassifier(JvmPackageScope.kt:58)
  at org.jetbrains.kotlin.resolve.scopes.ChainedMemberScope.getContributedClassifier(ChainedMemberScope.kt:35)
  at org.jetbrains.kotlin.resolve.scopes.AbstractScopeAdapter.getContributedClassifier(AbstractScopeAdapter.kt:44)
  at org.jetbrains.kotlin.resolve.LazyExplicitImportScope.getContributedClassifier(LazyExplicitImportScope.kt:45)
  at org.jetbrains.kotlin.resolve.lazy.LazyImportScope$getClassifier$1.invoke(LazyImportScope.kt:279)
  at org.jetbrains.kotlin.resolve.lazy.LazyImportScope$getClassifier$1.invoke(LazyImportScope.kt:274)
  at org.jetbrains.kotlin.storage.LockBasedStorageManager.compute(LockBasedStorageManager.java:290)
  at org.jetbrains.kotlin.resolve.lazy.LazyImportScope.getClassifier(LazyImportScope.kt:274)
  at org.jetbrains.kotlin.resolve.lazy.LazyImportScope.getContributedClassifier(LazyImportScope.kt:270)
  at org.jetbrains.kotlin.resolve.scopes.ResolutionScope$DefaultImpls.getContributedClassifierIncludeDeprecated(ResolutionScope.kt:40)
  at org.jetbrains.kotlin.resolve.scopes.HierarchicalScope$DefaultImpls.getContributedClassifierIncludeDeprecated(Scopes.kt:27)
  at org.jetbrains.kotlin.resolve.scopes.ImportingScope$DefaultImpls.getContributedClassifierIncludeDeprecated(Scopes.kt:113)
  at org.jetbrains.kotlin.resolve.lazy.LazyImportScope.getContributedClassifierIncludeDeprecated(LazyImportScope.kt:235)
  at org.jetbrains.kotlin.resolve.scopes.utils.ScopeUtilsKt.findFirstClassifierWithDeprecationStatus(ScopeUtils.kt:119)