Fixed bloating temp directory with copies of native libraries #2622
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Before submitting a pull request, please do the following steps:
Problem: when using catboost in Java a native library has to be loaded into JVM. To do so we extract corresponding native library into temp folder. After JVM shutdown this file stays there. In case when temp folder is system temp folder it may not be a problem. However when catboost used in managed environment like Application Server or Servlet Container, the temp directory will be the temp directory of this environment. Without any cleaning mechanism space on hard drive will be quickly bloated.
In this PR I suggest mechanism that deletes copies of native library in temp directory. To accomplish it I did the following changes:
This algorithm will work every time JVM shutsdown gracefully. In case of sudden shutdown no remedy exists.
Why this algorithm will work? (Here is my explanation, but I am not 100% sure on it)
The problem is in the fact that native library that we copied to temp folder won't be deleted despite the fact that method deleteOnExit called. That happens because this file was loaded by System classloader and will be released only after System ClassLoader garbage collected. But GC of System ClassLoader means that JVM shut down.
But this won't happen to simple file which is .lck file.
Another approach to solve this problem is to load native library with other ClassLoader. I don't think this is a good idea.
The source of inspiration for this PR can be found here: https://github.com/xerial/sqlite-jdbc/blob/master/src/main/java/org/sqlite/SQLiteJDBCLoader.java
In the attachment you can find two projects. JNA-WAR and JNA-JAR. Their intention is to reproduce the problem and check the solution. There are dependencies on catboost-prediction and sqllite. JNA-WAR should be run in Servlet Container. I tested on tomcat 10.
jna_jar.zip
jna_war.zip