Running clj-kondo with --parallel option sometimes fails with "Clj-kondo cache is locked by other thread or process"

[mrkam2]

clj-kondo v2023.10.20

I was running the following command:

clj-kondo --lint <563 multiple source paths> --parallel

Platform: Linux

I've got 16 errors like the following one:

<path>/<file>.clj:0:0: error: Can't parse <path>/<file>.clj:0:0, Clj-kondo cache is locked by other thread or process.

Where <path>/<file> belong to different source paths and are random for each run.

Running without --parallel succeeds.

[borkdude]

Do you have a test project where I can reproduce this? Without a repro there's not much I can do.

[mrkam2]

Unfortunately it is a private project, but what I can help with is running a modified version that produces more debug output. Best regards, Alexander

…

On Tue, Nov 21, 2023 at 1:20 PM Michiel Borkent ***@***.***> wrote: Do you have a test project where I can reproduce this? Without a repro there's not much I can do. — Reply to this email directly, view it on GitHub <#2218 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AGXVI5QRMHADJCDCIWEHCVTYFULIVAVCNFSM6AAAAAA7VB6X4GVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMRRG4YDMMRTGY> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[borkdude]

I guess you can run the modified version locally and see if you can find out more.

[mrkam2]

So I added some logging to clj-kondo.impl.cache/with-cache and saw that pool-1-thread-N threads (N is 1..9 on my computer) compete for the cache lock. They make 1323 attempts to lock the file and 117 times these fail. Out of those failures 102 are the 0th retry (i. e. first try), 12 - at 1st retry, 2 - at 2nd retry, 1 - at 3rd retry:

retry# - count
0 - 102
1 - 12
2 - 2
3 - 1

I suspect that the number of parallel threads (as when run on a powerful CI host) creates increased contention which results in reaching the max retry attempts for some threads. My logging shows that the max number of retries is set to 6. So on my computer with 9 parallel threads it seems unlikely (although possible) to run into this issue. The number of processors is predefined in https://github.com/clj-kondo/clj-kondo/blob/master/src/clj_kondo/impl/core.clj#L364 so there isn't much I can currently do to change this. The number of retries is predefined in code in https://github.com/clj-kondo/clj-kondo/blob/master/src/clj_kondo/impl/analyzer/namespace.clj#L290 and in https://github.com/clj-kondo/clj-kondo/blob/master/src/clj_kondo/impl/cache.clj#L179.

I tried running locally with increased number of threads. With 26 threads, it failed to obtain lock 291 times and the retry counts were also higher:

retry# - count
0 - 194
1 - 64
2 - 17
3 - 8
4 - 4

It seems to me that to resolve the issue we should make either the number of retries higher and/or configurable or the number of parallel threads lower and/or configurable.

Thoughts?

[borkdude]

I don't know the exact answer of what tuning will be best, but maybe we could make threads configurable at least.

[mrkam2]

I don't know the exact answer of what tuning will be best, but maybe we could make threads configurable at least.

I'd say we never want this to fail, so we probably should increase the number of retries. We could also potentially create a queue of cache updates so that they're guaranteed to execute in the order they happen and no thread is delayed too much.

[borkdude]

I like the queue idea