Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

limit addFile() processing time #4550

Open
vladak opened this issue Feb 22, 2024 · 2 comments
Open

limit addFile() processing time #4550

vladak opened this issue Feb 22, 2024 · 2 comments

Comments

@vladak
Copy link
Member

vladak commented Feb 22, 2024

When indexing bunch of projects, the indexer (based on 1.13.4) had a trailing worker thread that was spinning on the CPU for inordinate amount of time (minutes) while all the other files were finished. The stack trace looked like this (comes from my branch when experimenting with the fix for #4549 so thread name and line numbers do not match):

"OpenGrok-index-worker-48" #48 prio=5 os_prio=0 cpu=773733,68ms elapsed=1140,14s tid=0x00007f3648023000 nid=0x2134a7 runnable  [0x00007f364e5fd000]
   java.lang.Thread.State: RUNNABLE
	at org.opengrok.indexer.analysis.javascript.JavaScriptSymbolTokenizer.yylex(JavaScriptSymbolTokenizer.java:1645)
	at org.opengrok.indexer.analysis.JFlexTokenizer.incrementToken(JFlexTokenizer.java:106)
	at org.apache.lucene.index.IndexingChain$PerField.invertTokenStream(IndexingChain.java:1167)
	at org.apache.lucene.index.IndexingChain$PerField.invert(IndexingChain.java:1145)
	at org.apache.lucene.index.IndexingChain.processField(IndexingChain.java:697)
	at org.apache.lucene.index.IndexingChain.processDocument(IndexingChain.java:576)
	at org.apache.lucene.index.DocumentsWriterPerThread.updateDocuments(DocumentsWriterPerThread.java:243)
	at org.apache.lucene.index.DocumentsWriter.updateDocuments(DocumentsWriter.java:421)
	at org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1551)
	at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1836)
	at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1476)
	at org.opengrok.indexer.index.IndexDatabase.addFile(IndexDatabase.java:1278)
	at org.opengrok.indexer.index.IndexDatabase.lambda$indexParallel$9(IndexDatabase.java:1920)
	at org.opengrok.indexer.index.IndexDatabase$$Lambda$370/0x00000008401e7c40.call(Unknown Source)

Specifically, the file in question came from the https://hg.mozilla.org/mozilla-central , it was testing/modules/sinon-7.2.7.js which contains long lists of numbers. Such files are known to cause problems. Ideally the analyzers should skip/truncate the files with huge number of terms.

The indexer eventually finished, however there should be some time limit for addFile() or writer.addDocument() processing. There is already timeout enforced for xref production via the XrefWork class.

@vladak
Copy link
Member Author

vladak commented Feb 23, 2024

Rather than trying to terminate and cleanup the writer.addDocument() call it would be easier to check the term count in certain fields and skip the document altogether if the count is above given threshold.

Another idea is to trim the number if terms.

@saurav-max
Copy link

Review the code of the JavaScriptSymbolTokenizer class to understand what it's doing and where the error might be occurring. Look for any potential bugs, errors, or exceptions that could be causing the issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants