Complete document logs

[ScottCov]

How do I get logs of every document processed? At the default level it doesn't look like this is done in either documents.log or fscrawler.log. Do I need to change a setting in log4j2.xml?

[dadoonet]

Indeed. See https://fscrawler.readthedocs.io/en/latest/admin/logger.html

I think that using debug for DOC_LEVEL should work.

But may be you are looking for something else?

[ScottCov]

sorry one last question...using docker would it be --env sys:DOC_LEVEL=debug or --env DOC_LEVEL=debug?

[dadoonet]

I'm unsure but I guess it should be --env DOC_LEVEL=debug

[ScottCov]

OK I am pretty sure I set DOC_LEVEL=debug based on the output here:

root@1849266785f6:/usr/share/fscrawler/logs# /usr/bin/env
HOSTNAME=1849266785f6
LANGUAGE=en_US:en
JAVA_HOME=/opt/java/openjdk
PWD=/usr/share/fscrawler/logs
HOME=/root
LANG=en_US.UTF-8
LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.webp=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:
TERM=xterm
SHLVL=1
LC_ALL=en_US.UTF-8
PATH=/opt/java/openjdk/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
DOC_LEVEL=debug
OLDPWD=/usr/share/fscrawler
JAVA_VERSION=jdk-17.0.8.1+1
_=/usr/bin/env

However, neither of the logs are listing the filenames that have been indexed which is what I was looking for...is this possible for fscrawler?

[dadoonet]

I think you actually need to configure a FS_JAVA_OPTS env variable.

FS_JAVA_OPTS="-DDOC_LEVEL=debug" 

This should be picked up by the fscrawler script then.

I tested this locally (not with Docker):

FS_JAVA_OPTS="-DDOC_LEVEL=debug" bin/fscrawler --config_dir jobs --restart 

And that produced then:

2023-12-11 18:00:19,884 [�[36mDEBUG�[m] [2ab33d44a3494ea18d95b4c9e8132381][/test-de.txt] Indexing content
2023-12-11 18:00:20,202 [�[36mDEBUG�[m] [8544b34980183974ac9f4df6e1185bb][/test.docx] Indexing content
2023-12-11 18:00:20,208 [�[36mDEBUG�[m] [a2b121d13df2d518399f2fb5c65f9e5][/issue-418-中文名称.txt] Indexing content
2023-12-11 18:00:20,215 [�[36mDEBUG�[m] [f410335106bc7c635dea7109ffc8dc][/issue-369.txt] Indexing content
2023-12-11 18:00:20,222 [�[36mDEBUG�[m] [fa55323ad9638217c4911d46810b8][/.DS_Store] Indexing content
2023-12-11 18:00:20,316 [�[36mDEBUG�[m] [3c22a7cb389448f136b2e509d4a6f77][/test.doc] Indexing content
2023-12-11 18:00:20,325 [�[36mDEBUG�[m] [76781f489234e5ba8b82a1cb1b127d29][/test.mp3] Indexing content
2023-12-11 18:00:20,787 [�[36mDEBUG�[m] [9a169a0ca3d66ac3b46ea3df34a35ae][/test-ocr.docx] Indexing content
2023-12-11 18:00:20,792 [�[36mDEBUG�[m] [c1d96de9d355f8f778768014c2b2ce58][/test.json] Indexing content
2023-12-11 18:00:20,798 [�[36mDEBUG�[m] [23f8e12bc4c08385181012fc88b1d9d8][/issue-592.xml] Indexing content
2023-12-11 18:00:22,139 [�[36mDEBUG�[m] [db4dd9d42e83aabf247791f48e346ab][/test.pdf] Indexing content
2023-12-11 18:00:22,212 [�[36mDEBUG�[m] [3ada533127bf9aaa53b3f7e5d5925c][/test.html] Indexing content
2023-12-11 18:00:22,215 [�[36mDEBUG�[m] [eac57f35acca9ebe8bb0ded3f9332e0][/issue-163.xml] Indexing content
2023-12-11 18:00:22,219 [�[36mDEBUG�[m] [434df5a8e341e2614b1ec7b9d75bf23f][/test-enfrde.txt] Indexing content
2023-12-11 18:00:23,049 [�[36mDEBUG�[m] [4ee29099de179142423dfca82cbf1328][/test-ocr-notext.pdf] Indexing content
2023-12-11 18:00:23,052 [�[36mDEBUG�[m] [90246a41cede38869341ba86cab7f][/test.txt] Indexing content
2023-12-11 18:00:23,056 [�[36mDEBUG�[m] [78fbdfb1fc579fd49c8f47ece816350][/test-fr.txt] Indexing content
2023-12-11 18:00:23,082 [�[36mDEBUG�[m] [e1f2961cb50441bb912780e5f07f9b][/test.rtf] Indexing content
2023-12-11 18:00:24,579 [�[36mDEBUG�[m] [5e6fdfb82171bc111c13933c1dbe827e][/issue-221-doc2.pdf] Indexing content
2023-12-11 18:00:26,018 [�[36mDEBUG�[m] [a46c8144141c8cc0ce9777d79bc2fccb][/test-ocr.pdf] Indexing content
2023-12-11 18:00:26,370 [�[36mDEBUG�[m] [3e22f0c81561aa2cc2aee9ec828335][/test-ocr.png] Indexing content
2023-12-11 18:00:26,376 [�[36mDEBUG�[m] [c81c60ccbd22466eb82fedffe59a7a5f][/test.wav] Indexing content
2023-12-11 18:00:27,803 [�[36mDEBUG�[m] [e84a3e8e5f2b15e378acac21a262cb8][/issue-221-doc1.pdf] Indexing content
2023-12-11 18:00:31,563 [�[36mDEBUG�[m] [3e993379257ee3c14fe577933744f5][/test-ocr-heb.pdf] Indexing content
2023-12-11 18:00:31,574 [�[36mDEBUG�[m] [977ba397cefce6406d7ffc59130f54][/test.odt] Indexing content
2023-12-11 18:00:31,579 [�[36mDEBUG�[m] [66d72a58e32eeb754562b711c4986cfe][/issue-400-shiftjis.txt] Indexing content

Hope this helps.

[ScottCov]

it does! Just added --env FS_JAVA_OPTS=-DDOC_LEVEL=debug to docker and disregard my post from yesterday under issues for now...This may enable me to gain clarity and figure out the issue