Problem with dates when consuming images

[selfyguy]

I have attached two files to this post.
Even though both files were modified and created today, paperless-ngx sorts one of them incorrectly into march of this month instead of
November.
Could someone use these files to replicate the behavior?
Is it possible that the filenmaes are causing confusion in the system? If so how can I workaround/fix this ?

Here is how the log entries look like - that has a post script taken from the documentation.

[2023-11-13 14:53:30,003] [INFO] [paperless.consumer] Consuming 2023-11-05 13_52_28-test year 3- works fine.jpg
[2023-11-13 14:53:30,302] [INFO] [ocrmypdf._pipeline] Input file is not a PDF, checking if it is an image...
[2023-11-13 14:53:30,303] [INFO] [ocrmypdf._pipeline] Input file is an image
[2023-11-13 14:53:30,303] [INFO] [ocrmypdf._pipeline] Input image has no ICC profile, assuming sRGB
[2023-11-13 14:53:30,303] [INFO] [ocrmypdf._pipeline] Image seems valid. Try converting to PDF...
[2023-11-13 14:53:30,306] [INFO] [ocrmypdf._pipeline] Successfully converted to PDF, processing...
[2023-11-13 14:53:31,139] [INFO] [ocrmypdf._pipeline] page is facing ⇧, confidence 5.50 - no change
[2023-11-13 14:53:36,347] [INFO] [ocrmypdf._sync] Postprocessing...
[2023-11-13 14:53:36,859] [INFO] [ocrmypdf._pipeline] Image optimization ratio: 6.33 savings: 84.2%
[2023-11-13 14:53:36,859] [INFO] [ocrmypdf._pipeline] Total file size ratio: 6.81 savings: 85.3%
[2023-11-13 14:53:36,861] [INFO] [ocrmypdf._sync] Output file is a PDF/A-2B (as expected)
[2023-11-13 14:53:39,452] [INFO] [paperless.consumer] Executing post-consume script /usr/src/paperless/code/postprocessscript.sh
[2023-11-13 14:53:39,544] [INFO] [paperless.consumer] /usr/src/paperless/code/postprocessscript.sh exited 0
[2023-11-13 14:53:39,551] [INFO] [paperless.consumer] Script stdout:
[2023-11-13 14:53:39,558] [INFO] [paperless.consumer] Current directory: /usr/src/paperless/src
[2023-11-13 14:53:39,565] [INFO] [paperless.consumer] 
[2023-11-13 14:53:39,572] [INFO] [paperless.consumer] 
[2023-11-13 14:53:39,579] [INFO] [paperless.consumer] A document with an id of 13 was just consumed.  I know the
[2023-11-13 14:53:39,588] [INFO] [paperless.consumer] following additional information about it:
[2023-11-13 14:53:39,596] [INFO] [paperless.consumer] 
[2023-11-13 14:53:39,605] [INFO] [paperless.consumer] * Generated File Name: 2023-11-13 2023-11-05 13_52_28-test year 3- works fine.jpg
[2023-11-13 14:53:39,613] [INFO] [paperless.consumer] * Archive Path: /usr/src/paperless/media/documents/archive/2023/Nov/2023-11-05 13_52_28-test year 3- works fine.pdf
[2023-11-13 14:53:39,624] [INFO] [paperless.consumer] * Source Path: /usr/src/paperless/media/documents/originals/2023/Nov/2023-11-05 13_52_28-test year 3- works fine.jpg
[2023-11-13 14:53:39,634] [INFO] [paperless.consumer] * Created: 2023-11-13 09:22:23.461672+00:00
[2023-11-13 14:53:39,645] [INFO] [paperless.consumer] * Added: 2023-11-13 09:23:38.318743+00:00
[2023-11-13 14:53:39,657] [INFO] [paperless.consumer] * Modified: 2023-11-13 09:23:39.332621+00:00
[2023-11-13 14:53:39,667] [INFO] [paperless.consumer] * Thumbnail Path: /usr/src/paperless/media/documents/thumbnails/0000013.webp
[2023-11-13 14:53:39,677] [INFO] [paperless.consumer] * Download URL: /api/documents/13/download/
[2023-11-13 14:53:39,686] [INFO] [paperless.consumer] * Thumbnail URL: /api/documents/13/thumb/
[2023-11-13 14:53:39,695] [INFO] [paperless.consumer] * Correspondent: None
[2023-11-13 14:53:39,703] [INFO] [paperless.consumer] * Tags: 
[2023-11-13 14:53:39,709] [INFO] [paperless.consumer] 
[2023-11-13 14:53:39,716] [INFO] [paperless.consumer] It was consumed with the passphrase
[2023-11-13 14:53:39,724] [INFO] [paperless.consumer] Document 2023-11-13 2023-11-05 13_52_28-test year 3- works fine consumption finished
[2023-11-13 14:53:39,814] [INFO] [celery.app.trace] Task documents.tasks.consume_file[855c77e0-25f0-4d84-a512-e3f86b574fad] succeeded in 10.027175965000424s: 'Success. New document id 13 created'

[2023-11-13 14:53:40,275] [INFO] [paperless.consumer] Consuming 2023-11-06 15_03_56-CPRD_Backend_Insights - Microsoft Azure 2 - month wrongly marked march.jpg
[2023-11-13 14:53:40,557] [INFO] [ocrmypdf._pipeline] Input file is not a PDF, checking if it is an image...
[2023-11-13 14:53:40,557] [INFO] [ocrmypdf._pipeline] Input file is an image
[2023-11-13 14:53:40,557] [INFO] [ocrmypdf._pipeline] Input image has no ICC profile, assuming sRGB
[2023-11-13 14:53:40,557] [INFO] [ocrmypdf._pipeline] Image seems valid. Try converting to PDF...
[2023-11-13 14:53:40,560] [INFO] [ocrmypdf._pipeline] Successfully converted to PDF, processing...
[2023-11-13 14:53:41,348] [INFO] [ocrmypdf._pipeline] page is facing ⇧, confidence 10.37 - no change
[2023-11-13 14:53:43,612] [INFO] [ocrmypdf._sync] Postprocessing...
[2023-11-13 14:53:43,939] [INFO] [ocrmypdf._pipeline] Image optimization ratio: 1.64 savings: 39.1%
[2023-11-13 14:53:43,939] [INFO] [ocrmypdf._pipeline] Total file size ratio: 2.35 savings: 57.5%
[2023-11-13 14:53:43,941] [INFO] [ocrmypdf._sync] Output file is a PDF/A-2B (as expected)
[2023-11-13 14:53:46,544] [INFO] [paperless.consumer] Executing post-consume script /usr/src/paperless/code/postprocessscript.sh
[2023-11-13 14:53:46,620] [INFO] [paperless.consumer] /usr/src/paperless/code/postprocessscript.sh exited 0
[2023-11-13 14:53:46,626] [INFO] [paperless.consumer] Script stdout:
[2023-11-13 14:53:46,631] [INFO] [paperless.consumer] Current directory: /usr/src/paperless/src
[2023-11-13 14:53:46,637] [INFO] [paperless.consumer] 
[2023-11-13 14:53:46,647] [INFO] [paperless.consumer] 
[2023-11-13 14:53:46,654] [INFO] [paperless.consumer] A document with an id of 14 was just consumed.  I know the
[2023-11-13 14:53:46,660] [INFO] [paperless.consumer] following additional information about it:
[2023-11-13 14:53:46,667] [INFO] [paperless.consumer] 
[2023-11-13 14:53:46,673] [INFO] [paperless.consumer] * Generated File Name: 2023-03-11 2023-11-06 15_03_56-CPRD_Backend_Insights - Microsoft Azure 2 - month wrongly marked march.jpg
[2023-11-13 14:53:46,678] [INFO] [paperless.consumer] * Archive Path: /usr/src/paperless/media/documents/archive/2023/Mar/2023-11-06 15_03_56-CPRD_Backend_Insights - Microsoft Azure 2 - month wrongly marked march.pdf
[2023-11-13 14:53:46,683] [INFO] [paperless.consumer] * Source Path: /usr/src/paperless/media/documents/originals/2023/Mar/2023-11-06 15_03_56-CPRD_Backend_Insights - Microsoft Azure 2 - month wrongly marked march.jpg
[2023-11-13 14:53:46,689] [INFO] [paperless.consumer] * Created: 2023-03-10 18:30:00+00:00
[2023-11-13 14:53:46,694] [INFO] [paperless.consumer] * Added: 2023-11-13 09:23:45.045059+00:00
[2023-11-13 14:53:46,699] [INFO] [paperless.consumer] * Modified: 2023-11-13 09:23:46.441664+00:00
[2023-11-13 14:53:46,703] [INFO] [paperless.consumer] * Thumbnail Path: /usr/src/paperless/media/documents/thumbnails/0000014.webp
[2023-11-13 14:53:46,709] [INFO] [paperless.consumer] * Download URL: /api/documents/14/download/
[2023-11-13 14:53:46,714] [INFO] [paperless.consumer] * Thumbnail URL: /api/documents/14/thumb/
[2023-11-13 14:53:46,721] [INFO] [paperless.consumer] * Correspondent: None
[2023-11-13 14:53:46,726] [INFO] [paperless.consumer] * Tags: 
[2023-11-13 14:53:46,733] [INFO] [paperless.consumer] 
[2023-11-13 14:53:46,740] [INFO] [paperless.consumer] It was consumed with the passphrase
[2023-11-13 14:53:46,747] [INFO] [paperless.consumer] Document 2023-03-11 2023-11-06 15_03_56-CPRD_Backend_Insights - Microsoft Azure 2 - month wrongly marked march consumption finished
[2023-11-13 14:53:46,839] [INFO] [celery.app.trace] Task documents.tasks.consume_file[a17c84ad-8986-4a4e-bee2-862191fdbe91] succeeded in 6.768270749999829s: 'Success. New document id 14 created'

Here are the files used

This is sorted wrongly for the date.
This file ends up in : \media\documents\originals\2023\Mar

This is sorted correctly for its date ends up in : \media\documents\originals\2023\Nov

Originally posted by @selfyguy in #4515 (comment)

[nawramm]

Paperless parses the date from the text in the document, if a string is found in the document that resembles a date. Otherwise the fallback is the current date. You can configure paperless to parse the filename for a date if you set PAPERLESS_FILENAME_DATE_ORDER. You can also use a post consumption script to set the created date.

[selfyguy]

Hi @nawramm

You can configure paperless to parse the filename for a date if you set PAPERLESS_FILENAME_DATE_ORDER.

I have an environment setting created during docker spinup like so:
PAPERLESS_DATE_ORDER=YMD
Is this different from PAPERLESS_FILENAME_DATE_ORDER?
Is this used in paperless.conf file ?
If so, is it picked up when paperless is run in a docker instance ?

You can also use a post consumption script to set the created date.

I tried going through the documentation, but am not sure how to go about it ?
How do I tell paperless (from the post consume bash script) as to what the date should be?

[stumpylog]

Is this different from PAPERLESS_FILENAME_DATE_ORDER?

Yes

If so, is it picked up when paperless is run in a docker instance ?

Depending on how you configured it. Docker normally uses the .env or environment:, but can load from a file if you told it too.

[selfyguy]

Thanks @nawramm and @stumpylog

How do I tell paperless (from the post consume bash script) as to what the date should be?

I believe this would be the best way to deal with my image requirements.
I tried going over the documentation but not able to find any pointers to such a possibility.

[nawramm]

There is an example script here written in python to to assign some new values to a document. The script also has these variables available for the document. If you know that the created time of the original file is true, you could use os.path.getctime(DOCUMENT_SOURCE_PATH) or something similar but I cannot guarantee it will work.

[selfyguy]

Thanks @nawramm. I will explore it further.

Is there a way to disable paperless from checking for dates of any sort from either filenames or the file contents?
According to the documentation here, the default value of none for PAPERLESS_FILENAME_DATE_ORDER will disable checking for dates in the filename.

No such information is available for PAPERLESS_DATE_ORDER.
What value should it be set to for disabling its functionality?

Would setting PAPERLESS_NUMBER_OF_SUGGESTED_DATES=0 have paperless disable looking for and using dates from the file contents?

[github-actions[bot]]

This discussion has been automatically closed due to inactivity. Please see our contributing guidelines for more details.