Paperless Mail: consumer Permission denied in docker /tmp

[hendrik1120]

Hi, I am just trying out Paperless Mail and added tika aswell as gotenberg to the docker compose.

Mail consumption worked rly well until I wanted to try out different mail rules.
I deleted the already consumed documents and also the entry in processed mails to consume the same mail with different settings (eml and attachments) which worked fine.
Consuming other emails after that fails the consumption task with this strange error:

Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/celery/app/trace.py", line 477, in trace_task
R = retval = fun(*args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/celery/app/trace.py", line 760, in __protected_call__
return self.run(*args, **kwargs)
File "/usr/src/paperless/src/documents/tasks.py", line 156, in consume_file
document = Consumer().try_consume_file(
File "/usr/src/paperless/src/documents/consumer.py", line 352, in try_consume_file
self.pre_check_duplicate()
File "/usr/src/paperless/src/documents/consumer.py", line 140, in pre_check_duplicate
with open(self.path, "rb") as f:
PermissionError: [Errno 13] Permission denied: '/tmp/paperless/paperless-mail-fiddk6wh'

Since the tmp directory is in the docker image, I thought that compose down&up would fix it, but the error is persistent. Even deleting the mail account and rules to maybe cascade any leftover db entries doesn't fix it.
Consuming the same documents through the web ui works just fine.
I am using python mange.py mail_fetcher to instantly fetch the mails, which to my surprise executes the task in the shell instead of scheduling it for the worker. There also isn't any overlap with the scheduled task every 10 minutes, which could cause a permission issue.

root@d8d84fbd5875:/tmp/paperless# ls -la
total 4584
drwxr-xr-x 2 paperless users       8 Sep  7 22:12 .
drwxrwxrwt 4 root      root        4 Sep  7 22:12 ..
-rw------- 1 root      root   333815 Sep  7 22:12 paperless-mail-6u9v4arw
-rw------- 1 root      root    45898 Sep  7 22:03 paperless-mail-81m8rv7c.eml
-rw------- 1 root      root   574715 Sep  7 22:12 paperless-mail-fiddk6wh
-rw------- 1 root      root  2418206 Sep  7 22:12 paperless-mail-nb7onpr3.eml
-rw------- 1 root      root   438727 Sep  7 22:12 paperless-mail-qz5_njf2
-rw------- 1 root      root   430071 Sep  7 22:12 paperless-mail-v6w7gao4

I don't think I did something too unreasonable to cause this, despite trying to consume the same mail twice.
Maybe should have tested it with a dev instance instead 😕

Paperless 1.17.4

version: "3.8"
services:
  redis:
    image: redis:alpine
    restart: unless-stopped
    environment:
      - PUID=99
      - PGID=100
    volumes:
      - /mnt/user/appdata/Paperless/redis:/data
    networks:
      - default


  postgres:
    image: postgres:15-alpine
    restart: unless-stopped
    volumes:
      - /mnt/user/appdata/Paperless/postgres15:/var/lib/postgresql/data

    environment:
      - POSTGRES_DB=paperless
      - POSTGRES_USER=paperless
      - POSTGRES_PASSWORD=${PAPERLESS_DBPASS}
      - PUID=99
      - PGID=100
    networks:
      - default

  webserver:
    image: ghcr.io/paperless-ngx/paperless-ngx:latest
    restart: unless-stopped
    depends_on:
      - postgres
      - redis
    healthcheck:
      test: [ "CMD", "curl", "-fs", "-S", "--max-time", "2", "http://localhost:8000" ]
      interval: 30s
      timeout: 10s
      retries: 5
    environment:
      - PAPERLESS_SECRET_KEY=${PAPERLESS_SECRET_KEY}
      - PAPERLESS_COOKIE_PREFIX=paperless
      - PAPERLESS_URL=${PAPERLESS_URL}
      - PAPERLESS_ALLOWED_HOSTS=localhost,192.168.1.10
      - PAPERLESS_REDIS=redis://redis:6379
      - PAPERLESS_DBHOST=postgres
      - PAPERLESS_DBPASS=${PAPERLESS_DBPASS}
      - PAPERLESS_ENABLE_HTTP_REMOTE_USER=true
      - PAPERLESS_LOGOUT_REDIRECT_URL=https://auth.example.com/logout
      - PAPERLESS_OCR_LANGUAGES=eng deu
      - PAPERLESS_OCR_LANGUAGE=deu
      - PAPERLESS_FILENAME_FORMAT={created}-{correspondent}-{title}
      - PAPERLESS_CONSUMER_ENABLE_BARCODES=true
      - PAPERLESS_CONSUMER_ENABLE_ASN_BARCODE=true
      - PAPERLESS_CONSUMER_BARCODE_SCANNER=ZXING
      - PAPERLESS_TIKA_ENABLED=1
      - PAPERLESS_TIKA_GOTENBERG_ENDPOINT=http://gotenberg:3000
      - PAPERLESS_TIKA_ENDPOINT=http://tika:9998
      - PAPERLESS_TIME_ZONE=Europe/Berlin
      - USERMAP_UID=99
      - USERMAP_GID=100
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.paperless.rule=Host(`paper.example.com`)"
      - "traefik.http.routers.paperless.entryPoints=https"
      - "traefik.http.services.paperless.loadbalancer.server.port=8000"
      - "traefik.http.routers.paperless.middlewares=auth@file"
    volumes:
      - /mnt/user/appdata/Paperless/paperless-ngx:/usr/src/paperless/data
      - /mnt/user/paperless/:/usr/src/paperless/media
      - /mnt/user/paperless/export/:/usr/src/paperless/export
      - /mnt/user/paperless/consume/:/usr/src/paperless/consume
    networks:
      - default
      - proxylocal

  gotenberg:
    image: docker.io/gotenberg/gotenberg:7.8
    restart: unless-stopped
    # The gotenberg chromium route is used to convert .eml files. We do not
    # want to allow external content like tracking pixels or even javascript.
    command:
      - "gotenberg"
      - "--chromium-disable-javascript=true"
      - "--chromium-allow-list=file:///tmp/.*"

  tika:
    image: ghcr.io/paperless-ngx/tika:latest
    restart: unless-stopped

networks:
  default:
    ipam:
      config:
        - subnet: 172.21.4.0/24
  proxylocal:
    external: true

[hendrik1120]

Wow, this is indeed a bug and I probably should have noticed it earlier.

Because the mail_fetcher task is executed in the shell with root permissions when using the management command, the temp files receive the wrong permissions.

Changing tasks.process_mail_accounts() to tasks.process_mail_accounts.delay() in mail_fetcher.py to execute the task by the celery worker fixes this issue. :D

Let me know if I should open an issue or pr for this.

[stumpylog]

You should use the provided mail_fetcher command or be familiar with Docker permissions, users and usage of gosu

[hendrik1120]

I am indeed not familiar with gosu and thought “management command” means manage.py cmd and not that there are actual custom commands.
Thanks for your quick answer.

An opinion on user facing management commands: Just integrate them into the webui, you already have celery to execute them asynchronously. There is too much that can go wrong.

[github-actions[bot]]

This discussion has been automatically locked since there has not been any recent activity after it was closed. Please open a new discussion for related concerns.