iOS app background sync fails

[sidamos]

The bug

I get a notification on my iPhone that Immich background sync starts. Shortly after, I get another notification that background sync failed.
However, there are no new photos to sync since last time I synced manually.
When I then start the app, the manual backup start button is disabled, of course, because there are no new photos.
For the time when I got the 2 notifications (12:18), there is nothing in the app log.
I started the app at 12:20.

The OS that Immich Server is running on

Gentoo Linux Docker

Version of Immich Server

v1.103.1

Version of Immich Mobile App

v1.103.0 build 155

Platform with the issue

• Server
• Web
• Mobile

Your docker-compose.yml content

name: immich

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    command: ['start.sh', 'immich']
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - .env
    ports:
      - 2283:3001
    depends_on:
      - redis
      - database
    restart: always

  immich-microservices:
    container_name: immich_microservices
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    # extends: # uncomment this section for hardware acceleration - see https://immich.app/docs/features/hardware-transcoding
    #   file: hwaccel.transcoding.yml
    #   service: cpu # set to one of [nvenc, quicksync, rkmpp, vaapi, vaapi-wsl] for accelerated transcoding
    command: ['start.sh', 'microservices']
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - .env
    depends_on:
      - redis
      - database
    restart: always

  immich-machine-learning:
    container_name: immich_machine_learning
    # For hardware acceleration, add one of -[armnn, cuda, openvino] to the image tag.
    # Example tag: ${IMMICH_VERSION:-release}-cuda
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
    # extends: # uncomment this section for hardware acceleration - see https://immich.app/docs/features/ml-hardware-acceleration
    #   file: hwaccel.ml.yml
    #   service: cpu # set to one of [armnn, cuda, openvino, openvino-wsl] for accelerated inference - use the `-wsl` version for WSL2 where applicable
    volumes:
      - model-cache:/cache
    env_file:
      - .env
    restart: always

  redis:
    container_name: immich_redis
    image: registry.hub.docker.com/library/redis:6.2-alpine@sha256:84882e87b54734154586e5f8abd4dce69fe7311315e2fc6d67c29614c8de2672
    restart: always

  database:
    container_name: immich_postgres
    image: registry.hub.docker.com/tensorchord/pgvecto-rs:pg14-v0.2.0@sha256:90724186f0a3517cf6914295b5ab410db9ce23190a2d9d0b9dd6463e3fa298f0
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
    volumes:
      - ${DB_DATA_LOCATION}:/var/lib/postgresql/data
    restart: always

volumes:
  model-cache:

Your .env content

# The location where your uploaded files are stored
UPLOAD_LOCATION=/extra/immich
# The location where your database files are stored
DB_DATA_LOCATION=/root/immich/postgres

# The Immich version to use. You can pin this to a specific version like "v1.71.0"
IMMICH_VERSION=release

# Connection secret for postgres. You should change it to a random password
DB_PASSWORD=xxx

# The values below this line do not need to be changed
###################################################################################
DB_USERNAME=postgres
DB_DATABASE_NAME=immich

Reproduction steps

1. All content is in sync
2. Notifications mentioned above appear
3.
...

Relevant log output

2024-05-07 12:20:41.647711 | INFO     | BackupNotifier       | _getBackupAlbumsInfo: Found 18 available albums |
2024-05-07 12:20:41.647249 | INFO     | BackupNotifier       | Found 18 local albums |
2024-05-07 12:20:31.598336 | INFO     | AssetNotifier        | Load assets: 166ms |
2024-05-07 12:20:31.598318 | INFO     | AlbumService         | Syncing completed. Changes: false |
2024-05-07 12:20:31.585086 | INFO     | AlbumService         | Found 4 device albums |
2024-05-07 12:20:31.526868 | INFO     | AssetNotifier        | Load partner assets: 94ms |
2024-05-07 12:20:31.458273 | INFO     | BackupNotifier       | _getBackupAlbumsInfo: Found 18 available albums |
2024-05-07 12:20:31.456814 | INFO     | BackupNotifier       | Found 18 local albums |
2024-05-07 12:20:31.437550 | INFO     | BackupNotifier       | [_resumeBackup] Start back up |
2024-05-07 10:34:52.329521 | INFO     | AssetNotifier        | Load partner assets: 157ms |
2024-05-07 10:34:52.328028 | INFO     | AssetNotifier        | Load assets: 134ms |
2024-05-07 10:34:52.327995 | INFO     | AlbumService         | Syncing completed. Changes: false |
2024-05-07 10:34:52.314200 | INFO     | AlbumService         | Found 4 device albums |
2024-05-07 10:34:52.312388 | INFO     | AlbumService         | refreshDeviceAlbums is already in progress |
2024-05-07 10:34:52.196082 | INFO     | BackupNotifier       | _getBackupAlbumsInfo: Found 18 available albums |
2024-05-07 10:34:52.194957 | INFO     | BackupNotifier       | Found 18 local albums |
2024-05-07 10:34:52.168016 | INFO     | BackupNotifier       | [_resumeBackup] Start back up |
2024-05-07 10:34:52.106121 | INFO     | ApiService           | Pinging server with response code 200 |
2024-05-06 22:31:00.842105 | INFO     | AssetNotifier        | Load assets: 192ms |
2024-05-06 22:31:00.842080 | INFO     | AlbumService         | Syncing completed. Changes: false |
2024-05-06 22:31:00.828798 | INFO     | AlbumService         | Found 4 device albums |
2024-05-06 22:31:00.785363 | INFO     | AssetNotifier        | Load partner assets: 169ms |
2024-05-06 22:31:00.773737 | INFO     | BackupNotifier       | _getBackupAlbumsInfo: Found 18 available albums |
2024-05-06 22:31:00.768362 | INFO     | BackupNotifier       | Found 18 local albums |
2024-05-06 22:31:00.727138 | INFO     | AlbumService         | refreshDeviceAlbums is already in progress |
2024-05-06 22:31:00.612356 | INFO     | BackupNotifier       | [_resumeBackup] Start back up |
2024-05-06 22:31:00.529656 | INFO     | ApiService           | Pinging server with response code 200 |

Additional information

No response

[sidamos]

More info:
I am using a subdomain via internet, SSL certificate is for the main domain, SSL verification in the app is therefore set to off.
Also, I am using Apache reverse proxy with basic auth. So, I have basic auth user and password in the URL.
Since the app is otherwise (besides the described error) working fine, I would expect that my setup is supported.

[alextran1502]

I am not very familiar with Apache proxy, does it have any setting to limit the body size to have a certain size?

[alextran1502]

Can you also check this guide out? https://immich.app/docs/administration/reverse-proxy#apache-example-config

[sidamos]

I don't know if there is a specific file size limit for Apache reverse proxy, I could not find any.

I also checked Apache error, access and request logs and there is nothing anywhere in the logs at the time I got the notifications on the phone. I do see the requests of the app when I started and used it, so the Apache logs are working.

In the meantime, I took a new photo and it did not get uploaded in the background after 10 hours, so the background sync really is not working. Then I started the app and it uploaded the photo immediately without issues.

I am also wondering why I do not see any log entries in the app's log for the failed background sync. Is this normal?

I am using your guide as base for my Apache config. I had just configured everything inside a location tag.
I have now changed that to look exactly like in your guide and have only the Basic Auth config inside a location tag.
Will see today, if that changes anything.
Edit: This did not help. Just got another notification on the phone about failed background sync.

[sidamos]

I now tried using the local network Immich docker URL and with that, the background sync works.

Then I switched to using Caddy instead of Apache as reverse proxy and the background sync fails also.
I also tried Apache without Basic Auth, also fails.

So, it is either the invalid SSL cert (although I disabled SSL verification in the app) or the reverse proxy setup of Apache and Caddy.

But I do not understand, why foreground sync always works with all setups and background sync fails. This I think should be seen as a bug.

And it does not help that background sync does not write into the log.

[bo0tzz]

#5562

[sidamos]

Thanks for the pointer! Sounds like my problem.
I am currently testing iOS app, real certificate (Let's encrypt) for my sub domain, reverse proxy via Caddy, no Basic Auth.
From the linked issue, I'd suspect that this also won't work, which would be a bummer and a show-stopper for me.
I'll add another comment here and to that issue when I know if it works or not.

[sidamos]

Found a working combination (besides direct local): Caddy Reverse Proxy, Subdomain, valid Let's Encrypt cert.