Batch Import Stuck Pending

[timespaced]

Uploading a populated .csv (using the supplied template on the Import page) gets stuck on pending. The file is uploaded to my local uploads folder correctly (random filename.csv), owned by root and chmod'd 600.

Below is the output from docker logs sl-app

2021-06-04 04:29:31,088 - SL - DEBUG - "/code/server.py:444" - after_request() -  - 240.0.0.1 GET /dashboard/batch_import ImmutableMultiDict([]) 200
2021-06-04 04:29:31,768 - SL - DEBUG - "/code/server.py:444" - after_request() -  - 240.0.0.1 GET /api/notifications ImmutableMultiDict([('page', '0')]) 200
2021-06-04 04:29:37,064 - SL - DEBUG - "/code/app/dashboard/views/batch_import.py:30" - batch_import_route() -  - upload file <File gxehetnrsznhloekydmx.csv> to s3 at gxqhetndfvnhloekbnmx.csv
2021-06-04 04:29:37,067 - SL - DEBUG - "/code/app/dashboard/views/batch_import.py:34" - batch_import_route() -  - Add a batch import job <BatchImport 2> for <User 1  REDACTED@REDACTED>
2021-06-04 04:29:37,072 - SL - DEBUG - "/code/server.py:444" - after_request() -  - 240.0.0.1 POST /dashboard/batch_import ImmutableMultiDict([]) 302
2021-06-04 04:29:37,220 - SL - DEBUG - "/code/server.py:444" - after_request() -  - 240.0.0.1 GET /dashboard/batch_import ImmutableMultiDict([]) 200
2021-06-04 04:29:37,824 - SL - DEBUG - "/code/server.py:444" - after_request() -  - 240.0.0.1 GET /api/notifications ImmutableMultiDict([('page', '0')]) 200
2021-06-04 04:29:40,668 - SL - DEBUG - "/code/server.py:444" - after_request() -  - 240.0.0.1 GET /dashboard/batch_import ImmutableMultiDict([]) 200
2021-06-04 04:29:41,261 - SL - DEBUG - "/code/server.py:444" - after_request() -  - 240.0.0.1 GET /api/notifications ImmutableMultiDict([('page', '0')]) 200
2021-06-04 04:29:42,085 - SL - DEBUG - "/code/server.py:444" - after_request() -  - 240.0.0.1 GET /dashboard/batch_import ImmutableMultiDict([]) 200
2021-06-04 04:29:43,037 - SL - DEBUG - "/code/server.py:444" - after_request() -  - 240.0.0.1 GET /api/notifications ImmutableMultiDict([('page', '0')]) 200

Any help would be greatly appreciated.

[nguyenkims]

@haltdev there's a job_runner.py script that's used for running background jobs. This script can be simply run as python job_runner.py so if are using the self-hosting instructions, it can be run as

docker run -d \
    --name sl-email \
    -v $(pwd)/sl:/sl \
    -v $(pwd)/sl/upload:/code/static/upload \
    -v $(pwd)/simplelogin.env:/code/.env \
    -v $(pwd)/dkim.key:/dkim.key \
    -v $(pwd)/dkim.pub.key:/dkim.pub.key \
    -p 20381:20381 \
    --restart always \
    --network="sl-network" \
    simplelogin/app:3.4.0 python job_runner.py

The reason this one isn't included in the self-hosting instruction as the background job is mostly used for sending marketing emails which isn't needed for a self-hosting setup.

[timespaced]

@haltdev there's a job_runner.py script that's used for running background jobs. This script can be simply run as python job_runner.py so if are using the self-hosting instructions, it can be run as

docker run -d \
    --name sl-email \
    -v $(pwd)/sl:/sl \
    -v $(pwd)/sl/upload:/code/static/upload \
    -v $(pwd)/simplelogin.env:/code/.env \
    -v $(pwd)/dkim.key:/dkim.key \
    -v $(pwd)/dkim.pub.key:/dkim.pub.key \
    -p 20381:20381 \
    --restart always \
    --network="sl-network" \
    simplelogin/app:3.4.0 python job_runner.py

The reason this one isn't included in the self-hosting instruction as the background job is mostly used for sending marketing emails which isn't needed for a self-hosting setup.

That worked! I had to rename the existing sl-email container to run this. I imagine the run you listed only needs to occur when a batch import is being used? Should I destroy the container and run again the original from the initial setup ("email handler")?

Also, how can I remove the "stuck" pending jobs - I had deleted the .csv files before your reply in hopes that would clear them but it did not.

Thank you!

[nguyenkims]

That worked! I had to rename the existing sl-email container to run this. I imagine the run you listed only needs to occur when a batch import is being used? Should I destroy the container and run again the original from the initial setup ("email handler")?

Ah sorry, I should have used another container name in the last command (i.e. the one that runs the "job_runner.py"). You can leave the job_runner run forever like the "email handler" (this is how we use it on the SaaS version).

Also, how can I remove the "stuck" pending jobs - I had deleted the .csv files before your reply in hopes that would clear them but it did not.

You don't have to, the pending jobs that fail will just be ignored. And it's ok to import an alias that already exists: the import script will just ignore the alias in this case.