Use rq_redis_connection when listing RQ jobs in remove_ghost_locks maintenance task

[SchrosCat2013]

What type of PR is this?

• Refactor
• Feature
• Bug Fix
• New Query Runner (Data Source)
• New Alert Destination
• Other

Description

Fix for UnicodeDecodeError in remove_ghost_locks mainentance task.

Redash has two connections to Redis - one for RQ, which does not support decode_responses in it's connection string, and one for Redash direct access to Redis, which expects decode_responses to be set. There is one place in the remove_ghost_locks call stack where the Redash connection is used in place of the RQ one, causing UnicodeDecodeErrors.

Discovered while investigating #6424

How is this tested?

• Unit tests (pytest, jest)
• E2E Tests (Cypress)
• Manually
• N/A

• Modified the docker-compose file - set the worker command to worker instead of dev_worker (to make the healthcheck tasks run)
• Started a long-running query (SELECT pg_sleep(300))
• Before change - observe several UnicodeDecodeError: 'utf-8' codec can't decode byte 0x9c in position 1: invalid start byte in docker logs of redash-worker-1 during query execution
• After change - query completes without UnicodeDecodeError in redash-worker-1 logs

[justinclift]

This sounds logical to me, and seems pretty reasonable.

@guidopetri Assuming this passes our CI tests, are we ok to merge it? 😄

[guidopetri]

I don't think this change is correct. The only reference I see to rq_job_ids() is in redash/tasks/queries/maintenance.py, remove_ghost_locks(), which uses the jobs list returned by rq_job_ids to filter and delete locks present in redis_connection. So I think the function name must somehow be conflating what's going on. The title of this PR seems to be correct, but I don't think that the logic fix is correct.

@SchrosCat2013 can you verify if this function is being called correctly? Maybe look into the logging and see if the log line post-lock deletion is present + working as expected (i.e. actually removing locks that should be removed)?

[guidopetri]

(placeholder)