v4 keeps reporting HTTP Error 504 upstream request timeout very often

[ssbarnea]

First, please do not close this ticket because the problem is real and is far from being a one-off occasional network glitch. Apparently since v4 was released it was proved to be a huge source of problems even for people that started using tokens as recommended.

One last example at https://github.com/ansible/ansible-lint/actions/runs/8941534730/job/24562101600?pr=4131 - where 3/8 uploading jobs failed with 504, while 5 succeeded.

That is a 38% failure rate, at a time where both https://www.githubstatus.com/ and https://status.codecov.com/ report everything working perfectly.

Can we please do something to make codecov uploading more reliable?

debug - 2024-05-03 15:55:18,720 -- Upload result --- {"result": "RequestResult(error=RequestError(code='HTTP Error 504', params={}, description='upstream request timeout'), warnings=[], status_code=504, text='upstream request timeout')"}
[103](https://github.com/ansible/ansible-lint/actions/runs/8941534730/job/24562101600?pr=4131#step:13:104)
error - 2024-05-03 15:55:18,720 -- Upload failed: upstream request timeout

I am aware that others asked about #926 but we did not see any work being done to address it, neither to improve the reliability of the server side.

[rohan-at-sentry]

@ssbarnea happy to keep this open until you confirm.

We pushed changes 11 days ago that should help with this. I looked through some of the runs on ansible-lint from 2-3 days ago and it seems OK (I did notice a few failures on the macos runs which we'd be happy to help you debug if it's of interest; although they appear to be unrelated to this timeout issue)