perf(artifacts): pipeline the artifact file uploader

[moredatarequired]

Description

• Fixes WB-17571

Rewrite of ArtifactSaver.uploadFiles to use buffered channels and switch between available pending tasks

This is an alternative to #7274, and should be considered to supersede it. It's dramatically simpler, has similar (though slightly lower) performance, and handles retries better.

Key differences:

1. We don't impose an arbitrary minimum time (previously 1 hour) before retrying failed uploads
2. We bound the number of outstanding goroutines to limit maximum memory consumption
3. Overall this version is 20-25% faster for artifacts with many small files

Minor differences:
Differences:

1. I hope that this version is simplified and a little easier to follow, though YMMV
2. We use a smaller batch size (1000); this decreases stuttering and improves performance especially for 2k-10k file artifacts
3. We don't pause after requesting a batch of URLs, and can immediately start scheduling uploads made available from a previous batch
4. We process upload responses as they come in, rather than a tick-tock pattern where we alternate between scheduling them and processing them in ~10k batches
5. By selecting among all available tasks we are able to terminate earlier in case of fatal errors; this will also allow us to more easily integrate progress reporting with a time.Ticker channel in the future

Data flow:
Assemble batches
While not done:

1. If there are 1000 or fewer outstanding uploads, request upload URLs for a batch (in the background)
2. Select first available
   • Error to handle
   • File to schedule for upload
   • Upload result to check

If files fail to upload, collect them to retry. Retry all failed uploads by rerunning the above as long as each time we succeed for at least half the files.

• I updated CHANGELOG.md, or it's not applicable

Testing

Very extensively tested for performance (on extremely large artifacts).

Relies somewhat heavily on existing integration tests in CI for correctness.

[codecov]

Codecov Report

Attention: Patch coverage is 77.66990% with 23 lines in your changes are missing coverage. Please review.

Project coverage is 75.85%. Comparing base (849d919) to head (ea5e457).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #7586      +/-   ##
==========================================
+ Coverage   74.26%   75.85%   +1.58%     
==========================================
  Files         500      500              
  Lines       55732    54153    -1579     
==========================================
- Hits        41392    41080     -312     
+ Misses      13932    12663    -1269     
- Partials      408      410       +2     

Flag	Coverage Δ
func	41.37% <ø> (+0.33%)	⬆️
system	63.51% <76.69%> (+0.03%)	⬆️
unit	56.29% <0.97%> (+0.93%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files	Coverage Δ
...ore/internal/filetransfer/file_transfer_manager.go	85.13% <100.00%> (+18.81%)	⬆️
core/pkg/server/sender.go	71.65% <100.00%> (+9.01%)	⬆️
core/pkg/artifacts/saver.go	75.08% <76.76%> (+9.93%)	⬆️

... and 102 files with indirect coverage changes