Too many concurrent requests trigger DoS protection

[flit]

CPM is too aggressive in parallelizing the index download. While this is great for index update performance, it can trigger the servers' CDN's DoS protection and the requests are blocked. Only Keil and NXP CDNs seem to have this response (they both use Akamai).

Specifically what happens is that the request queue can result in many concurrent requests targeted at one server. If the number of requests is greater than the maximum allowed per source IP, the requests are blocked and either a 403 or other error is returned. After triggering the DoS protection, typically all new requests from the source IP are blocked for a timeout period.

This is made far worse by #162 and #155, since the responses are not checked and the HTML error page included in the response is saved to the .pdsc index file (and then you get PDSC parse errors, of course).

The solution is to throttle the number of concurrent requests. It would be nice to have a maximum per server domain, but limiting the total number is a good first step that would solve the issue.

A while back I wrote the cmsis-pack-index-monitor script to explore this issue outside of CPM. You can use it to see the problem for yourself and experiment with a maximum number of jobs.