Support range requests when downloading files

[zanieb]

The devpi server does not appear to support HTTP range requests when downloading files from the index. This can result in a significant slowdown of downloads during package resolution.

Are there blockers for supporting this?

[fschulze]

In what way are range requests used for package resolution?

There are no blockers I can think of right now, but it isn't trivial to implement fully. A first step would be to implement it when the file exists locally. If it has to be streamed from the primary server in a replicated setup or from a mirror, then it would require a possibly much more complex implementation.

I'm uncertain about the usefulness though, as it is much easier in most cases to put nginx in front and let it handle files directly. This only works with the default storage backend though.

[zanieb]

In what way are range requests used for package resolution?

For servers that do not support PEP 658, we need to download a wheel to determine package metadata for resolution. We can use range requests to download the necessary portion of the wheel instead of the entire thing.

pip does this e.g. https://github.com/pypa/pip/blob/e88d39ae49ab11a8b80609a018e5a36ea4ccad89/src/pip/_internal/network/lazy_wheel.py#L23-L37

I agree in production it'd make sense to place nginx in front of the server, but I'm just trying to run a lightweight server for testing purposes (which seems like a good part of the intent of this project).

[fschulze]

It should be fairly straight forward to add this to _pkgserv as long as you don't involve the should_fetch_remote_file branch. Maybe there even already is support for that when serving static files with pyramid, I didn't check yet. Basically at the end of the function where we check for HEAD requests one could check for range requests and return accordingly.