Proposal v4.0 b1 (HTTP/2+3, OS Trust Store, Custom DNS, OCSP, ...)

[Ousret]

This PR showcases how HTTPie could evolve outside of Requests.

Niquests is supposed to be a (mostly) compatible fork of Requests.

Try this preview:

$ pip install "git+https://github.com/Ousret/HTTPie.git@feature-tryout-niquests" -U

Here are the biggest pros of this:

• OS truststore by default, no more certifi!
• Object-oriented headers. Could bring additional features!
• Fully type-annotated!
• HTTP/3 over QUIC.
• HTTP/2 by default.
• Exit Python http.client! in favor of h11.
• Timeout by default.
• Inspect peer certificate, HTTP version, TLS version, cipher, and so on via hook/callback before a request is sent.
• All the features you expose are available in all three protocols.
• Python 3.7+, no sacrifice needed.
• Encrypted DNS support. w/ DNS-over-HTTPS, DNS-over-QUIC, DNS-over-TLS and plain DNS-over-UDP.

Obviously, cons:

• Stricter on emitted requests per RFCs
• Young project but based on solid bases, knowledge, and experiences.
• Need to publish new packages to distro. Easy but time-consuming
• Exit pyopenssl, not supported. more of a pro to me
• Require major bump? Could be. Should be.

Complete list of changes in the fork: https://github.com/jawah/niquests/blob/main/HISTORY.md

• Close httpie refuses to trust a self-signed certificate #576
• Close HTTPie ignores system certificates #480
• Close Support HTTP/2 (httpie-http2 plugin doesn't work) #692
• Close enable quic protocol #523
• Close show me the SSL certificate the server is using? #549
• Close System for meta information output options (timing, SSL info, …) #1023
• Close Include SSL/TLS and other metadata information in --meta/-vv #826
• Close "SSL: CERTIFICATE_VERIFY_FAILED" should show cert at least with --debug #800
• Close Display used server IP in verbose mode #1495
• Close SSL/TLS server certificate validation #1202
• Close Add equivalent cURL --resolve flag #99
• Close binding to an IP #1422
• Close Add -4 and -6 switches to force IP protocol version #94
• Close I would like the option to disable the DNS Cache and do name resolution on every request #1480
• Close Unable to install httpie via Chocolatey #1522
• Close Multiple response headers with same name combined to a single comma-separated list #1413
• Close Digest authentication with qop=auth-int fails "TypeError: expected string or bytes-like object"  #1446
• Close Should use the user-provided Host header for SNI #414
• Close HTTPie could be lead to believe data was passed in stdin when it was not #1551
• Close add local-port flag #1456
• Close can't install chocolatey issues with python error code 1603 #1338
• Close Bad filename when having non ASCII characters #1401
• Close Request is fired to packages.httpie.io on every invocation of http #1527

---

4.0.0.b1 (unreleased)

• Make it possible to unset the User-Agent, and Accept-Encoding headers. (#1502)
• Dependency on requests was changed in favor of compatible niquests. (#1531)
• Added support for HTTP/2, and HTTP/3 protocols. (#523) (#692) (#1531)
• Added request metadata for the TLS certificate, negotiated version with cipher, the revocation status and the remote peer IP address. (#1495) (#1023) (#826) (#1531)
• Added support to load the operating system trust store for the peer certificate validation. (#480) (#1531)
• Added detailed timings in response metadata with DNS resolution, established, TLS handshake, and request sending delays. (#1023) (#1531)
• Added support for using alternative DNS resolver using --resolver. DNS over HTTPS, DNS over TLS, DNS over QUIC, and DNS over UDP are accepted. (#99) (#1531)
• Added support for binding to a specific network adapter with --interface. (#1422) (#1531)
• Added support for specifying the local port with --local-port. (#1456) (#1531)
• Added support for forcing either IPv4 or IPv6 to reach the remote HTTP server with -6 or -4. (#94) (#1531)
• Removed support for pyopenssl. (#1531)
• Removed support for dead SSL protocols < TLS 1.0 (e.g. sslv3) as per pyopenssl removal. (#1531)
• Dropped dependency on requests_toolbelt in favor of directly including MultipartEncoder into HTTPie due to its direct dependency to requests. (#1531)
• Dropped dependency on multidict in favor of implementing an internal one due to often missing pre-built wheels. (#1522) (#1531)
• Fixed the case when multiple headers where concatenated in the response output. (#1413) (#1531)
• Fixed an edge case where HTTPie could be lead to believe data was passed in stdin, thus sending a POST by default. (#1551) (#1531)
  This fix has the particularity to consider 0 byte long stdin buffer as absent stdin. Empty stdin buffer will be ignored.
• Slightly improved performance while downloading by setting chunk size to -1 to retrieve packets as they arrive. (#1531)
• Added support for using the system trust store to retrieve root CAs for verifying TLS certificates. (#1531)
• Removed support for keeping the original casing of HTTP headers. This come from an outer constraint by newer protocols, namely HTTP/2+ that normalize header keys by default.
  From the HTTPie user perspective, they are "prettified" on the output by default. e.g. "x-hello-world" is displayed as "X-Hello-World".
• Fixed multipart form data having filename not rfc2231 compliant when name contain non-ascii characters. (#1401)
• Fixed issue where the configuration directory was not created at runtime that made the update fetcher run everytime. (#1527)

The plugins are expected to work without any changes. The only caveat would be that certain plugin explicitly require requests.
Future contributions may be made in order to relax the constraints where applicable.

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 91.26467% with 67 lines in your changes are missing coverage. Please review.

Project coverage is 94.31%. Comparing base (4d7d6b6) to head (eb93eab).
Report is 370 commits behind head on master.

❗ Current head eb93eab differs from pull request most recent head 6c55c94. Consider uploading reports for the commit 6c55c94 to get more accurate results

Files	Patch %	Lines
httpie/internal/encoder.py	90.86%	18 Missing ⚠️
httpie/cli/dicts.py	87.87%	12 Missing ⚠️
httpie/models.py	84.84%	10 Missing ⚠️
httpie/ssl_.py	82.69%	9 Missing ⚠️
httpie/client.py	89.58%	5 Missing ⚠️
httpie/internal/update_warnings.py	66.66%	5 Missing ⚠️
httpie/compat.py	60.00%	4 Missing ⚠️
tests/utils/__init__.py	90.00%	2 Missing ⚠️
tests/conftest.py	91.66%	1 Missing ⚠️
tests/test_cookie.py	90.00%	1 Missing ⚠️

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1531      +/-   ##
==========================================
- Coverage   97.28%   94.31%   -2.98%     
==========================================
  Files          67      117      +50     
  Lines        4235     8296    +4061     
==========================================
+ Hits         4120     7824    +3704     
- Misses        115      472     +357     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[Ousret]

So far, I identified this:

• Close httpie refuses to trust a self-signed certificate #576
• Close HTTPie ignores system certificates #480
• Close Support HTTP/2 (httpie-http2 plugin doesn't work) #692
• Close enable quic protocol #523
• Close show me the SSL certificate the server is using? #549

[Ousret]

We can add the following arguments and document them:

• --disable-http2
• --disable-http3
• --http3, e.g. force trying it.

[Ousret]

I've added and documented it.

• --disable-http2
• --disable-http3
• --http3, e.g. force trying it.

Ideally, the connection information (TLS, Cipher, Peer/Issuer Certificate, and OCSP) should be in the request metadata and with a lexer instruction.

[Ousret]

Now, the result is pleasant to the eye.
I have implemented the metadata for Request and the result is as follows:

Niquests now support tracking upload progress, this can also be done here. It should be straightforward enough.

[dwt]

Damn this looks so tantalizing. I'd love for this to get packaged as a tryout / beta. Something installable via e.g. pip install httpie[niquest] perhaps?

[Ousret]

So, I did some additional testing on my end and found nothing.
Additionally, to mark this milestone, the very last warning is now fixed!

This branch now have 0 warning. ============ 1039 passed, 5 skipped, 1 xfailed in 81.02s (0:01:21) =============

For a side note, Niquests just landed Happy Eyeballs feature in the last version, if this is of any interest to you, let me know.

and that commit cce24fe has nothing to do with Niquests or HTTPie really, you must have encountered this situation where a single MacOS test was flaky, this circumvent the random CI failures.

[Ousret]

I applied a minor patch to improve how the QuicCapabilityCache blend into HTTPie internals (the path is now generated within env::config and propagated appropriately). Tests added to ensure no regression happen in between.

[dwt]

@jkbrzt @Ousret is there anything I can do to help test this more? I really want to see this land. :)

[Ousret]

@dwt You can help by proof reading the modified documentation and look if I missed something regarding this PR in it: https://github.com/Ousret/httpie/blob/feature-tryout-niquests/docs/README.md

Other than that, I think the testing part is pretty much covered.

Finally I have added a test case for the following scenario: "A peer was H3 compatible but not anymore and expect a error msg that explain why you just got timeout and how to remedy".

[Ousret]

Issue #1401 was trivial to fix, so, it's fixed and tested.
Also #1527 this one was kind of tricky to track but fortunately fixed.
We've reached a total of 23 issues fixed 👍

[dwt]

@dwt You can help by proof reading the modified documentation and look if I missed something regarding this PR in it: https://github.com/Ousret/httpie/blob/feature-tryout-niquests/docs/README.md

I've had a look at the docs and so far everything seems fine. However I noticed the absence of a --http2 flag to force http2. Could you touch on the rationale behind that omission? I am currently debugging a server who has trouble spitting out correct http2, so this would be a useful feature to me.

[Ousret]

However I noticed the absence of a --http2 flag to force http2. Could you touch on the rationale behind that omission?

You cannot force HTTP/2 using Niquests. It is negotiated via ALPN/TLS. So if the server misinterpret ALPN or discard it, HTTP/2 won't come. And yes, RFC allows it, see https://datatracker.ietf.org/doc/html/rfc9113#name-starting-http-2-with-prior- but I decided to follow the simplest path forward (ALPN/TLS), as browsers do (https://stackoverflow.com/questions/34076231/why-do-browser-implementations-of-http-2-require-tls). I might reconsider it in the future, but the use cases are rare.

[dwt]

I may be missing something here, but does that imply that the client has no way to force http1 2 or 3 according to its wishes? When setting up server support for these protocols, I would like to have the ability to force all of these protocols for debugging purposes?

[Ousret]

I may be missing something here, but does that imply that the client has no way to force http1 2 or 3 according to its wishes?

There's many possibilities, the only missing piece is "force http2" or "disable http1".

• "--disable-http2" HTTP/1.1 or HTTP/3
• "--disable-http2 --disable-http3" HTTP/1.1
• "--disable-http3" HTTP/1.1 or HTTP/2
• "--http3" HTTP/3

Hope that clarify.

[Ousret]

Just updated the docs to include previous comment about protocol combo and the availability of HTTP/3 or not.
Since last time Niquests made HTTP/3 support far more enjoyable with less dependencies and a faster execution time.

Finally, we had to downgrade macos-14 (macos-latest) to macos-13 because it's relatively new and some dependencies like brotlicffi aren't ready for it.

[mshahat]

Hello @Ousret , nice work here ... is there an ETA on when to expect this release to become available? Thank you

[Ousret]

No precise ETA. We are waiting upon the owner approval.