Proposal: HTTP proxy support #309
Replies: 4 comments 16 replies
-
|
Any feedback from the team before I start implementing this feature? |
Beta Was this translation helpful? Give feedback.
-
|
Sorry for the late response! Some general thoughts: Something that we're lacking currently is the ability to use OSS packages through an internal mirror. This is a feature that we would like to land in our next release (0.26). I'm thinking that what we want here is actually a rewrite rule. As a practical example: it would be good for Also: I think we want the rewrite to be configurable on a machine level, via the settings file. Would also be good if this is configurable on a per-evaluator basis, and as well as a CLI flag, and via the settings in Sidenote: It might be good to discuss this in a more high throughput setting. A bunch of us hang out in the pkl-community discord channel. |
Beta Was this translation helpful? Give feedback.
-
|
A rewrite rule seems pretty similar to what a proxy does. What use-cases are there for proxies that a rewrite rule doesn't address? |
Beta Was this translation helpful? Give feedback.
-
|
The use case is to make Pkl work in environments where the Internet can only be accessed through an HTTP (forward) proxy. A rewrite rule doesn't help there--an HTTP proxy doesn't work like a package mirror. Your comments sound like you might be confusing forward proxy with reverse proxy. Here is how we'd want Pkl's |
Beta Was this translation helpful? Give feedback.
-
|
Thanks; I'm familiar with what (forward) proxies do. I'm wondering why you'd want a proxy, though? Both proxies and rewrite rules can achieve the goal of "I don't want to talk (directly) to the internet".
Yeah, makes sense. In that case, the need for proxying can't be avoided. |
Beta Was this translation helpful? Give feedback.
-
|
I'm not planning to work on the rewrite rule feature. But let me know if you'd like me to send a PR for the proxy feature. |
Beta Was this translation helpful? Give feedback.
-
|
This seems like something that would be useful, so I think go ahead and send in a PR. One thing that I'm not sure about here is the lack of HTTPS proxy support (i.e. ability to connect to a proxy under TLS). But, that's something that can always be added later. Some more notes:
Why do these seem pointless? Also under non-goals:
Mentioned this earlier, but: I think should have this as a setting in the Pkl settings file. This precedence seems to make the most sense:
Also: this should perhaps be an additional option in create evaluator request when Pkl is running in server mode. |
Beta Was this translation helpful? Give feedback.
-
Gradle uses the JVM proxy settings, so it should be good enough for Pkl to default to those. (I don't actually know if that's currently the case.) JVM proxy settings should also be good enough for Setting a proxy via |
Beta Was this translation helpful? Give feedback.
-
|
Thought about proxy env vars: it seems like a nightmare if different applications have a different syntax for what they expect Also: we try to not use environment variables to control how Pkl works. The only case today is |
Beta Was this translation helpful? Give feedback.
-
|
https://about.gitlab.com/blog/2021/01/27/we-need-to-talk-no-proxy/#http_proxy-and-https_proxy |
Beta Was this translation helpful? Give feedback.
-
|
@translatenix: were you still planning on submitting a PR for this? Absolutely no pressure to do so! If you weren't, let us know and we'll take this one over; this is something we'd like to ship in 0.26. |
Beta Was this translation helpful? Give feedback.
-
|
I haven’t started working on this, mainly because my HttpClient improvement PR is still open. |
Beta Was this translation helpful? Give feedback.
-
|
This one? #378 |
Beta Was this translation helpful? Give feedback.
-
I think it's best if you take over from here. |
Beta Was this translation helpful? Give feedback.
-
|
Sounds good |
Beta Was this translation helpful? Give feedback.
-
|
I went ahead and converted this into a SPICE, up for discussion here: apple/pkl-evolution#5 The main difference is how the proxy is configured; where it is funneled through the normal configuration means. Authentication is still an open question there |
Beta Was this translation helpful? Give feedback.
-
@stackoverflow suggested to have a discussion before I start working on this feature. Here is my proposal:
Features:
similar to cURL's
--noproxyand Java's-Dhttp.nonProxyHosts.http_proxy,HTTPS_PROXY,ALL_PROXYandNO_PROXYenvironment variables.https://everything.curl.dev/usingcurl/proxies/env
Java does not honor these environment variables:
https://bugs.openjdk.org/browse/JDK-8292530
API:
--proxy [http://]host[:port]https://curl.se/docs/manpage.html#-x
--noproxy <no-proxy-list>https://curl.se/docs/manpage.html#--noproxy
CliBaseOptionsHttpClient.Builderorg.pkl.executor.Executor(seems fairly pointless)Testing:
https://wiremock.org/docs/junit-jupiter/
https://www.mock-server.com/mock_server/running_mock_server.html#junit_test_extension
Open questions:
The only authentication method that is directly supported by
java.net.http.HttpClientis basic authentication.Basic proxy authentication is disabled by default (apparently because of the security risk) and can only be enabled via a system property.
--noproxy.cURL and Java support a slightly different syntax.
--noproxy.Java: "The default value excludes all common variations of the loopback address."
cURL: None.
Non-goals:
java.net.http.HttpClientdoes not support SOCKS proxies.Setting
java.net.Proxy.Type.SOCKShas no effect.https://bugs.openjdk.org/browse/JDK-8214516
java.net.http.HttpClientdoes not support HTTPS proxies.https://dev.to/kdrakon/httpclient-can-t-connect-to-a-tls-proxy-118a
Usually, SSL tunneling through an HTTP proxy should be good enough.
However, basic authentication with an HTTP proxy leaks credentials.
Java supports this feature (only) by using
java.net.ProxySelector.getDefault()and setting system property
java.net.useSystemProxies=true. The default isfalse.Neither cURL nor GraalVM native image supports this feature.
Using java.net.useSystemProxies has no effect oracle/graal#4285
PklProject,~/.pkl/someFile).Beta Was this translation helpful? Give feedback.
All reactions