Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEATURE] Allow setting up customized http client to the underlying API #969

Open
kchaber opened this issue Apr 18, 2024 · 5 comments
Open
Labels
enhancement New feature or request P2 High priority P3 Medium priority

Comments

@kchaber
Copy link

kchaber commented Apr 18, 2024

Is your feature request related to a problem? Please describe.
Currently, it is impossible to provide the customized http client using the available langchain4j builders because the OkHttpClient instantiation is hardcoded in numerous places, e.g. OllamaClient:

@Builder
public OllamaClient(String baseUrl, Duration timeout) {

    OkHttpClient okHttpClient = new OkHttpClient.Builder()
            .callTimeout(timeout)
            .connectTimeout(timeout)
            .readTimeout(timeout)
            .writeTimeout(timeout)
            .build();

    Retrofit retrofit = new Retrofit.Builder()
            .baseUrl(baseUrl)
            .client(okHttpClient)
            .addConverterFactory(GsonConverterFactory.create(GSON))
            .build();

    ollamaApi = retrofit.create(OllamaApi.class);
}

As stated above, only the timeout property is passed down to the http client using OllamaChatModel.builder().timeout(...) method. There are numerous cases, where developers need to override some other properties of the underlying http client, e.g. most often but not only disabling: the ssl verification (https://stackoverflow.com/questions/25509296/trusting-all-certificates-with-okhttp).

Describe the solution you'd like
Expose the builder method to pass the http client down to the underlying API, e.g. for ollama:

OllamaChatModel.builder()
      .httpClient(...)
      .build();
@kchaber kchaber added the enhancement New feature or request label Apr 18, 2024
@kchaber kchaber changed the title [FEATURE] Allow setting up customized OkHttpClient [FEATURE] Allow setting up customized http client to the underlying API Apr 18, 2024
@langchain4j
Copy link
Owner

Implementation: #911

@langchain4j langchain4j added P2 High priority P3 Medium priority labels Apr 22, 2024
@madroidmaq
Copy link

I think this feature is very useful, and we can implement a monitoring tool similar to Langsmith's based on Okhttp's interceptor mechanism, so I expect he'll be able to smash more models to be supported, such as OpenAI's.

OpenAiChatModel.builder()
      .httpClient(...)
      .build();

@langchain4j
Copy link
Owner

We plan to get rid of okhttp as default http client in the near future, so please do not implement anything okhttp-specific.

@madroidmaq
Copy link

We plan to get rid of okhttp as default http client in the near future, so please do not implement anything okhttp-specific.

Can you give us the direction of future evolution? Are there any plans to support Kotlin Multiplatform?

@langchain4j
Copy link
Owner

@madroidmaq this is planned in #1044
The idea is to introduce an abstraction (e.g. HttpClient) and decouple llm/vector store implementations from okhttp.
This way users will be able to use other http clients. Okhttp can stay (if needed) as one of the implementations, but most probably it won't be the default one.
II am not familiar with Kotlin Multiplatform, so can't comment here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request P2 High priority P3 Medium priority
Projects
None yet
Development

No branches or pull requests

3 participants