Set max_tokens on model-level

[hofq]

Hey there,
ive wanted to ask a quick question:

how can i set max_tokens for a specific model?

We use the Azure OpenAI Endpoint with gpt-35-turbo in Version 1106. The Model's Token Limit from OpenAI Docs is 16385, but Librechat is Returning an Error on anything higher than 4092.

How can i overwrite that?

[danny-avila]

This is planned and have been working on this

[danny-avila]

Also if you name your model according to its version, it won’t be an issue. You can decouple the deployment name from the model name for more control by using the librechat.yaml file as well for azure configuration

https://docs.librechat.ai/install/configuration/azure_openai.html

[hofq]

i'm currently doing this, but the deployment name is the same. Do you recommend changing it?

    endpoints:
      azureOpenAI:
        plugins: true
        assistants: true
        groups:
        - group: "azure-openai" # arbitrary name
          plugins: true
          assistants: true
          apiKey: "${AZURE_CFG_API_KEY}"
          instanceName: "prod-ai-schweden"  # name of the resource group or instance
          version: "2024-02-15-preview"
          models:
            gpt-3.5-turbo:
              deploymentName: gpt-35-turbo
            gpt-4-vision-preview:
              deploymentName: gpt-4-vision-preview

[danny-avila]

Change it to this:

gpt-3.5-turbo-1106:
              deploymentName: gpt-35-turbo

[danny-avila]

This is a temporary “fix” and providing your own limits is planned

[hofq]

Works Thanks!

Also Thank you for Maintaining Librechat and offer Far more than ChatGPT can Handle, its an amazing Tool!

[danny-avila]

thank you so much!

Also will update the limit for gpt-3.5-turbo like you had it as it has changed to be the higher limit.