Align Ollama DEFAULT_CONTEXT_WINDOW to match with Ollama CLI default: 2048 #13139
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Ollama library in llama index is set to CONTEXT WINDOW to 3094
n_ctx = 3904
which is higher than Ollama CLI default of 2048n_ctx = 2048
causing weird output when running even simple queries onllama3:instruct
model. This thereby deters new users/developers from being able to get llama-index up and running quickly with Ollama (llama3). In contrast, langchain works out of the box with the same configuration that aligns with the Ollama CLI so that performance/consistency is retained out of the box. In other words, running a query on Ollama interactive CLIollama run
, and running it on Llama-index library (with defaults) should be identical.Fixes #13106
Fixes Timeout, Junk Output, and
ggml_metal_graph_compute: command buffer 3 failed with status 5
error due to mismatch in default Context Window between Ollama CLI and Llama-index integration.New Package?
Did I fill in the
tool.llamahub
section in thepyproject.toml
and provide a detailed README.md for my new integration or package?Version Bump?
Did I bump the version in the
pyproject.toml
file of the package I am updating? (Except for thellama-index-core
package)Type of Change
Please delete options that are not relevant.
How Has This Been Tested?
Just a change in default value.
To reproduce:
Hardware/bootstrap logs:
Command:
On Llama-index:
Output:
Logs from Ollama server:
Suggested Checklist:
make format; make lint
to appease the lint gods