Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Inference shouldn't have a timeout #502

Closed
PlanetMacro opened this issue Apr 27, 2024 · 1 comment
Closed

Inference shouldn't have a timeout #502

PlanetMacro opened this issue Apr 27, 2024 · 1 comment

Comments

@PlanetMacro
Copy link

Especially with larger local models on OLLAMA, inference might take some time. Especially on the initial loading of the model. Currently devika throws a timeout, basically rendering its useless for such a setup.

@DGdev91
Copy link

DGdev91 commented May 1, 2024

Seems like the timeout is hardcoded in src/llm/llm.py

if int(elapsed_time) == 30:
    emit_agent("inference", {"type": "warning", "message": "Inference is taking longer than expected"})
if elapsed_time > 60:
    raise concurrent.futures.TimeoutError
time.sleep(1)
                    
response = future.result(timeout=60).strip()

As a quick hack to make it work you can increase those values, or you can just comment the fiirst 4 lines and remove the timeout on the last one.

Anyway, i agree. it shouldn't have a timeout, or at least it should be easly configurable to increase/disable it if needed

ARajgor added a commit that referenced this issue May 2, 2024
* chore: minor updates

* Add: send live inference time to frontend

* add: timeout in settings

* Improve: response parsing, add temperature to models

* patches

close #510, #507, #502, #468
@ARajgor ARajgor closed this as completed May 2, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants