Skip to content

Commit

Permalink
fix(autogptq): do not use_triton with qwen-vl (#1985)
Browse files Browse the repository at this point in the history
* Enhance autogptq backend to support VL models

* update dependencies for autogptq

* remove redundant auto-gptq dependency

* Convert base64 to image_url for Qwen-VL model

* implemented model inference for qwen-vl

* remove user prompt from generated answer

* fixed write image error

* fixed use_triton issue when loading Qwen-VL model

---------

Co-authored-by: Binghua Wu <[email protected]>
  • Loading branch information
2 people authored and mudler committed Apr 11, 2024
1 parent d692b2c commit 0004ec8
Showing 1 changed file with 0 additions and 1 deletion.
1 change: 0 additions & 1 deletion backend/python/autogptq/autogptq.py
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,6 @@ def LoadModel(self, request, context):
self.model_name = "Qwen-VL-Chat"
model = AutoModelForCausalLM.from_pretrained(model_path,
trust_remote_code=request.TrustRemoteCode,
use_triton=request.UseTriton,
device_map="auto").eval()
else:
model = AutoGPTQForCausalLM.from_quantized(model_path,
Expand Down

0 comments on commit 0004ec8

Please sign in to comment.