You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
It seems to be fixed in transformers now, but updating the transformers package in the TGI container to the latest main branch did not fix the issue. Probably this is because TGI uses a separate implementation of StarCoder2 and not the one from transformers?
The text was updated successfully, but these errors were encountered:
System Info
Latest Docker image (sha-a70b087)
Model: TechxGenus/starcoder2-15b-AWQ.
Options:
output of
curl 127.0.0.1:8080/info
:Hardware: RTX 3090
Information
Tasks
Reproduction
TechxGenus/starcoder2-15b-AWQ
model (options mentioned above)The GPTQ variant of the same model works fine (but slow).
Expected behavior
The model should generate useful output.
There have already been discussions on a similar issue here:
https://huggingface.co/TechxGenus/starcoder2-7b-AWQ/discussions/1
huggingface/transformers#30225
huggingface/transformers#30074
It seems to be fixed in
transformers
now, but updating thetransformers
package in the TGI container to the latestmain
branch did not fix the issue. Probably this is because TGI uses a separate implementation of StarCoder2 and not the one fromtransformers
?The text was updated successfully, but these errors were encountered: