请问是否有q4精度的模型提供？

[yangzhaofeng]

llama.cpp 里面那个转换脚本转q4的功能是坏的。原版的llama模型huggingface上有人转好了q4放出来，但是中文模型还没有。请问以后是否有考虑放出q4精度的模型？

[ymcui]

你创建一个新issue吧，描述一下使用llama.cpp具体遇到什么问题。

[yangzhaofeng]

目前看来是llama.cpp中 convert.py 的问题

$ python3 ~/llama.cpp/convert.py --outtype q4_0 pytorch_model-00001-of-00002.bin

Loading model file pytorch_model-00001-of-00002.bin
Loading model file pytorch_model-00001-of-00002.bin
Loading model file pytorch_model-00002-of-00002.bin
vocabtype: spm
Loading vocab file tokenizer.model
params: n_vocab:55296 n_embd:4096 n_mult:5504 n_head:32 n_layer:32
Traceback (most recent call last):
  File "/home/steven/llama.cpp/convert.py", line 1326, in <module>
    main()
  File "/home/steven/llama.cpp/convert.py", line 1319, in main
    model = convert_to_output_type(model, output_type)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/steven/llama.cpp/convert.py", line 1153, in convert_to_output_type
    return {name: tensor.astype(output_type.type_for_tensor(name, tensor))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/steven/llama.cpp/convert.py", line 1153, in <dictcomp>
    return {name: tensor.astype(output_type.type_for_tensor(name, tensor))
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/steven/llama.cpp/convert.py", line 626, in astype
    self.validate_conversion_to(data_type)
  File "/home/steven/llama.cpp/convert.py", line 637, in validate_conversion_to
    raise Exception(f"Can't turn an unquantized tensor into a quantized type ({data_type})")
Exception: Can't turn an unquantized tensor into a quantized type (QuantizedDataType(groupsize=32, have_addends=False, have_g_idx=False))

根据 ggerganov/llama.cpp#1347 convert.py 无法转q4是正常现象

[ymcui]

正常应该是两步：

1. python convert.py转成F16的GGUF
2. 然后再用./quantize 把F16转成Q4或其他量化版

另外，最新版llama.cpp的convert.py不支持直接转成Q4。

  --outtype {f32,f16,q8_0}