InternLM-XComposer2-4KHD-7B 多卡推理

[Cloopen-ReLiNK]

No description provided.

[Cloopen-ReLiNK]

RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:2 and cuda:0! (when checking argument for argument weight in method wrapper_CUDA__cudnn_convolution)

[Kyoani-0718]

Did you use lora to fine-tune it? Just load your model like this:

model = AutoPeftModelForCausalLM.from_pretrained(
    '//InternLM-XComposer/finetune/output/lora_finetune_1',
    # device_map="auto",
    trust_remote_code=True
).cuda().eval()

tokenizer = AutoTokenizer.from_pretrained('internlm/internlm-xcomposer2-4khd-7b', trust_remote_code=True)