Llama-2-7b-chat-hf or similar working?

[mark-ov]

I have been trying a dozen different way. Asking Claude 2, GPT-4, Code Interpreters you name it. I for the life of me cannot figure out how to get the llama-2 models either to download or load the ones I have pulled from hugging face.

I have already been accepted via META and Hugging face, these models are not gated for me.

Gated model You have been granted access to this model

I have logged into my HF account through huggingface-cli and added the token and all that.

I have pip install "openllm[llama]"

I have tried

openllm download llama  --model-version meta-llama/Llama-2-7b-chat

To which it stops for a minute as if it's going to do something and then spits out:

Traceback (most recent call last):
  File "/home/mark_ov/anaconda3/envs/openllm/lib/python3.11/site-packages/openllm/_llm.py", line 730, in _infer_tag_from_model_id
    return bentoml.Tag.from_taglike(model_id)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mark_ov/anaconda3/envs/openllm/lib/python3.11/site-packages/bentoml/_internal/tag.py", line 96, in from_taglike
    return cls.from_str(taglike)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/mark_ov/anaconda3/envs/openllm/lib/python3.11/site-packages/bentoml/_internal/tag.py", line 101, in from_str
    return cls(tag_str, None)
           ^^^^^^^^^^^^^^^^^^
  File "/home/mark_ov/anaconda3/envs/openllm/lib/python3.11/site-packages/bentoml/_internal/tag.py", line 55, in __init__
    validate_tag_str(lname)
  File "/home/mark_ov/anaconda3/envs/openllm/lib/python3.11/site-packages/bentoml/_internal/tag.py", line 40, in validate_tag_str
    raise ValueError(
ValueError: huggyllama/llama-7b is not a valid BentoML tag: a tag's name or version must consist of alphanumeric characters, '_', '-', or '.', and must start and end with an alphanumeric character

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/mark_ov/anaconda3/envs/openllm/lib/python3.11/site-packages/transformers/utils/hub.py", line 417, in cached_file
    resolved_file = hf_hub_download(
                    ^^^^^^^^^^^^^^^^
  File "/home/mark_ov/anaconda3/envs/openllm/lib/python3.11/site-packages/huggingface_hub/utils/_validators.py", line 118, in _inner_fn
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/home/mark_ov/anaconda3/envs/openllm/lib/python3.11/site-packages/huggingface_hub/file_download.py", line 1291, in hf_hub_download        
    raise LocalEntryNotFoundError(
huggingface_hub.utils._errors.LocalEntryNotFoundError: Connection error, and we cannot find the requested files in the disk cache. Please try again or make sure your Internet connection is on.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/mark_ov/anaconda3/envs/openllm/bin/openllm", line 8, in <module>
    sys.exit(cli())
             ^^^^^
  File "/home/mark_ov/anaconda3/envs/openllm/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mark_ov/anaconda3/envs/openllm/lib/python3.11/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/home/mark_ov/anaconda3/envs/openllm/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mark_ov/anaconda3/envs/openllm/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mark_ov/anaconda3/envs/openllm/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mark_ov/anaconda3/envs/openllm/lib/python3.11/site-packages/openllm/cli.py", line 387, in wrapper
    return func(*args, **attrs)
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/mark_ov/anaconda3/envs/openllm/lib/python3.11/site-packages/openllm/cli.py", line 361, in wrapper
    return_value = func(*args, **attrs)
                   ^^^^^^^^^^^^^^^^^^^^
  File "/home/mark_ov/anaconda3/envs/openllm/lib/python3.11/site-packages/openllm/cli.py", line 337, in wrapper
    return f(*args, **attrs)
           ^^^^^^^^^^^^^^^^^
  File "/home/mark_ov/anaconda3/envs/openllm/lib/python3.11/site-packages/openllm/cli.py", line 1303, in download_models_command
    llm = openllm.infer_auto_class(impl).for_model(
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mark_ov/anaconda3/envs/openllm/lib/python3.11/site-packages/openllm/models/auto/factory.py", line 130, in for_model
    llm = model_class.from_pretrained(model_id, model_version=model_version, llm_config=llm_config, **attrs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mark_ov/anaconda3/envs/openllm/lib/python3.11/site-packages/openllm/_llm.py", line 708, in from_pretrained
    _tag = cls._infer_tag_from_model_id(model_id, model_version)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mark_ov/anaconda3/envs/openllm/lib/python3.11/site-packages/openllm/_llm.py", line 732, in _infer_tag_from_model_id
    return make_tag(
           ^^^^^^^^^
  File "/home/mark_ov/anaconda3/envs/openllm/lib/python3.11/site-packages/openllm/_llm.py", line 172, in make_tag
    transformers.AutoConfig.from_pretrained(
  File "/home/mark_ov/anaconda3/envs/openllm/lib/python3.11/site-packages/transformers/models/auto/configuration_auto.py", line 983, in from_pretrained
    config_dict, unused_kwargs = PretrainedConfig.get_config_dict(pretrained_model_name_or_path, **kwargs)
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mark_ov/anaconda3/envs/openllm/lib/python3.11/site-packages/transformers/configuration_utils.py", line 617, in get_config_dict      
    config_dict, kwargs = cls._get_config_dict(pretrained_model_name_or_path, **kwargs)
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mark_ov/anaconda3/envs/openllm/lib/python3.11/site-packages/transformers/configuration_utils.py", line 672, in _get_config_dict     
    resolved_config_file = cached_file(
                           ^^^^^^^^^^^^
  File "/home/mark_ov/anaconda3/envs/openllm/lib/python3.11/site-packages/transformers/utils/hub.py", line 452, in cached_file
    raise EnvironmentError(
OSError: We couldn't connect to 'https://huggingface.co' to load this file, couldn't find it in the cached files and it looks like huggyllama/llama-7b is not the path to a directory containing a file named config.json.
Checkout your internet connection or see how to run the library in offline mode at 'https://huggingface.co/docs/transformers/installation#offline-mode'.

I have also tried downloading llama-2 7b-chat and 13b-chat directly from huggingface via GIT and I do have the files but I cannot get openLLM to utilize/locate them. I also think there's an issue with the weights not being tied to the model and I am not sure how I do that or how I even get back to a place to replicate that error.

So I have been trying to download/start them with OPENLLM to no avail.

Anyone able to assist me with getting the 7b-chat and 13b-chat models working? Or help point me in the right direction.

[aarnphm]

This is incorrect, it should be openllm import llama meta-llama/Llama-2-7b-chat-hf. Currently I haven't implemented the weight conversion to huggingface format yet. I can add it promptly.

Can you add the version of openllm you are using?

[mark-ov]

openllm, version 0.2.0

openllm import llama meta-llama/Llama-2-7b-chat-hf
Converting 'meta-llama/Llama-2-7b-chat-hf' to lowercase: 'meta-llama/llama-2-7b-chat-hf'.
Converting 'meta-llama/Llama-2-7b-chat-hf' to lowercase: 'meta-llama/llama-2-7b-chat-hf'.
Converting 'pt-meta-llama-Llama-2-7b-chat-hf' to lowercase: 'pt-meta-llama-llama-2-7b-chat-hf'.
Converting 'pt-meta-llama-Llama-2-7b-chat-hf' to lowercase: 'pt-meta-llama-llama-2-7b-chat-hf'.
'llama' with 'model_id=meta-llama/Llama-2-7b-chat-hf' does not exists in local store. Saving...
bin /home/mark_ov/anaconda3/envs/openllm/lib/python3.11/site-packages/bitsandbytes/libbitsandbytes_cuda118.so
Loading checkpoint shards:   0%|                                                                                        | 0/2 [00:00<?, ?it/s]
Killed

[aarnphm]

can you send python -m bitsandbytes?

[aarnphm]

Yes this should work with latest version. Please try again