AWQ Model Works from UI in Windows, But Fails When Launched from .bat File

[Yusuf-Demiryurek]

Hi,

I need your help :)

I'm trying to launch H2OGPT from a .bat file on Windows with an AWQ model as a parameter, but it doesn't seem to work. Below is the content of the .bat file :
cd %HOMEDRIVE%\%HOMEPATH%\h2ogpt_data
C:\Users\yuyu\AppData\Local\Programs\h2oGPT\Python\pythonw.exe "C:\Users\yuyu\AppData\Local\Programs\h2oGPT\h2oGPT.launch.pyw" --load_awq=model --base_model=TheBloke/zephyr-7B-beta-AWQ

When I use a GGUF model instead of AWQ, it works fine. Also, loading the AWQ model from the UI works without any issues.

I've tried various parameters and searched through the documentation, but nothing has helped me to get it running from the .bat file.

I'm running it on a computer with 128GB of RAM and an RTX 4090 GPU.

Could someone please help me? Thanks. 😄

[pseudotensor]

I'm not an expert at .bat files. I used chatgpt to help me with .bat file stuff for h2oGPT, so I recommend doing same with your errors or seeing if someone else will respond.

Also, you didn't say what the issue is.

[Yusuf-Demiryurek]

I don't believe the issue lies with the .bat file itself. The .bat file functions similarly to the python generate.py command with parameters. The problem arises when attempting to run the AWQ model through the .bat file. Interestingly, the AWQ model runs perfectly fine when loaded from the UI, but H2OGPT doesn't start when using the .bat file.

Furthermore, if I use --base_model=TheBloke/zephyr-7B-beta-GGUF instead of --base_model=TheBloke/zephyr-7B-beta-AWQ in the .bat file, H2OGPT starts without any issues with the GGUF model loaded.

There appears to be a discrepancy in behavior between loading the model from the UI and using the --base_model input in the .bat file.

[pseudotensor]

Hi, please say what the issue actually is. You haven't said what the error or stack trace is etc.

[Yusuf-Demiryurek]

Sorry, I didn't know where to find the error.
After consulting GPT, I found the error in the h2ogpt_exception.log file:

Traceback (most recent call last):
  File "C:\Users\balas\AppData\Local\Programs\h2oGPT\pkgs\win_run_app.py", line 150, in main
    _main()
  File "C:\Users\balas\AppData\Local\Programs\h2oGPT\pkgs\win_run_app.py", line 136, in _main
    main_h2ogpt()
  File "C:\Users\balas\AppData\Local\Programs\h2oGPT\generate.py", line 12, in entrypoint_main
    H2O_Fire(main)
  File "C:\Users\balas\AppData\Local\Programs\h2oGPT\src\utils.py", line 65, in H2O_Fire
    fire.Fire(component=component, command=args)
  File "C:\Users\balas\AppData\Local\Programs\h2oGPT\pkgs\fire\core.py", line 141, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "C:\Users\balas\AppData\Local\Programs\h2oGPT\pkgs\fire\core.py", line 475, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "C:\Users\balas\AppData\Local\Programs\h2oGPT\pkgs\fire\core.py", line 691, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "C:\Users\balas\AppData\Local\Programs\h2oGPT\src\gen.py", line 1977, in main
    model0, tokenizer0, device = get_model_retry(reward_type=False,
  File "C:\Users\balas\AppData\Local\Programs\h2oGPT\src\gen.py", line 2305, in get_model_retry
    model1, tokenizer1, device1 = get_model(**kwargs)
  File "C:\Users\balas\AppData\Local\Programs\h2oGPT\src\gen.py", line 2455, in get_model
    model_loader, tokenizer_loader, conditional_type = get_loaders(**loader_kwargs)
  File "C:\Users\balas\AppData\Local\Programs\h2oGPT\pkgs\..\src\loaders.py", line 100, in get_loaders
    from awq import AutoAWQForCausalLM
  File "C:\Users\balas\AppData\Local\Programs\h2oGPT\pkgs\awq\__init__.py", line 2, in <module>
    from awq.models.auto import AutoAWQForCausalLM
  File "C:\Users\balas\AppData\Local\Programs\h2oGPT\pkgs\awq\models\__init__.py", line 1, in <module>
    from .mpt import MptAWQForCausalLM
  File "C:\Users\balas\AppData\Local\Programs\h2oGPT\pkgs\awq\models\mpt.py", line 1, in <module>
    from .base import BaseAWQForCausalLM
  File "C:\Users\balas\AppData\Local\Programs\h2oGPT\pkgs\awq\models\base.py", line 16, in <module>
    from awq.modules.linear import WQLinear_GEMM, WQLinear_GEMV
  File "C:\Users\balas\AppData\Local\Programs\h2oGPT\pkgs\awq\modules\linear.py", line 4, in <module>
    import awq_inference_engine  # with CUDA kernels
ImportError: DLL load failed while importing awq_inference_engine: Le module sp cifi  est introuvable.

[pseudotensor]

I can import upper level thing but I don't have the other thing:

(h2ogpt) jon@pseudotensor:~/h2ogpt$ python
Python 3.10.14 | packaged by conda-forge | (main, Mar 20 2024, 12:45:18) [GCC 12.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import awq_inference_engine
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'awq_inference_engine'
>>> from awq import AutoAWQForCausalLM
>>> 

Probably the awq install instructions are out of date for windows, will check on it.

[Yusuf-Demiryurek]

I got this:

Traceback (most recent call last):
  File "C:\Users\balas\AppData\Local\Programs\h2oGPT\Python\test.py", line 1, in <module>
    import awq_inference_engine
ImportError: DLL load failed while importing awq_inference_engine: Le module spécifié est introuvable.

I don't understand why it works when I load the model from UI and not in this way