Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] App gets unresponsive after bitsandbytes training error #565

Closed
maxjeblick opened this issue Jan 15, 2024 · 1 comment 路 Fixed by #720
Closed

[BUG] App gets unresponsive after bitsandbytes training error #565

maxjeblick opened this issue Jan 15, 2024 · 1 comment 路 Fixed by #720
Assignees
@pascal-pfeiffer
Labels
type/bug Bug in code

Comments

@maxjeblick
Copy link
Contributor

馃悰 Bug

App hangs after hitting some quantization issue in bitsandbytes.
The issue seems related to bitsandbytes, raising AssertionErrors in random parts of the training pipeline does not result in UI issues.

To Reproduce

I ran into this issue while testing #564. To reproduce on current main:

  1. Disable configuration checks in the code
  2. Run default experiment with 0 gpus and 4 bit quantization
  3. App gets unresponsive after the AssertionError is raised

LLM Studio version

8d1c136

INFO:     127.0.0.1:34268 - "POST / HTTP/1.1" 200 OK
2024-01-15 14:47:42,145 - INFO: Initializing client True
2024-01-15 14:47:42,255 - INFO: {'home/compute_stats', 'dataset/list', 'home/disk_usage', 'home/experiments_stats', 'home/gpu_stats', 'init_app'}
2024-01-15 14:47:42,368 - INFO: PREV None text_causal_language_modeling_config None 1 None 14 
2024-01-15 14:47:42,368 - INFO: Starting from CFG
2024-01-15 14:47:42,380 - INFO: From dataset True
2024-01-15 14:47:42,380 - INFO: From cfg True
2024-01-15 14:47:42,380 - INFO: From default True
2024-01-15 14:47:42,380 - INFO: Config file: text_causal_language_modeling_config
INFO:     127.0.0.1:34268 - "POST / HTTP/1.1" 200 OK
2024-01-15 14:47:46,272 - INFO: Initializing client True
2024-01-15 14:47:46,387 - INFO: Starting experiment
2024-01-15 14:47:46,387 - INFO: experiment/start/cfg_file
2024-01-15 14:47:46,387 - INFO: CFG: text_causal_language_modeling_config
2024-01-15 14:47:46,511 - INFO: Percentage of RAM memory used: 24.0
2024-01-15 14:47:46,511 - INFO: Process: 258437, Queue: [], GPUs: ()
2024-01-15 14:47:46,853 - INFO: {'home/compute_stats', 'experiment/start/footer', 'dataset/list', 'home/disk_usage', 'home/experiments_stats', 'home/gpu_stats', 'init_app', 'experiment/start'}
[2024-01-15 14:47:47,948] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect)
INFO:     127.0.0.1:34268 - "POST / HTTP/1.1" 200 OK
2024-01-15 14:47:48,584 - INFO: Initializing client True
2024-01-15 14:47:48,709 - INFO: {'experiment/list', 'home/compute_stats', 'experiment/start/footer', 'dataset/list', 'home/disk_usage', 'home/experiments_stats', 'dataset/display/footer', 'home/gpu_stats', 'init_app', 'experiment/start'}
/home/max/.local/share/virtualenvs/h2o-llmstudio-0x1AIe7C/lib/python3.10/site-packages/transformers/deepspeed.py:23: FutureWarning: transformers.deepspeed module is deprecated and will be removed in a future version. Please import deepspeed modules directly from transformers.integrations
  warnings.warn(
2024-01-15 14:47:49,287 - WARNING: Training on CPU. This will be slow.
2024-01-15 14:47:49,287 - INFO: Problem Type: text_causal_language_modeling
2024-01-15 14:47:49,287 - INFO: Global random seed: 85069
2024-01-15 14:47:49,287 - INFO: Preparing the data...
2024-01-15 14:47:49,287 - INFO: Setting up automatic validation split...
2024-01-15 14:47:49,339 - INFO: Preparing train and validation data
2024-01-15 14:47:49,339 - INFO: Loading train dataset...
INFO:     127.0.0.1:34268 - "POST / HTTP/1.1" 200 OK
2024-01-15 14:47:49,512 - INFO: Initializing client True
2024-01-15 14:47:49,628 - INFO: {'experiment/list', 'home/compute_stats', 'experiment/start/footer', 'dataset/list', 'home/disk_usage', 'home/experiments_stats', 'dataset/display/footer', 'home/gpu_stats', 'init_app', 'experiment/start'}
2024-01-15 14:47:49,743 - INFO: Stop token ids: [tensor([  529, 29989, 12011, 29989, 29958]), tensor([  529, 29989,  5205, 29989, 29958]), tensor([  529, 29989, 14032,   415, 29989, 29958])]
2024-01-15 14:47:49,768 - INFO: Loading validation dataset...
2024-01-15 14:47:49,968 - INFO: Stop token ids: [tensor([  529, 29989, 12011, 29989, 29958]), tensor([  529, 29989,  5205, 29989, 29958]), tensor([  529, 29989, 14032,   415, 29989, 29958])]
2024-01-15 14:47:49,972 - INFO: Number of observations in train dataset: 8191
2024-01-15 14:47:49,972 - INFO: Number of observations in validation dataset: 83
2024-01-15 14:47:50,310 - INFO: Stop token ids: [tensor([  529, 29989, 12011, 29989, 29958]), tensor([  529, 29989,  5205, 29989, 29958]), tensor([  529, 29989, 14032,   415, 29989, 29958])]
2024-01-15 14:47:50,312 - WARNING: PAD token id not matching between config and tokenizer. Overwriting with tokenizer id.
2024-01-15 14:47:50,312 - INFO: Setting pretraining_tp of model config to 1.
2024-01-15 14:47:50,314 - INFO: Using int4 for backbone
2024-01-15 14:47:50,314 - INFO: Loading h2oai/h2ogpt-4096-llama2-7b. This may take a while.
INFO:     127.0.0.1:34268 - "POST / HTTP/1.1" 200 OK
2024-01-15 14:47:50,392 - INFO: Initializing client True
2024-01-15 14:47:50,519 - INFO: {'experiment/list', 'home/compute_stats', 'experiment/start/footer', 'dataset/list', 'home/disk_usage', 'home/experiments_stats', 'dataset/display/footer', 'home/gpu_stats', 'init_app', 'experiment/start'}
Loading checkpoint shards: 100%|鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻坾 2/2 [00:00<00:00,  8.74it/s]
2024-01-15 14:47:51,058 - INFO: Loaded h2oai/h2ogpt-4096-llama2-7b.
INFO:     127.0.0.1:34268 - "POST / HTTP/1.1" 200 OK
2024-01-15 14:47:51,224 - INFO: Initializing client True
2024-01-15 14:47:51,368 - INFO: {'experiment/list', 'home/compute_stats', 'experiment/start/footer', 'dataset/list', 'home/disk_usage', 'home/experiments_stats', 'dataset/display/footer', 'home/gpu_stats', 'init_app', 'experiment/start'}
INFO:     127.0.0.1:35864 - "POST / HTTP/1.1" 200 OK
2024-01-15 14:47:56,412 - INFO: Initializing client True
2024-01-15 14:47:56,538 - INFO: {'experiment/list', 'home/compute_stats', 'experiment/start/footer', 'dataset/list', 'home/disk_usage', 'home/experiments_stats', 'dataset/display/footer', 'home/gpu_stats', 'init_app', 'experiment/start'}
INFO:     127.0.0.1:35864 - "POST / HTTP/1.1" 200 OK
2024-01-15 14:47:57,353 - INFO: Initializing client True
2024-01-15 14:47:57,477 - INFO: {'experiment/list', 'home/compute_stats', 'experiment/start/footer', 'dataset/list', 'home/disk_usage', 'home/experiments_stats', 'dataset/display/footer', 'home/gpu_stats', 'init_app', 'experiment/start'}
2024-01-15 14:48:16,556 - INFO: Lora module names: ['q_proj', 'k_proj', 'v_proj', 'o_proj', 'gate_proj', 'up_proj', 'down_proj']
trainable params: 9,994,240 || all params: 13,224,415,232 || trainable%: 0.0755741545064032
2024-01-15 14:48:44,508 - INFO: Enough space available for saving model weights.Required space: 26515.43MB, Available space: 511215.53MB.
2024-01-15 14:48:44,754 - INFO: Training Epoch: 1 / 1
2024-01-15 14:48:44,755 - INFO: train loss:   0%|          | 0/4095 [00:00<?, ?it/s]
2024-01-15 14:48:46,332 - INFO: Evaluation step: 4095
2024-01-15 14:48:49,137 - INFO: Stop token ids: [tensor([  529, 29989, 12011, 29989, 29958]), tensor([  529, 29989,  5205, 29989, 29958]), tensor([  529, 29989, 14032,   415, 29989, 29958])]
/home/max/.local/share/virtualenvs/h2o-llmstudio-0x1AIe7C/lib/python3.10/site-packages/torch/utils/checkpoint.py:429: UserWarning: torch.utils.checkpoint: please pass in use_reentrant=True or use_reentrant=False explicitly. The default value of use_reentrant will be updated to be False in the future. To maintain current behavior, pass use_reentrant=True. It is recommended that you use use_reentrant=False. Refer to docs for more details on the differences between the two variants.
  warnings.warn(
FP4 quantization state not initialized. Please call .cuda() or .to(device) on the LinearFP4 layer first.
2024-01-15 14:48:49,556 - ERROR: Exception occurred during H2O LLM Studio run:
Traceback (most recent call last):
  File "/media/max/3tbdrive/PycharmProjects/h2o-llmstudio/train_wave.py", line 106, in <module>
    run(cfg=cfg)
  File "/media/max/3tbdrive/PycharmProjects/h2o-llmstudio/train.py", line 963, in run
    val_loss, val_metric = train_function(
  File "/media/max/3tbdrive/PycharmProjects/h2o-llmstudio/train.py", line 268, in run_train
    output_dict = model.forward(batch)
  File "/media/max/3tbdrive/PycharmProjects/h2o-llmstudio/llm_studio/src/models/text_causal_language_modeling_model.py", line 95, in forward
    output = self.backbone(
  File "/home/max/.local/share/virtualenvs/h2o-llmstudio-0x1AIe7C/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/max/.local/share/virtualenvs/h2o-llmstudio-0x1AIe7C/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/max/.local/share/virtualenvs/h2o-llmstudio-0x1AIe7C/lib/python3.10/site-packages/peft/peft_model.py", line 918, in forward
    return self.base_model(
  File "/home/max/.local/share/virtualenvs/h2o-llmstudio-0x1AIe7C/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/max/.local/share/virtualenvs/h2o-llmstudio-0x1AIe7C/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/max/.local/share/virtualenvs/h2o-llmstudio-0x1AIe7C/lib/python3.10/site-packages/peft/tuners/tuners_utils.py", line 94, in forward
    return self.model.forward(*args, **kwargs)
  File "/home/max/.local/share/virtualenvs/h2o-llmstudio-0x1AIe7C/lib/python3.10/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = old_forward(*args, **kwargs)
  File "/home/max/.local/share/virtualenvs/h2o-llmstudio-0x1AIe7C/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py", line 1181, in forward
    outputs = self.model(
  File "/home/max/.local/share/virtualenvs/h2o-llmstudio-0x1AIe7C/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/max/.local/share/virtualenvs/h2o-llmstudio-0x1AIe7C/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/max/.local/share/virtualenvs/h2o-llmstudio-0x1AIe7C/lib/python3.10/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = old_forward(*args, **kwargs)
  File "/home/max/.local/share/virtualenvs/h2o-llmstudio-0x1AIe7C/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py", line 1058, in forward
    layer_outputs = self._gradient_checkpointing_func(
  File "/home/max/.local/share/virtualenvs/h2o-llmstudio-0x1AIe7C/lib/python3.10/site-packages/torch/_compile.py", line 24, in inner
    return torch._dynamo.disable(fn, recursive)(*args, **kwargs)
  File "/home/max/.local/share/virtualenvs/h2o-llmstudio-0x1AIe7C/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 328, in _fn
    return fn(*args, **kwargs)
  File "/home/max/.local/share/virtualenvs/h2o-llmstudio-0x1AIe7C/lib/python3.10/site-packages/torch/_dynamo/external_utils.py", line 17, in inner
    return fn(*args, **kwargs)
  File "/home/max/.local/share/virtualenvs/h2o-llmstudio-0x1AIe7C/lib/python3.10/site-packages/torch/utils/checkpoint.py", line 451, in checkpoint
    return CheckpointFunction.apply(function, preserve, *args)
  File "/home/max/.local/share/virtualenvs/h2o-llmstudio-0x1AIe7C/lib/python3.10/site-packages/torch/autograd/function.py", line 539, in apply
    return super().apply(*args, **kwargs)  # type: ignore[misc]
  File "/home/max/.local/share/virtualenvs/h2o-llmstudio-0x1AIe7C/lib/python3.10/site-packages/torch/utils/checkpoint.py", line 230, in forward
    outputs = run_function(*args)
  File "/home/max/.local/share/virtualenvs/h2o-llmstudio-0x1AIe7C/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/max/.local/share/virtualenvs/h2o-llmstudio-0x1AIe7C/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/max/.local/share/virtualenvs/h2o-llmstudio-0x1AIe7C/lib/python3.10/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = old_forward(*args, **kwargs)
  File "/home/max/.local/share/virtualenvs/h2o-llmstudio-0x1AIe7C/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py", line 796, in forward
    hidden_states, self_attn_weights, present_key_value = self.self_attn(
  File "/home/max/.local/share/virtualenvs/h2o-llmstudio-0x1AIe7C/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/max/.local/share/virtualenvs/h2o-llmstudio-0x1AIe7C/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/max/.local/share/virtualenvs/h2o-llmstudio-0x1AIe7C/lib/python3.10/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = old_forward(*args, **kwargs)
  File "/home/max/.local/share/virtualenvs/h2o-llmstudio-0x1AIe7C/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py", line 691, in forward
    query_states = self.q_proj(hidden_states)
  File "/home/max/.local/share/virtualenvs/h2o-llmstudio-0x1AIe7C/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/max/.local/share/virtualenvs/h2o-llmstudio-0x1AIe7C/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/max/.local/share/virtualenvs/h2o-llmstudio-0x1AIe7C/lib/python3.10/site-packages/peft/tuners/lora.py", line 1208, in forward
    result = super().forward(x)
  File "/home/max/.local/share/virtualenvs/h2o-llmstudio-0x1AIe7C/lib/python3.10/site-packages/bitsandbytes/nn/modules.py", line 248, in forward
    out = bnb.matmul_4bit(x, self.weight.t(), bias=bias, quant_state=self.weight.quant_state)
  File "/home/max/.local/share/virtualenvs/h2o-llmstudio-0x1AIe7C/lib/python3.10/site-packages/bitsandbytes/autograd/_functions.py", line 567, in matmul_4bit
    assert quant_state is not None
AssertionError
@maxjeblick maxjeblick added the type/bug Bug in code label Jan 15, 2024
@maxjeblick maxjeblick self-assigned this Jan 15, 2024
@maxjeblick
Copy link
Contributor Author

maxjeblick commented Jan 15, 2024 •
edited

Will check if it this issue occurs with cpu only and push a fix to #564 eventually.
What's puzzling is that the error in the subprocess freezes the parent process.

@pascal-pfeiffer pascal-pfeiffer mentioned this issue May 23, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@pascal-pfeiffer pascal-pfeiffer

Labels
type/bug Bug in code
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix unresponsive app h2oai/h2o-llmstudio
2 participants
@pascal-pfeiffer @maxjeblick