TE FP8 integration #1096
TE FP8 integration #1096
Conversation
Signed-off-by: Sanju C Sudhakaran <[email protected]>
| @@ -574,6 +574,46 @@ python3 ../gaudi_spawn.py --use_deepspeed --world_size 8 run_lora_clm.py \ | |||
| --flash_attention_causal_mask True | |||
| ``` | |||
|
|
|||
| - Multi-card finetuning of Llama2-70B with DeepSpeed ZeRO-3 optimization, LoRA and FP8 precision: | |||
There was a problem hiding this comment.
Maybe we can just modify existing command for FP8 precision (instead of adding a new one).
| @@ -291,14 +291,6 @@ class GaudiTrainingArguments(TrainingArguments): | |||
| metadata={"help": "Whether to use fp8 for training."}, | |||
| ) | |||
|
|
|||
| fp8_recipe_format: Optional[str] = field( | |||
There was a problem hiding this comment.
@libinta please note we deleted this because we do not want to expose low level numeric selection to user as of now. In future we can revisit this.
|
@regisss @libinta please help review this PR. This is a cleaned up version of TE integration (tested with Llama-7B, Llama-70B, Mixtral-7B fine-tuning tasks). Changes were done mainly for following reasons,
FYI @SanjuCSudhakaran will create another PR adding Llama-7B test for this feature. It is tested with 1.16.0 release and can be merged without waiting for 1.17.0 |
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
|
Related PR with tests #1106 |
vivekgoe
added
the
synapse1.17
What does this PR do?
Refactoring
transformer_engineFP8 support in optimum-habana. This removes model specific changes and makes the code more model agnostic.Fixes # (issue)
Before submitting