What does this PR do?

This PR updates the examples/language-modeling/run_lora_clm.py script with the flexibility to support more named datasets from Hugging Face Hub. Previously, this example script was written to only support two named datasets for non-SQL prompts: tatsu-lab/alpaca and timdettmers/openassistant-guanaco (specifying other datasets would raise an error saying that the dataset is unsupported). The tatsu-lab/alpacadataset has standard columns for instruction, input, and output that are formed into a prompt string before tokenization. timdettmers/openassistant-guanaco has a single text column.

To support more datasets, I added arguments to allow the user to specify column names for "instruction", "input", and "output" (or in the case of using SQL prompts it's "question", "context", and "answer"). If the user provides custom column names, I am renaming those columns to use the standard names (instruction/input/output or question/context/answer) so that the rest of the code in the script will work as expected. Also, for non-SQL prompt datasets, the "input" is optional (which is why there's a "prompt_with_input" and "prompt_without_input" template). The existing code was handing the case where the "input" column was blank, but was not able to handle the the "input" column not existing, so I updated the code to also handle the "input" column not existing at all.

I left the code specific to timdettmers/openassistant-guanaco alone, since that was a more special use case with it only having the one column and not getting preprocessed with the prompt template.

I tested this update with several different HF datasets: databricks/databricks-dolly-15k, ruggsea/stanford-encyclopedia-of-philosophy_instruct, b-mc2/sql-create-context, flytech/python-codes-25k, gbharti/finance-alpaca, medalpaca/medical_meadow_medical_flashcards.

As an example, databricks/databricks-dolly-15k is a dataset that has different column names than the original code expects (it has "instruction", "context", and "response" instead of "instruction", "input", and "output"). To use run_lora_clm.py with this dataset, you can include args for --input_column_name "context" and --output_column_name "response".

  python run_lora_clm.py \
  --model_name_or_path huggyllama/llama-7b \
  --dataset_name databricks/databricks-dolly-15k \
  --dataset_concatenation True \
  --per_device_train_batch_size 16 \
  --evaluation_strategy "no" \
  --save_strategy "no" \
  --learning_rate 1e-4 \
  --warmup_ratio  0.03 \
  --lr_scheduler_type "constant" \
  --max_grad_norm  0.3 \
  --gradient_accumulation_steps 1 \
  --learning_rate 2e-4 \
  --num_train_epochs 3 \
  --output_dir /tmp/output \
  --overwrite_output_dir \
  --validation_split_percentage 20 \
  --use_fast_tokenizer False \
  --lora_rank 8 \
  --lora_alpha 16 \
  --lora_dropout 0.1 \
  --lora_target_modules q_proj v_proj \
  --do_train True \
  --do_eval True \
  --use_habana \
  --use_lazy_mode \
  --adam_epsilon 1e-08 \
  --lr_scheduler_type "constant" \
  --max_grad_norm  0.3 \
  --bf16 \
  --throughput_warmup_steps 3 \
  --input_column_name "context" \
  --output_column_name "response" \
  --max_steps 50 \
  --max_eval_samples 50

This can also be run without the --input_column_name arg, which would then test the use case where a dataset does not have an "input" column at all.

Fixes #629

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you make sure to update the documentation with your changes?
• Did you write any new necessary tests?