FEAT / Trainer: Experimental feature - add GGUF conversion when pushing the model to Hub

[younesbelkada]

What does this PR do?

Introduces a new quantization_config that is intended to be used only for trainer.push_to_hub(), it calls ` a GGUF conversion Space under the hood - (for now: https://huggingface.co/spaces/ybelkada/gguf-my-repo that we will move to a more "official" org)

The API looks like the following:

from datasets import load_dataset
from trl import SFTTrainer, SFTConfig
from transformers import TrainingArguments, GgufConfig

dataset = load_dataset("imdb", split="train")

args = SFTConfig(output_dir="test-gguf-trainer")

trainer = SFTTrainer(
    "ybelkada/tiny-random-llama",
    train_dataset=dataset,
    dataset_text_field="text",
    max_seq_length=512,
    args=args
)

gguf_config = GgufConfig(
    quantization_method="Q8_0",
    private=True,
    space_name="ybelkada/gguf-my-repo"
)

trainer.push_to_hub(gguf_config=gguf_config)

cc @amyeroberts @Vaibhavs10

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[younesbelkada]

Suggested change

	space_name: Optional[str] = "ggml-org/gguf-my-repo",
	space_name: Optional[str] = "ggml-org/gguf-my-repo-trainer",

[younesbelkada]

Would love to get a first round of review before adding tests and documentation ! cc @amyeroberts @muellerzr

[younesbelkada]

Example generated GGUF repo: https://huggingface.co/ybelkada/test-gguf-trainer-Q8_0-GGUF

[muellerzr]

Overall this makes sense from a it works perspective, but is there a reason why we are dedicating a space to this rather than using something like a Hub util? (or baking this in here, for an offline util instead of online)

E.g. I'd rather this be able to be applied locally rather than on the Hub personally... (ran into this same thought process with safetensors the other day)

Going transformers -> gradio -> hf_hub seems a bit middle-man to me.

[younesbelkada]

This is possible but would require quite a huge work, the conversion script maintained by the community: https://github.com/ggerganov/llama.cpp/blob/master/convert.py evolves quite quickly, which would make maintaining that feature quite hard. On the other hand, the official Space uses docker and is build on that conversion script: https://huggingface.co/spaces/ggml-org/gguf-my-repo/blob/main/app.py#L102
one direction could be to build a separate python package / or submodule within transformers to convert locally transformers model to GGUF, with the caveat being the maintenance burden between the official llama.cpp conversion script and our submodule. What do you think? cc @Vaibhavs10 as well

[younesbelkada]

cc @LysandreJik wdyt ?

[Vaibhavs10]

This sounds like a good approach to me too! With Quantisation we have seen that the script evolves wayyy too quickly sometimes from day to day.

That's why the initial recommendation was for it to be used via the space! That said, for offline use-cases perhaps we just use the convert script and update it periodically via a corn - maybe everyday?

This way the only script we'd need to update would be this: https://github.com/ggerganov/llama.cpp/blob/master/convert-hf-to-gguf.py

Then we use this to convert to fp16 and the quantisation code remains the same.

[amyeroberts]

Thanks for opening this!

[amyeroberts]

I really don't think either of these arguments make sense to be included as part of a "quantization config" - they're not configuring quantization, they're configuring a space. It also creates a clash of arguments with the current trainer. If I have private=True in this config and call hub_private_repo=False in the training arguments, which one is should be respected?

I would move these somewhere else (how configurable do we need them to be?)

[younesbelkada]

Hmm yes makes sense, let me think how best we could address that

[amyeroberts]

We need some protection around whether gradio_client is available

[amyeroberts]

hmmmm.... I'm really not sure about this, my gut feeling is that it's pretty dirty piggy-backing gradio here.

I'd like for this general logic to be moved out to a function rather than being injected directly into a core trainer method

[younesbelkada]

Makes sense, will move that out in a separate method !