SDFT

Overview

SDFT is a self-educational project, aimed to overview main Stable Diffusion fine-tuning techniques. Stable Diffusion implementation is taken from HuggingFace diffusers library.

Techniques to overview:

• Low-Rank Adaptation
• Textual Inversion
• DreamBooth

Table of Contents

• Overview
• Dataset
• Techniques

Dataset

All fine-tuning techniques were performed on a hand-built toy dataset named "Dark Fantasy". The dataset was collected using fine-grained prompts with the Stable Diffusion XL Base-1.0 model from StabilityAI to generate dark-fantasy-like images in a style reminiscent of the 1970s and 1980s. The goal is to demonstrate how all the techniques outlined work on this dataset.

The dataset could be found under the datasets/ directory.

Techniques

LoRA

Usage

To fine-tune SDXL with LoRA:

accelerate launch train_lora_sdxl.py \
    --pretrained_model_name_or_path=stabilityai/stable-diffusion-xl-base-1.0 \
    --pretrained_vae_model_name_or_path=madebyollin/sdxl-vae-fp16-fix \
    --allow_tf32 \
    --mixed_precision="fp16" \
    --rank=32 \
    --train_data_dir=datasets/dark_fantasy/ \
    --caption_column="text" \
    --dataloader_num_workers=16 \
    --resolution=512 \
    --use_center_crop \
    --use_random_flip \
    --train_batch_size=2 \
    --gradient_accumulation_steps=4 --gradient_checkpointing \
    --max_train_steps=1500 \
    --learning_rate=1e-04 \
    --max_grad_norm=5 \
    --lr_scheduler="cosine_with_restarts" \
    --lr_warmup_steps=100 \
    --output_dir=runs/lora_run/ \
    --checkpointing_steps=100 \
    --validation_epochs=10 \
    --num_validation_images=4 \
    --save_images_on_disk \
    --validation_prompt="A picture of a misterious figure in cape, back view." \
    --logging_dir="logs" \
    --seed=1337

To run inference with LoRA checkpoint:

accelerate launch run_lora_inference.py \
    --pretrained_model_name_or_path=stabilityai/stable-diffusion-xl-base-1.0 \
    --pretrained_vae_model_name_or_path=madebyollin/sdxl-vae-fp16-fix \
    --output_dir=runs/lora_v1/ \
    --lora_checkpoint_path=runs/lora_run/checkpoint-100/ \
    --resolution=1024 \
    --num_images_to_generate=5 \
    --guidance_scale=5.0 \
    --num_inference_steps=40 \
    --prompt="A picture of a misterious figure in cape, back view." \
    --negative_prompt="logo, watermark, text, blurry" \
    --seed=1337

Results

No LoRA - LoRA images comparison. Pairs of images were generated using the same latents.

"A picture of a heavy red Kenworth truck riding in the night across the abanoned city streets."

"A picture of a wounded orc warrior, climbing in misty mountains, front view, exhausted face, looking at the camera."

"A picture of space rocket launching, Earth on the background, candid photo."

"A picture of a supermassive black hole, devouring the galaxy, cinematic picture"

"A picture of a human woman warrior, black hair, looking at the camera, front view."

---

Textual Inversion

Usage

To fine-tune SDXL with Textual Inversion (TI):

accelerate launch train_ti_sdxl.py \
    --pretrained_model_name_or_path=stabilityai/stable-diffusion-xl-base-1.0 \
    --pretrained_vae_model_name_or_path=madebyollin/sdxl-vae-fp16-fix \
    --allow_tf32 \
    --mixed_precision="fp16" \
    --train_data_dir=datasets/skull \
    --learnable_property="style" \
    --placeholder_token="<skull_lamp>" \
    --initializer_token="skull" \
    --num_vectors=8 \
    --resolution=1024 \
    --repeats=1 \
    --train_batch_size=2 \
    --gradient_accumulation_steps=4 --gradient_checkpointing \
    --max_train_steps=5000 \
    --learning_rate=3e-3 \
    --lr_scheduler="piecewise_constant" \
    --lr_warmup_steps=30 \
    --output_dir="runs/ti_run" \
    --validation_prompt="A painting of Eiffel tower in the style of <skull_lamp>" \
    --num_validation_images=4 \
    --validation_steps=100 \
    --embeddings_save_steps=500 \
    --save_images_on_disk \
    --use_random_flip \
    --use_center_crop \
    --seed=1337 \

To run inference with trained TI embeddings:

accelerate launch run_ti_inference.py \
    --pretrained_model_name_or_path=stabilityai/stable-diffusion-xl-base-1.0 \
    --pretrained_vae_model_name_or_path=madebyollin/sdxl-vae-fp16-fix \
    --output_dir=runs/ti_run \
    --path_to_embeddings=runs/ti_run/ti-embeddings-final.safetensors \
    --resolution=1024 \
    --num_images_to_generate=1 \
    --guidance_scale=5.0 \
    --num_inference_steps=50 \
    --placeholder_token="<skull_lamp>" \
    --prompt="A <skull_lamp>, made of lego" \
    --negative_prompt="logo, watermark, text, blurry, bad quality" \
    --seed=1337

Results

No TI - TI images comparison. Pairs of images were generated using the same latents.

NOTE: since training dataset consists of 5 images only with default captions, results are not that inspiring, but introducing more fine-graided captions would make it much better.

"A <skull_lamp>, made of lego."

"A painting of Eiffel tower in the style of <skull_lamp>."

"A painting of the great pyramids in the style of <skull_lamp>."

"An oil painting of a skyscraper in the style of <skull_lamp>."

"The painting of a mug in the style of <skull_lamp>."

---