stable diffusion image variation/InstructPix2Pix/StableDiffusionXLImg2ImgPipeline pipeline

[sywangyi]

What does this PR do?

Fixes # (issue)

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you make sure to update the documentation with your changes?
• Did you write any new necessary tests?

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[sywangyi]

image_to_image_generation.py.txt
benchmark result with A100
stabilityai/stable-diffusion-xl-refiner-1.0:

command in A100
python image_to_image_generation.py
--model_name_or_path "stabilityai/stable-diffusion-xl-refiner-1.0"
--src_image_path "https://raw.githubusercontent.com/timothybrooks/instruct-pix2pix/main/imgs/example.jpg"
--prompts "turn him into cyborg"
--num_images_per_prompt 1
--cuda
--bf16
--guidance_scale 7.5
--num_inference_steps 10
--image_save_dir /tmp/stable_diffusion_images

command in Gaudi2
python image_to_image_generation.py
--model_name_or_path "stabilityai/stable-diffusion-xl-refiner-1.0"
--src_image_path "https://raw.githubusercontent.com/timothybrooks/instruct-pix2pix/main/imgs/example.jpg"
--prompts "turn him into cyborg"
--num_images_per_prompt 1
--guidance_scale 7.5
--num_inference_steps 10
--bf16
--use_habana
--use_hpu_graphs
--gaudi_config Habana/stable-diffusion
--image_save_dir /tmp/stable_diffusion_images

perf result A100 233ms Gaudi2 212ms

[sywangyi]

"lambdalabs/sd-image-variations-diffusers"
python image_to_image_generation.py
--model_name_or_path "lambdalabs/sd-image-variations-diffusers"
--src_image_path "https://github.com/SHI-Labs/Versatile-Diffusion/blob/master/assets/demo/reg_example/ghibli.jpg?raw=true"
--num_images_per_prompt 1
--image_save_dir /tmp/stable_diffusion_images
--guidance_scale 3
--bf16
--cuda

Gaudi2 command
python image_to_image_generation.py
--model_name_or_path "lambdalabs/sd-image-variations-diffusers"
--src_image_path "https://github.com/SHI-Labs/Versatile-Diffusion/blob/master/assets/demo/reg_example/ghibli.jpg?raw=true"
--num_images_per_prompt 1
--image_save_dir /tmp/stable_diffusion_images
--guidance_scale 3
--bf16
--use_habana
--use_hpu_graphs
--gaudi_config Habana/stable-diffusion
perf result A100 1.593s Gaudi2 1.205s

[sywangyi]

timbrooks/instruct-pix2pix:
command in A100
python image_to_image_generation.py
--model_name_or_path "timbrooks/instruct-pix2pix"
--src_image_path "https://raw.githubusercontent.com/timothybrooks/instruct-pix2pix/main/imgs/example.jpg"
--prompts "turn him into cyborg"
--num_images_per_prompt 1
--batch_size 1
--guidance_scale 7.5
--image_guidance_scale 1
--num_inference_steps 10
--image_save_dir /tmp/stable_diffusion_images
--bf16 --cuda

command in Gaudi2:
python image_to_image_generation.py
--model_name_or_path "timbrooks/instruct-pix2pix"
--src_image_path "https://raw.githubusercontent.com/timothybrooks/instruct-pix2pix/main/imgs/example.jpg"
--prompts "turn him into cyborg"
--num_images_per_prompt 1
--batch_size 1
--guidance_scale 7.5
--image_guidance_scale 1
--num_inference_steps 10
--image_save_dir /tmp/stable_diffusion_images
--use_habana
--use_hpu_graphs
--gaudi_config Habana/stable-diffusion
--bf16

perf result. A100 468ms Gaudi2 398ms

[dsocek]

Very clean PR. Please update warmup logic steps (see comments).