Add the Stable diffusion inpaint support

[yuanwu2017]

What does this PR do?

The PR tried to reproduce the stable diffusion inpaint on Gaudi2 according to Huggingface inpaint doc.

1. Add the GaudiStableDiffusionInpaintPipeline. It is based on diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_inpaint.py
2. Add the GaudiStableDiffusionXLInpaintPipeline. It is based on diffusers/pipelines/stable_diffusion_xl/pipeline_stable_diffusion_xl_inpaint.py
3. Add the auto pipeline support. It is based on auto_pipeline.py
4. Add an example for stable diffusion inpaint
5. Add the unitest

TODO:
Add the batch_size support.

Fixes # (issue)

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you make sure to update the documentation with your changes?
• Did you write any new necessary tests?

[yuanwu2017]

Run the examples:
1 GaudiStableDiffusionInpaintPipeline example:
python text_to_image_inpainting.py --use_habana --use_hpu_graphs --num_images_per_prompt 1 --model_name_or_path runwayml/stable-diffusion-inpainting

Generated image:

2 GaudiStableDiffusionXLInpaintPipeline example:
python text_to_image_inpainting.py --use_habana --use_hpu_graphs --num_images_per_prompt 1 --model_name_or_path diffusers/stable-diffusion-xl-1.0-inpainting-0.1

Generated image:

[yuanwu2017]

@regisss Please help to review. Thanks.

[yuanwu2017]

Performance benchmark:
benchmark_inpaint.py.txt

1. GaudiStableDiffusionInpaintPipeline
   python benchmark_inpaint.py --use_habana --use_hpu_graphs --num_images_per_prompt 8 --model_name_or_path runwayml/stable-diffusion-inpainting
   prompt number: 2
   Gaudi2:
   average time [ms]: 14585.93477010727
   A100:
   average time [ms]: 15131.173181533813
2. GaudiStableDiffusionXLInpaintPipeline
   python benchmark_inpaint.py --use_habana --use_hpu_graphs --num_images_per_prompt 4 --model_name_or_path diffusers/stable-diffusion-xl-1.0-inpainting-0.1
   prompt number: 2
   Gaudi2:
   average time [ms]: 37599.966740608215
   A100:
   average time [ms]: 43453.8525223732

[yuanwu2017]

@yao-matrix

[yuanwu2017]

@libinta Please help to review.

[yuanwu2017]

CI Test report:
pytest test_diffusers.py
========= 40 passed, 8 skipped, 24 warnings in 562.81s (0:09:22) =========
test_diffusers.log

pytest test_stable_diffusion_inpaint.py
============== 22 passed, 7 skipped, 37 warnings in 373.11s (0:06:13) ===============
test_sd_inpaint.log

pytest test_stable_diffusion_xl_inpaint.py
============= 28 passed, 8 skipped, 4 warnings in 182.60s (0:03:02) =======
test_sd_xl_inpaint.log

[yuanwu2017]

Done: stable_diffusion_xl_inpaint batch performance
Done: fix the failed pytest cases.

[yuanwu2017]

Gaudi2 Pytest:

pytest test_diffusers.py

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
============ 40 passed, 8 skipped, 29 warnings in 568.80s (0:09:28) ============

pytest test_stable_diffusion_inpaint.py

============ 22 passed, 8 skipped, 35 warnings in 313.43s (0:05:13) ============

pytest test_stable_diffusion_xl_inpaint.py
============ 29 passed, 8 skipped, 4 warnings in 384.30s (0:06:24) =============

Gaudi1 Pytest:

pytest test_diffusers.py

============ 40 passed, 8 skipped, 29 warnings in 465.83s (0:07:45) ============

pytest test_stable_diffusion_inpaint.py

============ 22 passed, 8 skipped, 35 warnings in 231.46s (0:03:51) ============

pytest test_stable_diffusion_xl_inpaint.py
============ 29 passed, 8 skipped, 4 warnings in 514.44s (0:08:34) =============

Performance benchamrk

num_inference_steps =10

Gaudi2:

model: runwayml/stable-diffusion-inpainting

model: diffusers/stable-diffusion-xl-1.0-inpainting-0.1

A100 80G

model: runwayml/stable-diffusion-inpainting

model: diffusers/stable-diffusion-xl-1.0-inpainting-0.1

[yuanwu2017]

@regisss @libinta @yeonsily Please help to review the patch.

[JoeyTPChou]

Hi all, I am using this PR for one of our opensource research code. Would like to know when will this PR get merged so I can use it without pointing to the PR?

[yao-matrix]

Hi all, I am using this PR for one of our opensource research code. Would like to know when will this PR get merged so I can use it without pointing to the PR?

@libinta @yeonsily @regisss, do you have a plan to merge it?

[dsocek]

This should be incorporated into text_to_image_generation.py sample as all SD samples are

[yuanwu2017]

This patch has a dependency on MC patch , it also modified the text_to_image_generation.py. Once the mc patch is merged. I will update this patch

[yuanwu2017]

Update the pytest result.
command:
pytest test_diffusers.py

[yuanwu2017]

@dsocek Please help to review.

[libinta]

@yuanwu2017 can we rebase this to main? @dsocek can you review this?

[yuanwu2017]

Yes. It has been the latest.

[dsocek]

Looks good now. Please rebase and merge text_to_image_inpainting.py into text_to_image_generation.py when MC patch merges.

[yuanwu2017]

Ok. No problem.