Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update options in notebook #98

Open
wants to merge 3 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
24 changes: 13 additions & 11 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,14 +17,15 @@ You can either dream up different versions of the same prompt, or morph between
The app is built with [Gradio](https://gradio.app/), which allows you to interact with the model in a web app. Here's how I suggest you use it:

1. Use the "Images" tab to generate images you like.
- Find two images you want to morph between
- These images should use the same settings (guidance scale, scheduler, height, width)
- Keep track of the seeds/settings you used so you can reproduce them

- Find two images you want to morph between
- These images should use the same settings (guidance scale, scheduler, height, width)
- Keep track of the seeds/settings you used so you can reproduce them

2. Generate videos using the "Videos" tab
- Using the images you found from the step above, provide the prompts/seeds you recorded
- Set the `num_interpolation_steps` - for testing you can use a small number like 3 or 5, but to get great results you'll want to use something larger (60-200 steps).
- You can set the `output_dir` to the directory you wish to save to
- Using the images you found from the step above, provide the prompts/seeds you recorded
- Set the `num_interpolation_steps` - for testing you can use a small number like 3 or 5, but to get great results you'll want to use something larger (60-200 steps).
- You can set the `output_dir` to the directory you wish to save to

## Python Package

Expand Down Expand Up @@ -57,6 +58,7 @@ pipeline = StableDiffusionWalkPipeline.from_pretrained(
video_path = pipeline.walk(
prompts=['a cat', 'a dog'],
seeds=[42, 1337],
fps=5, # use 5 for testing, 25 or 30 for better quality
num_interpolation_steps=3,
height=512, # use multiples of 64 if > 512. Multiples of 8 if < 512.
width=512, # use multiples of 64 if > 512. Multiples of 8 if < 512.
Expand All @@ -69,7 +71,7 @@ video_path = pipeline.walk(

### Making Music Videos

*New!* Music can be added to the video by providing a path to an audio file. The audio will inform the rate of interpolation so the videos move to the beat 馃幎
_New!_ Music can be added to the video by providing a path to an audio file. The audio will inform the rate of interpolation so the videos move to the beat 馃幎

```python
from stable_diffusion_videos import StableDiffusionWalkPipeline
Expand All @@ -92,13 +94,16 @@ num_interpolation_steps = [(b-a) * fps for a, b in zip(audio_offsets, audio_offs
video_path = pipeline.walk(
prompts=['a cat', 'a dog'],
seeds=[42, 1337],
fps=fps, # important to set yourself based on the num_interpolation_steps you defined
num_interpolation_steps=num_interpolation_steps,
audio_filepath='audio.mp3',
audio_start_sec=audio_offsets[0],
fps=fps,
height=512, # use multiples of 64 if > 512. Multiples of 8 if < 512.
width=512, # use multiples of 64 if > 512. Multiples of 8 if < 512.
output_dir='dreams', # Where images/videos will be saved
name=None, # Subdir of output dir. will be timestamp by default
batch_size=1, # increase until you go out of memory
guidance_scale=7.5, # Higher adheres to prompt more, lower lets model take the wheel
num_inference_steps=50, # Number of diffusion steps per image generated. 50 is good default
)
Expand All @@ -122,8 +127,7 @@ interface.launch()

## Credits

This work built off of [a script](https://gist.github.com/karpathy/00103b0037c5aaea32fe1da1af553355
) shared by [@karpathy](https://github.com/karpathy). The script was modified to [this gist](https://gist.github.com/nateraw/c989468b74c616ebbc6474aa8cdd9e53), which was then updated/modified to this repo.
This work built off of [a script](https://gist.github.com/karpathy/00103b0037c5aaea32fe1da1af553355) shared by [@karpathy](https://github.com/karpathy). The script was modified to [this gist](https://gist.github.com/nateraw/c989468b74c616ebbc6474aa8cdd9e53), which was then updated/modified to this repo.

## Contributing

Expand Down Expand Up @@ -168,5 +172,3 @@ from stable_diffusion_videos import RealESRGANModel
model = RealESRGANModel.from_pretrained('nateraw/real-esrgan')
model.upsample_imagefolder('path/to/images/', 'path/to/output_dir')
```