New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature Request]: Add ControlNet support for 2D frames #176

Open

1 task done

kabachuha opened this issue Jun 10, 2023 · 2 comments

Labels

enhancement help wanted high priority

Owner

kabachuha commented Jun 10, 2023

Is there an existing issue for this?

I have searched the existing issues and checked the recent builds/commits

What would your feature do ?

ControlNet controlling the 2D frames slice, possibly controlling the 3D stuff later as well (But it's called InfiNet for that 😉).

So the things such as below will be possible in ModelScope as well

Proposed workflow

Add a ControlNet tab and do some stuff with the 2D frames using it

Additional information

So, it's basically implemented in VideoCrafter under the name of T2I adapter

kabachuha added enhancement help wanted labels

Owner Author

kabachuha commented Jul 1, 2023

Control a video

See

https://arxiv.org/abs/2305.13840

https://github.com/camenduru/control-a-video-colab

Owner Author

kabachuha commented Jul 2, 2023

https://github.com/Weifeng-Chen/control-a-video - main repo

kabachuha added the high priority label

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment