Successfully trained the CViViT! Working on the second step

[obvious-research]

Hello @lucidrains !

Based on your great repo, we have been working on replicating phenaki for a few months. Thus we worked on the first step of training, the CViViT. Things were pretty tough to setup with our cluster, and we made quite a lot of changes from your code. Since there were too many changes and we wanted to release the first step of training only without the code for the second step, we decided to create a separate repo here instead of doing a PR here which would have been a mess.

Here's the repo: https://github.com/obvious-research/phenaki-cvivit, with the model weights release on huggingface: https://huggingface.co/obvious-research/phenaki-cvivit

The model works well, it's been trained on Webvid10M and we intend on doing the same for the second step of training.

As you seemed interested in knowing about progress based on this repo (#28), we thought it was ok to open an issue here just to contact you :) We are a collective of artists working with AI that are opening our AI research lab, and we are interested in creating image and video models and releasing them open source.

What's the best way to contact you/dm you ? We are working on the second step of training at the moment. We have access to quite a lot of computing power so we think we have everything we need to actually do it. As we are making good progress towards a full replication, it'll be nice to collaborate if needed, maybe we'll have some questions and your help would be incredible! No pressure though, it's only if this is interesting to you.

Have a great day, and thanks again for the great repo!

[lucidrains]

@obvious-research hey, sorry i missed this! congratulations on successful training and on the open sourcing!

i'm actually working on a cvivit successor over at this repository

definitely welcome you to give it a whirl once it is ready in a few days to see if the paper's conclusions check out

thank you!

[lucidrains]

best way to contact me is just [email protected] lol

[obvious-research]

Definitely interested in MAGViT as well :) If we manage to find some time we'll try it out. And once again thanks for your amazing work ! 🫡

[RomanticWithoutStatus]

where is CViViT.pt which we can download ? thanks bro

[obvious-research]

Here: https://huggingface.co/obvious-research/phenaki-cvivit